HK40048959B - Implementation of core cellular networking stack on cloud infrastructure - Google Patents

Description

Implementation of the core cellular network stack on cloud infrastructure

Background Technology

Cellular networks have traditionally operated using custom hardware and software solutions by telecom providers. While some components of cellular networks can operate within cloud infrastructure (e.g., using public cloud providers), certain components and technologies cannot be directly or effectively transferred to the cloud. For example, in LTE cellular networks, the control plane network and the user plane network are considered separate networks. The control plane is responsible for establishing, configuring, and tearing down connections in the user plane. The user plane directly carries user data (network packets from user equipment). The Serving Gateway/Packet Gateway (SPGW) is the component through which user packets flow. To handle large volumes of traffic, many SPGW instances may be required. In cloud implementations, each SPGW instance can run on its own virtual machine. These SPGW instances need to receive control plane messages so that they can correctly process user plane packets.

Load balancers in cloud environments are designed to distribute packets across (e.g., evenly across) multiple virtual machines. Typically, a load balancer has little control over which packets reach which virtual machine, except for attempting to route packets with the same source and destination addresses to the same virtual machine. Because SPGW instances run behind the load balancer in a cloud environment, control plane and user plane data for a given user can be sent to different SPGW instances. In this scenario, it can be difficult or impossible to send control plane information to or have it read by the SPGW instance that needs it.

Therefore, there are numerous opportunities to improve technologies related to implementing cellular network components within cloud environments.

Summary of the Invention

This summary provides a simplified overview of some concepts that will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

This describes techniques for operating the core cellular network stack on cloud computing environments (e.g., public cloud infrastructure). For example, virtualized packet gateways can run on virtual machines in the cloud computing environment, and standard load balancers can distribute cellular network traffic among the virtualized packet gateways. The number of virtualized packet gateways can be scaled up (adding new virtualized packet gateways) or scaled down (removing virtualized packet gateways), and the load balancer distributes network traffic accordingly. Virtualized packet gateways can be set up using a local cache of an external key-value store (KVS) containing the cellular network's bearers, and the local cache can be used to process received data plane network packets. Bearers can be updated within the cellular network using the external key-value store, and virtualized packet gateways can retrieve and use updated bearer details.

For example, a virtualized packet gateway within a cloud computing environment can perform operations to process data plane network packets for cellular networks. The virtualized packet gateway can receive a complete copy of an external key-value store, which includes bearer details for all current bearers of the cellular network, where the bearer details define the network flow associated with the user equipment using the cellular network. The virtualized packet gateway can maintain a complete copy of the external key-value store in a local cache. The virtualized packet gateway can receive data plane network packets from a load balancer. The virtualized packet gateway can process data plane network packets, wherein the processing includes identifying the bearers associated with the data plane network packets in the local cache.

As another example, a virtualized packet gateway operating within a cloud computing environment can receive a first data plane network packet associated with a first bearer. Upon determining that the first bearer is not in its local cache, the virtualized packet gateway can retrieve bearer details for the first bearer from an external key-value store and store the retrieved bearer details for the first bearer in its local cache. The virtualized packet gateway can process the first data plane network packet based at least in part on the bearer details for the first bearer stored in its local cache. The virtualized packet gateway can receive network communication indicating that the first bearer has been updated. In response to receiving network communication, the virtualized packet gateway can retrieve updated bearer details for the first bearer from an external key-value store and store the retrieved updated bearer details in its local cache. The virtualized packet gateway can receive a second data plane network packet associated with the first bearer and process the second data plane network packet based at least in part on the updated bearer details for the first bearer stored in its local cache.

As described in this article, a variety of other features and advantages can be incorporated into the technology as needed.

Attached Figure Description

Figure 1 is a diagram depicting an example cloud computing environment in which a cellular network is implemented, including components such as a virtualized packet gateway.

Figure 2 is a diagram depicting an example cloud computing environment in which a cellular network including SPGW-U components is implemented.

Figure 3 is an example state machine illustrating the packet flow and bearer state of a data plane network.

Figure 4 is a flowchart of an example method for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for cellular networks.

Figure 5 is a flowchart of an example method for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for cellular networks.

Figure 6 is a diagram of an example computing system in which some of the described embodiments can be implemented.

Figure 7 shows an example mobile device that can be used in conjunction with the techniques described herein.

Figure 8 shows an example cloud support environment that can be used in conjunction with the technologies described in this article.

Detailed Implementation

Overview

As described in this article, various technologies and solutions can be applied to operate the core cellular network stack within a cloud computing environment (e.g., public cloud infrastructure). For example, virtualized packet gateways can run on virtual machines in a cloud computing environment, and standard load balancers can distribute cellular network traffic among virtualized packet gateways. The number of virtualized packet gateways can be scaled up (adding new virtualized packet gateways) or scaled down (removing virtualized packet gateways), and the load balancer distributes network traffic accordingly.

To provide cellular network services within a cloud computing environment, a key-value store (e.g., a database, flat file, or another type of data storage) is maintained. The key-value store stores bearer details for the current bearers of the cellular network. These bearer details define the network flows associated with user devices using the cellular network (e.g., mobile phones or other computing devices with cellular connectivity). The key-value store is a data storage separate from other components of the cellular network, such as virtualized packet gateways. In other words, the key-value store is external to the virtualized packet gateway (also known as an external key-value store). In some implementations, the key-value store maintains bearer details for all current bearers of the cellular network (e.g., representing all current network flows active within the cellular network operating within a cloud computing environment).

In the techniques described herein, data plane network packets are processed by virtualized packet gateways. For example, multiple virtualized packet gateways can be instantiated and run on virtual machines in a cloud computing environment to process data plane network packets communicating with and from user equipment. In some implementations (e.g., implementations operating LTE cellular networks), the virtualized packet gateway is a Serving Gateway (SGW), a Packet Gateway (PGW), or a Serving Gateway/Packet Gateway (SPGW). A Serving Gateway/Packet Gateway that processes data plane (also known as user plane) network packets is also referred to as SPGW-U. A Serving Gateway/Packet Gateway that processes control plane network packets is also referred to as SPGW-C.

In traditional cellular networks, components of the telecommunications provider's operating system can include the following.

- User equipment (UE). A device connected to a cellular network. For example, a user equipment may include a mobile phone or other computing device with cellular communication capabilities.

- Evolved Node B (eNodeB) - This component refers to a cellular network base station to which a user equipment connects via a wireless communication connection (e.g., cellular radio).

- Mobility Management Entity (MME). This component controls user equipment's access to the cellular network. - Serving Gateway/Packet Gateway (SPGW). Network packets to and from user equipment flow through the SPGW. The SPGW provides a single anchor point that can be used to relay packets to user equipment regardless of where it moves (e.g., when user equipment moves from one cell tower to another).

In an LTE cellular network environment, the control plane and data plane (user plane) are handled separately. The control plane is responsible for establishing, configuring, and tearing down connections in the data plane. The data plane carries network packets to/from user equipment.

In a cloud computing environment, many SPGW instances may be needed to handle network traffic, and each SPGW instance runs in its own virtual machine (VM). These VM instances require information from the control plane so that they can correctly process data plane network packets.

Load balancers in cloud computing environments are designed to distribute network packets among SPGW instances. A typical load balancer will base its operation on packet header information (e.g., source and destination IP addresses from IP headers, User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) header information, and/or other packet header information). However, there are problems with running SPGW instances in cloud computing environments using traditional techniques. Specifically, when SPGW instances are running behind a load balancer, the control plane and data plane for a given user will often be routed to different SPGW instances. In this scenario, control plane information may not be able to be sent to or read by the SPGW instances that need it. One possible solution is to have the MME (e.g., the source of the control plane information) broadcast its control data to all SPGW instances. However, this possible solution is inefficient (e.g., requires a large amount of network traffic) and may not scale to a small number of SPGWs.

Using the techniques described herein, cellular networks within a cloud computing environment can be operated more efficiently and reliably. For example, scaling out can be performed by preloading the contents of a key-value store (e.g., the entire contents of the key-value store) onto a newly instantiated virtualized packet gateway. The virtualized packet gateway can then be prepared to handle any data plane network packets that a load balancer routes to the virtualized packet gateway during a scaling event. As another example, bearers can be moved between virtualized packet gateways. For example, ownership information can be stored (e.g., in the key-value store) indicating which virtualized packet gateway is currently handling data plane network packets for a given bearer. If data plane network packets are routed to another virtualized packet gateway (with a different owner than the current one), the other virtualized packet gateway can acquire ownership (e.g., register itself in the key-value store to replace the previous owner). In this way, network traffic to bearers can move between virtualized packet gateways (e.g., due to a scaling-out event or some other reason such as network congestion or failure) without having to close the connection and start a new network traffic. As another example, bearers can be updated (e.g., when they are active and there is no need to close the bearers and set up new ones). For example, a control plane management component (e.g., an MME and/or gateway that processes control plane traffic) can update bearer details in a key-value store. The control plane management component can signal to a virtualized packet gateway that has the updated bearer, and the virtualized packet gateway can obtain the updated bearer details and use the updated bearer details during the processing of data plane network packets for the bearer.

Operating cellular networks in a cloud computing environment

In the techniques described herein, components of the cellular network can be implemented within a cloud computing environment (e.g., a public cloud environment and/or a private cloud environment). For example, a standard load balancer in a cloud environment can be used to distribute cellular network traffic (e.g., data plane network packets) across a collection (e.g., a cluster) of virtualized packet gateways (e.g., SPGW-U). An external key-value store can store bearer details of current network flows within the cellular network. Virtualized packet gateways can be extended by copying the entire contents of the external key-value store into a local cache of a new virtualized packet gateway, making the new virtualized packet gateway ready to handle any network flows sent to it by the load balancer. This technique also enables bearers to be updated (e.g., for the virtualized packet gateway to receive and apply updated bearer details) and moved (e.g., for moving bearers between virtualized packet gateways). Bearers can be moved and/or updated while remaining active without needing to shut down the bearer and set up a new bearer.

Figure 1 is a diagram depicting an example cloud computing environment 110, within which components of a cellular network (e.g., a virtual cellular network) are implemented. For example, the cloud computing environment 110 can be a public cloud environment or a private cloud environment. The cloud computing environment 110 provides computing resources (e.g., servers, virtual machines, storage resources, network services, etc.) on which the components of the cellular network run.

As shown in Figure 1, user equipment 120 and 122 (e.g., mobile phones or other computing devices with cellular network connectivity) connect to the cellular network via cellular base station 130 (e.g., a cell tower). For example, if the cellular network is a Long Term Evolution (LTE) cellular network, then cellular base station 130 can be an eNodeB. Other types of cellular networks, such as 3G or 5G cellular networks, can also be used (e.g., which may have different types of cellular base stations 130). Although only two example user equipment devices 120 and 122 are shown, cellular networks can typically support many more user equipment. Similarly, cellular networks will typically support multiple cellular base stations located in different geographical locations.

As shown in Figure 1, multiple cellular network components are implemented within the cloud computing environment 110. The control plane management component 140 provides control plane services and handles control plane network traffic for the cellular network. For example, the control plane management component 140 can configure network flows (e.g., bearers) for user equipment 120 and 122 to access service 180 (e.g., to a public data network (PDN) such as the Internet or a cellular data network). Depending on the type of cellular network, the control plane management component 140 may include many different services. For example, for an LTE cellular network, the control plane management component 140 may include a Mobility Management Entity (MME), services that handle control plane network traffic, and/or packet gateways, load balancers, etc.

Key-value store 170 is a data store (e.g., a centralized or distributed data store) containing bearer details for a cellular network bearer. For example, when control plane management component 140 receives a request from user equipment 120 to access a website on the Internet (one of services 180) or to make a voice call, control plane management component 140 establishes a bearer (e.g., this may include determining whether user equipment 120 is authorized to access the service, such as checking data plans, billing information, etc.). As part of setting up the bearer, control plane management component 140 stores bearer details for the bearer (e.g., TEID, QoS value, network bitrate such as download speed, etc.) in key-value store 170. A bearer represents a type of network service (e.g., network flow) for a specific user equipment device. Different bearers can be created for different types of network services and/or different applications, including telephone calls, website browsing, special traffic (e.g., streaming video, specific VoIP applications, etc.), and/or different qualities of service (QoS). A given user equipment device may have multiple currently active bearers representing different network flows. In some implementations, in order for user equipment devices to use cellular networks, bearers for network flows must first be established within the cellular network.

Virtualized packet gateway 160 (e.g., multiple virtualized packet gateways in a cluster running on virtual machines) processes data plane network packets for user equipment 120 and 122. For example, virtualized packet gateway 160 processes data plane network packets transmitted between user equipment 120 and 122 and service 180 (e.g., the Internet and/or other external networks) (“to” and “from”).

When one of the virtualized packet gateways 160 receives a first data plane network packet for a given network flow (e.g., a voice call, access to a specific website, etc.) from a particular user equipment device, the virtualized packet gateway obtains bearer details for that bearer from key-value store 170 (e.g., the bearer has been pre-configured in the key-value store by the control plane management component 140). The virtualized packet gateway may store the bearer details for the bearer in a local cache (e.g., the virtualized packet gateway's local key-value store).

Data plane network packets (as Internet Protocol (IP) network packets) between user equipment 120 and virtualized packet gateway 160 are encapsulated. In some implementations, encapsulation is performed according to a GPRS tunneling protocol (GTP) such as GTP-U. For example, encapsulation provides a way to manage different network flows for a given user equipment device. The tunneling protocol assigns identifiers (e.g., tunnel endpoint identifiers (TEIDs)) to bearers associated with a given network flow, and separate identifiers can be used for uplink data (data flowing from user equipment device to service 180) and downlink data (data flowing from service 180 to user equipment device). For example, a specific bearer can be set up for a specific network flow in which user equipment 120 is accessing a specific website, where the bearer details include two identifiers (e.g., two TEIDs)—one identifier associated with the uplink traffic of the network flow and one identifier associated with the downlink traffic of the network flow.

Load balancer 150 distributes data plane network packets among virtualized packet gateways 160. Load balancer 150 is a standard load balancer in the cloud computing environment 110 (e.g., a standard load balancer provided by a public cloud service provider), which directs network traffic based on IP header information. Because load balancer 150 is a standard load balancer, it does not direct network packets based on encapsulated packet information (e.g., it is not a dedicated load balancer configured to work with encapsulation protocols such as GTP-U). Although an example load balancer 150 is shown in the figures, multiple load balancers (e.g., multiple load balancer network devices) can be used.

Because load balancer 150 is a standard load balancer, it typically routes network packets for a given network flow to the same virtualized packet gateway based on external IP header information (e.g., source IP address and destination IP address and/or port number). However, since load balancer 150 is not aware of the bearer (e.g., it does not inspect the encapsulated packets or the encapsulated header), it cannot determine which virtualized packet gateway will receive network packets for a given bearer. One possible solution is to pass all bearer details to all virtualized packet gateways. However, this possible solution is inefficient (e.g., using significant computing resources including network bandwidth, processing resources, and storage).

Using the techniques described herein, a more efficient solution can be provided. For example, by using key-value store 170 as a repository of bearer details for cellular network bearers, virtualized packet gateway 160 can obtain bearer details when needed. For instance, when one of the virtualized packet gateways 160 receives a data plane network packet associated with an unknown bearer (e.g., a bearer not in the virtualized packet gateway's local cache), the virtualized packet gateway can obtain the bearer details for the bearer from external key-value store 170 and store those details in the virtualized packet gateway's local cache. This provides an improvement in computational resources (e.g., network bandwidth, processing resources, and storage) compared to a solution that stores all bearer details at every location in virtualized packet gateway 160.

Another potential problem that may arise during the scaling up of virtualized packet gateway 160 is the operation of cellular network components within a cloud computing environment. For example, during a scaling up event where a new virtualized packet gateway is added to virtualized packet gateway 160 (e.g., a scaling up event where a new virtualized packet gateway instance is instantiated on a virtual machine), load balancer 150 will begin sending data plane network packets to the new virtual gateway in proportion to the virtualized packet gateway 160 (e.g., evenly distributing data plane network traffic among virtualized packet gateways 160). If the new virtualized packet gateway begins receiving a large volume of network traffic from bearers it has not yet encountered, the new virtualized packet gateway may become overloaded (e.g., causing network packets to be delayed or dropped, which could lead to network service delays or interruptions). One possible solution is to have the new virtualized packet gateway retrieve bearer details from key-value store 170 for each new bearer it encounters (e.g., upon receiving the first network packet associated with a given bearer). However, when the new virtualized packet gateway starts operating and the load balancer 150 begins boots up network packets, the new virtualized packet gateway can still become overloaded in the event of a sudden influx of network traffic and the need to obtain the corresponding new bearer details.

Using the techniques described herein, problems such as cascading events can be reduced or eliminated. For example, after a new virtualized packet gateway is created and before it begins receiving data plane network packets from load balancer 150, bearer details can be copied from key-value store 170 to the new virtualized packet gateway's local cache. In some implementations, a complete copy of key-value store 170 is stored in the new virtualized packet gateway's local cache. Once the copy of key-value store 170 is stored at the new virtualized packet gateway, load balancer 150 begins routing data plane network packets to the new virtualized packet gateway. Because the new virtualized packet gateway's local cache is already populated (e.g., pre-populated) with bearer details, any data plane network packets it receives from load balancer 150 will have associated bearer details in the local cache, and the new virtualized packet gateway will be able to process them efficiently without retrieving the bearer details from key-value store 170. This results in reliability (e.g., reduced chance of network service disruption to user equipment) and savings in network responsiveness (e.g., reduced latency).

A similar issue may arise during the scaling down of virtualized packet gateway 160. For example, the remaining virtualized packet gateways 160 may receive a portion of the network traffic previously served by the disconnected virtualized packet gateways. However, in this case, additional network traffic directed to the given remaining virtualized packet gateways via load balancer 150 should not overload them. Therefore, the remaining virtualized packet gateways can obtain new bearer details for the additional network traffic from key-value store 170 when needed.

Another potential challenge in operating cellular network components within a cloud computing environment arises when it's necessary to move bearers between virtualized packet gateways and/or when bearer details need to be updated. For example, during expansion events, contraction events, or for other reasons (e.g., network congestion or overloaded virtualized packet gateways), it may be necessary to move bearers representing ongoing network flows. In some solutions, bearers are bound to a specific virtualized packet gateway and cannot be moved (e.g., a failure of a virtualized packet gateway would cause the failure of all bearers served by that gateway and associated network flows). In the techniques described herein, bearers and associated network flows can be moved between virtualized packet gateways. For example, if load balancer 150 directs a network flow of a bearer to another virtualized packet gateway, that other virtualized packet gateway can obtain bearer details from key-value store 170 upon receiving the first data plane network packet of the network flow and serve that bearer. This solution can be implemented during expansion events, contraction events, or for other reasons that cause network flows to move to other virtualized packet gateways.

Similar issues may arise when bearers need to be updated. For example, a bearer update may be required due to changes in account status (e.g., bandwidth limitations, accessed service types, billing issues, etc.). In some solutions, bearers cannot be updated during network flows (e.g., if changes are needed, a new bearer needs to be established, which is inefficient and requires additional computational resources). In the techniques described herein, bearers can be updated. To update bearers, key-value store 170 can store bearer ownership information indicating which virtualization packet gateway owns each bearer (e.g., the data plane network packet currently responsible for processing the bearer). For example, when a virtualization packet gateway receives a first data plane network packet associated with a specific bearer (e.g., where the specific bearer is not in the virtualization packet gateway's local cache, and/or where no packet for the specific bearer has been processed), the virtualization packet gateway can obtain the bearer details for the specific bearer from key-value store 170 and register itself as the owner of the specific bearer in key-value store 170 (e.g., key-value store 170 can maintain a separate table and establish an association between the bearer and the virtualization packet gateway). When a bearer needs to be updated, the control plane management component 140 can update the bearer details in the key-value store 170, determine which virtualized packet gateway owns the bearer, and alert the virtualized packet gateway so that it can retrieve the updated bearer details from the key-value store 170. This solution allows bearers to be updated during network flows without stopping and re-establishing new network flows, saving time and computational resources and reducing or eliminating network outages.

Figure 2 is a diagram depicting an example cloud computing environment 210, within which components, including the SPGW-U, implement an LTE cellular network (e.g., a virtual LTE cellular network). For example, cloud computing environment 210 can be a public cloud environment or a private cloud environment. Cloud computing environment 210 provides computing resources (e.g., servers, virtual machines, storage resources, network services, etc.) on which the cellular network components run. Many of the components described in Figure 2 perform the same or similar functions as the corresponding components described above with respect to Figure 1.

As shown in Figure 2, user equipment 220 and 222 (e.g., mobile phones or other computing devices with cellular network connectivity) connect to the cellular network via eNodeB 230 (e.g., a cellular base station within an LTE cellular network). Other types of cellular networks, such as 3G or 5G cellular networks, can also be used. Although only two example user equipment devices 220 and 222 are shown, cellular networks can typically support many more user equipment devices. Similarly, cellular networks will typically support multiple cellular base stations located in different geographical locations.

As shown in Figure 2, multiple cellular network components are implemented within a cloud computing environment 210. These components provide control plane services and handle control plane network traffic for the cellular network, including an MME 240 and a Service Packet Gateway (SPGW-C) 262. For example, the MME 240, together with the SPGW-C 262, can establish network flows (e.g., bearers) for user equipment 220 and 222 to access services 290 (e.g., to a Public Data Network (PDN) such as the Internet or a cellular data network). A load balancer 252 (or multiple load balancers in some implementations) can distribute control plane network traffic (e.g., to handle the control plane network traffic load of the cellular network) among multiple SPGW-Cs 262.

Key-value store 270 is a data storage (e.g., centralized or distributed data storage) containing bearer details for use in the cellular network. For example, when MME 240 receives a request from user equipment 220 to access a website (one of services 290) on the Internet or to make a voice call, MME 240 establishes a bearer (e.g., this may include determining whether user equipment 220 is authorized to access the service, such as checking data plans, billing information, etc.). For example, MME 240 may instruct SPGW-C 262 to configure the bearer and store its bearer details in key-value store 270. A bearer represents a network service type (e.g., network flow) for a specific user equipment device. Different bearers can be created for different types of network services and/or different applications, including telephone calls, website browsing, special traffic (e.g., streaming video, specific VoIP applications, etc.), and/or different qualities of service (QoS). A given user equipment device may have multiple currently active bearers representing different network flows. In some implementations, for a user equipment device to use the cellular network, a bearer must first be established within the cellular network for the network flow.

The SPGW-U 260 (e.g., multiple SPGW-Us in a cluster running on virtual machines) processes data plane network packets for user equipment 220 and 222. For example, the SPGW-U 260 processes data plane network packets that are transmitted between user equipment 220 and 222 and service 290 (e.g., the Internet and/or other external networks) (“to” and “from”). In some implementations, a Network Address Translation (NAT) service 280 is used for IP address translation between the SPGW-U 260 and service 290 (e.g., the Internet).

When one of the SPGW-Us 260 receives a first data plane network packet from a specific user equipment device for a given network flow (e.g., a voice call, access to a specific website, etc.), the SPGW-U obtains bearer details from the key-value store 270 (e.g., the bearer will have been pre-configured in the key-value store by the MME 240 and/or SPGW-C 262). The SPGW-U may store the bearer details in a local cache (e.g., the SPGW-U's local key-value store).

Data plane network packets (as Internet Protocol (IP) network packets) between User Equipment 220 and SPGW-U 260 are encapsulated. In some implementations, encapsulation is performed according to a GPRS tunneling protocol (GTP) such as GTP-U. For example, encapsulation provides a way to manage different network flows for a given User Equipment device. The tunneling protocol assigns an identifier (e.g., a tunnel endpoint identifier (TEID)) to the bearer associated with a given network flow, and separate identifiers can be used for uplink data (data flowing from the User Equipment device to Service 290) and downlink data (data flowing from Service 290 to the User Equipment device). For example, a specific bearer can be set up for a specific network flow in which User Equipment 220 is accessing a specific website, where the bearer details include two identifiers (e.g., two TEIDs), one associated with the uplink traffic of the network flow and one associated with the downlink traffic of the network flow.

Load balancer 250 distributes data plane network packets among SPGW-U 260. Load balancer 250 is a standard load balancer in the cloud computing environment 210 (e.g., a standard load balancer provided by a public cloud service provider), which directs network traffic based on IP header information. Because load balancer 250 is a standard load balancer, it does not direct network packets based on encapsulated packet information (e.g., network packets are not target-specific load balancers configured to work with encapsulation protocols such as GTP-U). Although an example load balancer 250 is shown in the figures, multiple load balancers (e.g., multiple load balancer network devices) can be used.

Because load balancer 250 is a standard load balancer, it typically routes network packets for a given network flow to the same SPGW-U based on external IP header information (e.g., source and destination IP addresses and/or port numbers). However, since load balancer 250 is not aware of the bearer (e.g., it does not inspect the encapsulated packets or the encapsulated headers), it cannot determine which SPGW-U will receive network packets for a given bearer. One possible solution is to communicate all bearer details to all SPGW-Us. However, this possible solution is inefficient (e.g., using significant computing resources including network bandwidth, processing resources, and storage).

Using the techniques described herein, a more efficient solution can be provided. For example, by using key-value store 270 as a repository of bearer details for cellular network bearers, SPGW-U 260 can obtain bearer details when needed. For instance, when one of the SPGW-U 260 receives a data plane network packet associated with an unknown bearer (e.g., a bearer not in the SPGW-U's local cache), the SPGW-U can obtain the bearer details from the external key-value store 270 and store them in its local cache. This provides improvements in computational resources (e.g., network bandwidth, processing resources, and storage) compared to a solution where all bearer details are stored at each SPGW-U within the SPGW-U 260.

Another potential problem that may occur during the scaling of SPGW-U 262 is operating cellular network components within a cloud computing environment. For example, during a scaling event where a new SPGW-U is added to SPGW-U 262 (e.g., a scaling event where a new SPGW-U instance is instantiated on a virtual machine), load balancer 250 will begin sending data plane network packets to the new SPGW-U in proportion to the number of SPGW-U 262 (e.g., evenly distributing data plane network traffic among SPGW-U 262). If the new SPGW-U begins receiving a large amount of network traffic from bearers that do not yet exist, the new SPGW-U may become overloaded (e.g., causing network packets to be delayed or dropped, which could lead to network service delays or interruptions). One possible solution is to have the new SPGW-U retrieve bearer details from key-value store 270 for each new bearer it encounters (e.g., upon receiving the first network packet associated with a given bearer). However, when the new SPGW-U starts operating and the load balancer 250 begins boots up network packets, the new SPGW-U may still become overloaded in the event of a sudden influx of network traffic and the need to obtain the corresponding new bearer details.

Using the techniques described herein, such problems as scaling events can be reduced or eliminated. For example, after a new SPGW-U is created and before it begins receiving data plane network packets from load balancer 250, bearer details can be copied from key-value store 270 to the new SPGW-U's local cache. In some implementations, a complete copy of key-value store 270 is stored in the new SPGW-U's local cache. Once the copy of key-value store 270 is stored at the new SPGW-U, load balancer 250 begins routing data plane network packets to the new SPGW-U. Because the new SPGW-U's local cache is already populated with bearer details (e.g., pre-populated), any data plane network packets received by the new SPGW-U from load balancer 250 will have associated bearer details in the local cache, and the new SPGW-U will be able to efficiently process data plane network packets without retrieving bearer details from key-value store 270. This results in savings in network response (e.g., reduced latency) and improved reliability (e.g., reduced chance of network service disruptions to user equipment).

A similar issue may arise during the scaling down of SPGW-U 262. For example, the remaining SPGW-U 262 may receive a portion of the network traffic previously served by the disconnected SPGW-U. However, in this case, the additional network traffic directed to a given remaining SPGW-U via load balancer 250 should not overload the remaining SPGW-U. Therefore, the remaining SPGW-U can obtain new bearer details for the additional network traffic from key-value store 270 when needed.

Another potential problem that can arise when operating cellular network components within a cloud computing environment is the need to move bearers between SPGW-Us and/or when bearer details need to be updated. For example, a bearer representing an ongoing network flow may need to be moved during an expansion event, a contraction event, or for some other reason (e.g., network congestion or SPGW-U overload). In some solutions, bearers are bound to a specific SPGW-U and cannot be moved (e.g., a failure of an SPGW-U would cause the failure of all bearers served by that SPGW-U and associated network flows). In the techniques described herein, bearers and associated network flows can be moved between SPGW-Us. For example, if load balancer 250 directs a network flow of a bearer to another SPGW-U, that other SPGW-U can retrieve the bearer details from key-value store 270 upon receiving the first data plane network packet of the network flow and serve that bearer. This solution can be implemented during an expansion event, a contraction event, or for some other reason that causes a network flow to move to another SPGW-U.

Similar issues may arise when bearers need to be updated. For example, a bearer update may be required due to changes in account status (e.g., bandwidth limitations, accessed service types, billing issues, etc.). In some solutions, bearers cannot be updated during network flows (e.g., if changes are needed, a new bearer needs to be established, which is inefficient and requires additional computational resources). In the techniques described herein, bearers can be updated. To update bearers, key-value store 170 can store bearer ownership information indicating which SPGW-U owns each bearer (e.g., the data plane network packet currently handling the bearer). For example, when an SPGW-U receives a first data plane network packet associated with a specific bearer (e.g., where the specific bearer is not in the SPGW-U's local cache, and/or where a packet for the specific bearer has not yet been processed), the SPGW-U can obtain the bearer details for the specific bearer from key-value store 270 and register itself as the owner of the specific bearer in key-value store 170 (e.g., key-value store 270 can maintain a separate table and establish a relationship between the bearer and the SPGW-U). When a bearer needs to be updated, the MME 240 and/or SPGW-C 262 can update the bearer details in the key-value store 270, determine which SPGW-U is the owner of the bearer, and alert the SPGW-U so that the SPGW-U can retrieve the updated bearer details from the key-value store 270. This solution allows bearers to be updated during network flows without stopping and re-establishing new network flows, saving time and computational resources, and reducing or eliminating network outages.

In the example cloud computing environment 210, the SPGW-U 260 is described as a combined solution where the Serving Gateway and Packet Gateway for data plane traffic are combined into an integrated or co-located Serving Gateway/Packet Gateway component. However, in some implementations, the Serving Gateway and Packet Gateway are separate components that operate independently. For example, there may be several reasons why the individual UEs, SGWs, and PGWs may be located in different locations. One reason is Domain Name Service (DNS) selection. When a UE attaches, the Evolved Packet Core (EPC) can use a DNS server to select the SGW and PGW separately. The selection algorithm can use different criteria for each algorithm; the SGW can be based on location (typically selecting the SGW closest to the base station), while the PGW can be based on Access Point Name (APN) (effectively limiting the type of service and network the UE needs to connect to). Another reason is roaming. It is possible for the PGW to be located in the local network while the SGW is located in the visited network. This is typical for mobile virtual network operators (MVNOs), for example.

In implementations where the SGW and PGW are separate (e.g., geographically separated, such as running on different computing devices), the SGW and PGW can communicate via an S5/S8 interface. Two features may exist: one based on GTP tunneling, and the other based on Proxy Mobile IPv6 (PMIPv6) (RFC 5213).

In implementations using a combined SPGW, the S5/S8 interface is not required. The SPGW-U maps IP to GTP TEID in the downlink and TEID to IP in the uplink. However, in implementations where the SGW and PGW are separate and use S5/S8 interfaces, the following mapping can be used:

-PGW-u (based on GTP): Downlink: IP to GTP TEID, Uplink: GTP TEID to IP

-SGW-u (based on GTP): Downlink: GTP TEID to GTP TEID, Uplink: GTP TEID to GTP TEID

-PGW-u (based on PMIPv6): Downlink: IP to Generic Routing Encapsulation (GRE) tunnel; Uplink: GRE tunnel to IP.

-SGW-u (based on PMIPv6): Downlink: GRE tunnel to GTP TEID, Uplink: GTP TEID to GRE tunnel

Figure 3 shows an example state machine 300, illustrating data plane (user plane) network packet flows and bearer states (e.g., ownership, addition, deletion, and update). For example, example state machine 300 can be implemented via a virtualized packet gateway (e.g., via virtualized packet gateway 160) and/or via an SPGW (e.g., via SPGW-U 260 and/or via SPGW-C262). In some implementations, there is an instance of a state machine for each bearer.

Regarding example state machine 300, there are several possible events, including:

1. Uplink packets arrive at the user plane, that is, packets are transmitted from the UE toward the PDN.

2. Downlink packets arrive at the user plane, i.e., from the PDN to the UE.

3. Bearer update packets arrive at the control plane, which can come from MME or KVS.

4. Carry unknown packets from KVS to the control plane.

5. The bearer deletes the packet from the MME to the control plane.

6. The owner adds a confirmation group from KVS to the control plane.

7. The owner deletes the confirmation group from KVS to the control plane.

There are several variables associated with the example state machine 300, and the variables are defined as follows.

Am_owner. This variable indicates whether we (e.g., a virtualized packet gateway) have informed the KVS owner table that we are the owner of this bearer. The following values are possible.

0 We have not yet informed KVS that we are the owners. Add pending We have informed KVS that we are the owners, but have not yet received confirmation. 1 We have informed KVS that we are the owners and have received confirmation from KVS. Deletion pending We have informed KVS that we are no longer the owners, but have not yet received confirmation.

`Read_route_pending` is a boolean value that records whether the SPGW has requested bearer details from the KVS. This variable is used to prevent the SPGW from repeatedly requesting bearer details while waiting to read bearer updates from the KVS.

0 No request to read bearer details was sent. 1 The request has been sent.

`Cached_route_seq` is an integer that records the sequence number present in the bearer update currently stored in our local cache. A value of -1 indicates that the local cache is invalid.

The following states are example state machine 300.

State 1: This state indicates that the bearer is unknown and there is no corresponding entry in the local cache.

States 2, 3, 4, and 5: These are all the intermediate states that are passed through when the system is correctly configured to handle the user plane.

State 6: An intermediate state when the carried cognition is removed from the local cache.

Status 7: We are in a stable state and are correctly configured to process user plane data.

Methods for operating cellular networks within a cloud computing environment

In any of the examples in this document, methods for operating virtual cellular networks within a cloud computing environment can be provided. For example, a virtualized packet gateway (e.g., a service and/or packet gateway) can be implemented on a virtual machine running within a cloud computing environment and process data plane network packets for cellular networks.

Figure 4 is a flowchart of an example method 400 for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for cellular networks. For example, example method 400 can be executed by one of the virtualized packet gateways 160 or by one of the SPGW-Us 260.

At 410, a complete copy of the external key-value store is received. The external key-value store includes bearer details for all current bearers of the cellular network (e.g., all current bearers of a cellular network implemented within a cloud computing environment). The bearer details define the network flows associated with user equipment using the cellular network. For example, the external key-value store could be key-value store 170 or key-value store 270. At 420, the complete copy of the external key-value store received at 410 is stored in a local cache.

At 430, data plane network packets are received from the load balancer. For example, the load balancer may guide network packets based on Internet Protocol (IP) header information rather than on encapsulated packet information (e.g., not based on tunnel identifiers or TEIDs).

At 440, data plane network packets are processed by a virtualized packet gateway. This processing includes identifying the bearers associated with the data plane network packets in the local cache. For example, the virtualized packet gateway can retrieve bearer details from the local cache based on the encapsulated network packet header information (e.g., TEID) to identify network flows (and their associated data plane network packets) and manage them according to their associated bearers (e.g., setting network bit rates, QoS parameters, etc.).

In some implementations, example method 400 is executed during an extended event, in which a virtualized packet gateway is instantiated on a virtual machine within the cloud computing environment. For example, when the virtualized packet gateway starts up and runs, and maintains a complete copy of the key-value store in its local cache, it can instruct the load balancer (e.g., from which it signals or responds to its communication) that it is ready to receive data plane network packets. In response, the load balancer can begin routing network traffic to the virtualized packet gateway.

After the virtualized packet gateway starts up, runs, and processes data plane network packets (e.g., as shown in 440), it can receive new data plane network packets associated with new bearers not in its local cache. When this happens, the virtualized packet gateway can retrieve the bearer details of the new bearer from an external key-value store, store the bearer details in its local cache, and process the new data plane network packets based at least in part on the bearer details of the new bearer. The virtualized packet gateway can also register itself as the owner of the new bearer in the external key-value store.

In some implementations, the virtualized packet gateway tracks when it was last processed for each bearer data plane network packet in its local cache. For example, when the virtualized packet gateway processes a packet for a bearer, it may record timing information (e.g., timestamps or other timing information) associated with the bearer in its local cache. The virtualized packet gateway can use the recorded timing information to purge bearers from its local cache. Although example method 400 allows the virtualized packet gateway to start operations efficiently using a complete copy of an external key-value store, the virtualized packet gateway will not use many bearers in the key-value store (e.g., load balancers will route them to other virtualized packet gateways). Therefore, if a data plane network packet is not processed by the virtualized packet gateway for a period of time (e.g., if the packet is not processed within a threshold time period), the virtualized packet gateway can remove the packet from its local cache, which can save local storage resources.

Figure 5 is a flowchart of an example method 500 for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for cellular networks. For example, example method 500 can be executed by one of the virtualized packet gateways 160 or by one of the SPGW-Us 260.

At point 510, a first data plane network packet associated with the first bearer is received via a virtualized packet gateway. For example, the first data plane network packet can be received from a load balancer at the start of a network flow.

At 520, in response to determining that the first bearer is not in the local cache of the virtualized packet gateway, the bearer details of the first bearer are retrieved from an external key-value store. The retrieved bearer details of the first bearer are stored in the local cache at the virtualized packet gateway (e.g., in a local key-value store). For example, the external key-value store could be key-value store 170 or key-value store 270.

At point 530, the first data plane network packets are processed at least in part based on the first bearer details stored locally (stored in the local cache).

At point 540, communication indicating that the first bearer has been updated is received. This communication can be received, for example, from the MME or from another component of the cellular network (e.g., from a virtualized packet gateway in the management control plane). This communication can be a network control message.

At 550, in response to a communication indicating that the first bearer has been updated, the updated bearer details of the first bearer are retrieved from an external key-value store and stored in a local cache. For example, the first bearer can be updated to reflect a different download bit rate (e.g., based on a user exceeding their monthly limit).

At 560, a second data plane network packet associated with the first bearer is received via a virtualized packet gateway. The second data plane network packet is a subsequent network packet received from the load balancer, which is part of the same network flow as the first data plane network packet.

At point 570, the second data plane network packets are processed at least in part based on the updated bearer details of the first bearer stored locally.

In some implementations, one or more of the following components can be implemented within a cloud computing environment to process data plane network packets. The components can perform one or more of the following operations: For example, a virtualized packet gateway can be configured via computer-executable instructions to store bearer details for a bearer in a local key-value store, wherein the bearer details are obtained from an external key-value store, to register ownership of bearers served by the virtualized packet gateway in the external key-value store, and to process network packets based at least in part on the bearer details stored in the local key-value store. A mobility management entity can be configured via computer-executable instructions to set up bearers in the external key-value store, update bearer details for bearers in the external key-value store, and notify the virtualized packet gateway of the updated bearer details. The external key-value store can be configured via computer-executable instructions to store bearer details of current bearers in the cellular network and to store ownership information indicating the ownership relationship between the virtualized packet gateway and the bearers.

Example Host Establishment

This section illustrates how to establish bearers in specific implementations of LTE cellular networks using the techniques described herein. To illustrate the differences, the bearer establishment sequence is first described for conventional LTE systems, followed by a description of the techniques described herein.

The following sequence illustrates the traditional bearer setup (without using a cloud computing environment) and the processing of first-user plane data services:

- The UE/eNodeB sends an MME bearer establishment request (control plane message).

- The MME sends the bearer establishment request to the SPGW.

-SPGW stores bearer details from the setup request.

-The SPGW responds to the MME, indicating that the bearer establishment is complete.

- Then, if the first data is uplink data:

-SPGW receives user plane packets from eNodeB.

-SPGW decapsulates user plane packets and forwards them to the target Public Data Network (PDN).

-SPGW receives user plane packets from PDN.

The SPGW locates the target IP address and finds the address that matches the UE's IP address from the bearer details stored above. If the packet matches the traffic template (TFT) from the bearer details, the packet is encapsulated using the eNodeB TEID from the bearer details.

-SPGW forwards the encapsulated packets to the eNodeB.

- Alternatively, if the data is primarily downlink data:

-SPGW receives user plane packets from the target PDN.

-SPGW searches for the target IP address, finds a match, and then continues the uplink operation.

Using the techniques described in this article, where an external key-value store (also known as a shared key-value store) is used to implement a virtualized packet gateway (e.g., SPGW-C and/or SPGW-U) within a cloud computing environment, the sequence is as follows:

-eNodeB sends an MME bearer establishment request (control plane message).

-MME sends the bearer setup request to the SPGW-C load balancer.

- For example, the load balancer forwards requests to SPGW-C1 (Instance 1).

The SPGW-C1 stores the UE IP address, TEID (both SPGW and eNodeB values), and eNodeB IP in a shared key-value store.

The SPGW-C1 response to the MME indicates that the bearer establishment has been completed.

-eNodeB sends user plane packets to the SPGW-U load balancer.

- For example, the load balancer forwards user plane packets to SPGW-U2 (Instance 2).

- Then, if the first data is uplink data:

-SPGW-U2 receives user plane packets from the eNodeB.

-SPGW-U2 looks up the TEID from the packets in the shared key-value store. It then retrieves the eNodeB's TEID as a response.

The SPGW-U2 writes its IP address as the "cache owner" for this network flow to a shared key-value store.

The SPGW-U2 writes the UE IP address, TEID (both SPGW and eNodeB values), and eNodeB IP address into the local cache.

-SPGW-U2 decapsulates user plane packets and forwards them to the target PDN.

-SPGW-U2 receives user plane packets from the PDN.

The SPGW-U2 looks up the target IP address in its local cache and finds a match between the target IP address and the UE IP address from the bearer details stored above. If the packet matches the TFT from the bearer details, the packet is encapsulated using the eNodeB TEID from the bearer details.

-SPGW-U2 forwards the encapsulated packets to the eNodeB.

- Alternatively, if the data is primarily downlink data:

The SPGW-U load balancer receives user plane packets from the target PDN.

- The load balancer forwards the data to SPGW-U2.

-SPGW-U2 is searching for bearer details in its local cache. No match found.

The SPGW-U2 looks up the UE IP address in the shared key-value store. It retrieves bearer details as a response and writes them to its local cache.

-SPGW-U2, acting as the "cache owner" of this stream, writes to the IP address of the shared key-value store.

-SPGW-U2 forwards the encapsulated packets to the eNodeB.

The SPGW-U2 receives user plane packets from the eNodeB, decapsulates them, and forwards them to the target PDN.

Computing System

Figure 6 depicts a general example of a suitable computing system 600 in which the described techniques can be implemented. Since the techniques can be implemented in a variety of general-purpose or special-purpose computing systems, computing system 600 is not intended to imply any limitation on its use or scope of function.

Referring to Figure 6, computing system 600 includes one or more processing units 610, 615 and memories 620, 625. In Figure 6, this basic configuration 630 is included within the dashed lines. Processing units 610, 615 execute computer-executable instructions. The processing units may be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. Processing units may also include multiple processors. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 6 illustrates a central processing unit 610 and a graphics processing unit or coprocessor unit 615. Tangible memories 620, 625 may be volatile memories (e.g., registers, caches, RAM), non-volatile memories (e.g., ROM, EEPROM, flash memory, etc.) accessible by the processing units, or some combination of both. Memories 620, 625 store software 680 implementing one or more technologies described herein in the form of computer-executable instructions suitable for execution by the processing units.

The computing system may have additional features. For example, computing system 600 includes memory 640, one or more input devices 650, one or more output devices 660, and one or more communication connections 670. Interconnection mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of computing system 600. Typically, operating system software (not shown) provides an operating environment for other software executing in computing system 600 and coordinates the activities of the components of computing system 600.

The physical memory 640 may be removable or non-removable and includes a magnetic disk, magnetic tape or cassette, CD-ROM, DVD, or any other medium that can be used to store information and is accessible within the computing system 600. The memory 640 stores instructions for implementing software 680 that carries out one or more of the technologies described herein.

Input device 650 may be a touch input device such as a keyboard, mouse, pen, or trackball, a voice input device, a scanning device, or another device that provides input to computing system 600. For video encoding, input device 650 may be a camera, video card, TV tuner card, or a similar device that accepts video input in analog or digital form, or reads video samples into the CD-ROM or CD-RW of computing system 600. Output device 660 may be a monitor, printer, speaker, CD burner, or other device that provides output from computing system 600.

Communication connection 670 enables communication with another computing entity via a communication medium. The communication medium transmits information such as computer-executable instructions, audio or video input or output, or other data in modulated data signals. A modulated data signal is a signal whose characteristics are set or altered in such a way that information is encoded in that signal. By way of example and not limitation, the communication medium may be electrical, optical, RF, or other carriers.

This technology can be described in the general context of computer-executable instructions included in a program module, which execute on a computing system on a target real or virtual processor. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. In various embodiments, the functionality of a program module can be combined or divided among program modules. The computer-executable instructions for a program module can execute within a local or distributed computing system.

The terms “system” and “device” are used interchangeably herein. Neither term implies any limitation on the type of computing system or computing device unless the context clearly indicates otherwise. Generally, a computing system or computing device can be local or distributed and can include any combination of dedicated hardware and/or general-purpose hardware with software that implements the functions described herein.

For ease of presentation, detailed descriptions use terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions of operations performed by a computer and should not be confused with operations performed by humans. The actual computer operations corresponding to these terms depend on the implementation scheme.

mobile devices

Figure 7 is a system diagram depicting an example mobile device 700, which includes various optional hardware and software components generally shown at 702. Although not all connections are shown for ease of illustration, any component 702 of the mobile device can communicate with any other component. The mobile device can be any of a variety of computing devices (e.g., mobile phone, smartphone, handheld computer, personal digital assistant (PDA), etc.) and can allow for two-way wireless communication with one or more mobile communication networks 704, such as cellular networks, satellite networks, or other networks.

For example, the example mobile device 700 may be a user equipment device (e.g., user equipment 120, 122, 220, or 222) using a cellular network (e.g., mobile communication network 704). The example mobile device 700 may communicate with the cellular network via a wireless modem 760.

The illustrated mobile device 700 may include a controller or processor 710 (e.g., a signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing tasks such as signal encoding, data processing, input/output processing, power control, and/or other functions. An operating system 712 may control the allocation and use of component 702 and support one or more applications 714. Applications may include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) or any other computing application. A function 713 for accessing an app store may also be used to obtain and update applications 714.

The illustrated mobile device 700 may include a memory 720. The memory 720 may include non-removable memory 722 and/or removable memory 724. Non-removable memory 722 may include RAM, ROM, flash memory, hard disk, or other known memory storage technologies. Removable memory 724 may include flash memory or a subscriber identity module (SIM) card, well-known in GSM communication systems, or other known memory storage technologies such as a "smart card." The memory 720 may be used to store data and/or code for running an operating system 712 and applications 714. Example data may include web pages, text, images, sound files, video data, or other datasets to be sent to or received from one or more network servers via one or more wired or wireless networks. The memory 720 may be used to store subscriber identifiers such as International Mobile Subscriber Identity (IMSI) and device identifiers such as International Mobile Equipment Identity (IMEI). Such identifiers may be sent to network servers to identify users and devices.

Mobile device 700 may support one or more input devices 730 such as touchscreen 732, microphone 734, camera 736, physical keyboard 738 and/or trackball 740, and one or more output devices 750 such as speaker 752 and display 754. Other possible output devices (not shown) may include piezoelectric or other haptic output devices. Some devices may provide more than one input/output function. For example, touchscreen 732 and display 754 may be combined into a single input/output device.

Input device 730 may include a Natural User Interface (NUI). An NUI is any interface technology that enables a user to interact with a device in a “natural” manner, free from the artificial constraints imposed by input devices such as a mouse, keyboard, or remote control. Examples of NUI methods include those relying on speech recognition, touch and stylus recognition, on-screen and adjacent gesture recognition, air gestures, head and eye tracking, speech and voice, vision, touch, gestures, and machine intelligence. Other examples of NUIs include motion gesture detection using accelerometers/gyroscopes, facial recognition, 3D displays, head, eye, and gaze tracking, immersive augmented reality and virtual reality systems, all of which provide a more natural interface, as well as technologies for sensing brain activity using electric field-sensing electrodes (EEG and related methods). Thus, in a particular example, operating system 712 or application 714 may include speech recognition software as part of a voice user interface that allows the user to operate device 700 via voice commands. Furthermore, device 700 may include input devices and software that allow user interaction via spatial gestures, such as detecting and interpreting gestures to provide input to a game application.

As is known in the art, the wireless modem 760 may be coupled to an antenna (not shown) and may support bidirectional communication between the processor 710 and an external device. The modem 760 is generally shown and may include a cellular modem for communicating with a mobile communication network 704 and/or other radio-based modems (e.g., Bluetooth 764 or Wi-Fi 762). The wireless modem 760 is typically configured to communicate with one or more cellular networks, such as a GSM network for data and voice communication between cellular networks within a single cellular network or between a mobile device and the Public Switched Telephone Network (PSTN).

The mobile device may further include at least one input/output port 780, a power supply 782, a satellite navigation system receiver 784 such as a Global Positioning System (GPS) receiver, an accelerometer 786, and/or a physical connector 790 that may be a USB port, an IEEE 1394 (FireWire) port, and/or an RS-232 port. The components 702 shown are not required or all included; therefore, any component can be removed and other components can be added.

Cloud Support Environment

Figure 8 illustrates a general example of a suitable cloud-supported environment 800 in which the described embodiments, methods, and techniques can be implemented. In the example environment 800, various types of services (e.g., computing services) are provided via a cloud 810. For example, the cloud 810 may include a collection of computing devices, which may be centrally located or distributed thereon, thereby providing cloud-based services to various types of users and devices via a network such as the Internet. The implementation environment 800 can be used in different ways to perform computing tasks. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on local computing devices (e.g., connected devices 830, 840, 850), while other tasks (e.g., storing data for use in subsequent processing) may be performed in the cloud 810.

In example environment 800, cloud 810 provides services to connected devices 830, 840, and 850 with various screen functionalities. Connected device 830 represents a device with a computer screen 835 (e.g., a medium-sized screen). For example, connected device 830 can be a personal computer such as a desktop computer, laptop computer, notebook computer, netbook, etc. Connected device 840 represents a device with a mobile device screen 845 (e.g., a small-sized screen). For example, connected device 840 can be a mobile phone, smartphone, personal digital assistant, tablet computer, etc. Connected device 850 represents a device with a large screen 855. For example, connected device 850 can be a television screen (e.g., a smart TV) or another device connected to a television (e.g., a set-top box or game console) or the like. One or more of connected devices 830, 840, and 850 may include touchscreen functionality. Touchscreens can accept input in various ways. For example, a capacitive touchscreen detects touch input when an object (e.g., a fingertip or stylus) twists or interrupts the current flowing on its surface. As another example, a touchscreen can use an optical sensor to detect touch input when the light beam from the optical sensor is interrupted. For some inputs detected by a touchscreen, physical contact with the screen surface is not required. Devices without screen capabilities can also be used in example environment 800. For example, cloud 810 can provide services to one or more computers without displays (e.g., server computers).

The service can be provided by the cloud 810 through service provider 820 or through other online service providers (not shown). For example, the cloud service can be customized for the screen size, display capabilities, and/or touchscreen capabilities of a specific connected device (e.g., connected devices 830, 840, 850).

In example environment 800, cloud 810 utilizes service provider 820 at least in part to provide the technologies and solutions described herein to various connected devices 830, 840, and 850. For example, service provider 820 may provide centralized solutions for various cloud-based services. Service provider 820 may manage service subscriptions for users and/or devices (e.g., connected devices 830, 840, 850 and/or their respective users).

Example Implementation

Although some operations of the disclosed methods are described in a specific order for ease of presentation, it should be understood that this descriptive approach includes rearrangement unless the specific language used in the following description requires a particular order. For example, in some cases, the operations described in order may be rearranged or performed simultaneously. Furthermore, for simplicity, the accompanying drawings may not show various ways in which the disclosed methods can be combined with other methods.

Any disclosed method can be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media and executed on a computing device (i.e., any available computing device, including smartphones or other mobile devices containing computing hardware). A computer-readable storage medium is one or more optical disc media (such as DVDs or CDs, volatile memory (such as DRAM or SRAM), or non-volatile memory (such as flash memory or hard disk drives) that can be accessed within a computing environment). As an example and referring to Figure 6, the computer-readable storage medium includes memories 620 and 625 and memory 640. As an example and referring to Figure 7, the computer-readable storage medium includes memory and memories 720, 722, and 724. The term "computer-readable storage medium" does not include signals and carrier waves. Furthermore, the term "computer-readable storage medium" does not include communication connections such as 670, 760, 762, and 764.

Any computer-executable instructions used to implement the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable storage media. The computer-executable instructions may be, for example, part of a dedicated software application or a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment using one or more networked computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or other such networks).

For clarity, only specific aspects of the software-based implementation scheme are described. Other details well-known in the art are omitted. For example, it should be understood that the disclosed techniques are not limited to any particular computer language or program. For instance, the disclosed techniques can be implemented using software written in C++, Java, Perl, or any other suitable programming language. Similarly, the disclosed techniques are not limited to any particular computer or hardware type. Specific details regarding suitable computers and hardware are well-known and do not need to be elaborated in this disclosure.

Furthermore, any software-based implementation (e.g., including computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, intranets, software applications, cable (including optical fiber), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

The disclosed methods, apparatus, and systems should not be interpreted in a limiting way. Rather, this disclosure is directed only to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations of each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor are the disclosed embodiments required to have any one or more particular advantages or problems solved.

The techniques from any example can be combined with the techniques described in any one or more of the other examples. Given the many possible embodiments to which the principles of the disclosed techniques can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed techniques and should not be considered as limiting the scope of the disclosed techniques.

Claims (20)

1. One or more computing devices, including: processor; and Memory; The one or more computing devices are configured via computer-executable instructions to perform operations for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the operations including: Receive a complete copy of an external key-value store, wherein the external key-value store includes bearer details for all current bearers of the cellular network, wherein the bearer details define network flows associated with user equipment using the cellular network; The complete copy of the external key-value store is stored in the local cache; Receive data plane network packets from the load balancer; and The virtualized packet gateway processes the data plane network packets, wherein the processing includes identifying the bearer associated with the data plane network packets in the local cache. 2. The one or more computing devices of claim 1, wherein the operation is performed during an extended event, in which the virtualized packet gateway is instantiated within the cloud computing environment. 3. The computing device according to claim 1, wherein the virtualized packet gateway is a service gateway/packet gateway for data plane network packets, the SPGW-U operating on a virtual machine running in the cloud computing environment. 4. The computing device of claim 3, wherein the SPGW-U operates as part of a telecommunications provider’s Long Term Evolution (LTE) cellular network. 5. The computing device of claim 1, further comprising: Receive new data plane network packets from the load balancer, the new data plane network packets being associated with new bearers that do not exist in the local cache; In response to receiving the new data plane network packet: Retrieve detailed bearer information for the new bearer from the external key-value store; The detailed bearer information for the new bearer is stored in the local cache; and The new data plane network packets are processed at least in part based on the bearer details for the new bearer. 6. The computing device of claim 5, further comprising: In response to receiving the new data plane network packet: The virtualized packet gateway is registered as the owner of the new bearer in the external key-value store. 7. The one or more computing devices of claim 1, wherein the load balancer directs network packets based on Internet Protocol (IP) headers and User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) port numbers, rather than based on encapsulated packet information. 8. The one or more computing devices of claim 1, wherein for each of the one or more bearers in the local cache, the operation further comprises: The local cache marks the data plane network packets for the bearer with an indication of the last time the bearer was processed. 9. The computing device of claim 8, further comprising: The bearer is cleared from the local cache based on the indication of the last time a data plane network packet was processed. 10. A method implemented by a computing device for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the method comprising: During the extended event where the virtualized packet gateway is instantiated within the cloud computing environment, the virtualized packet gateway: Receive a complete copy of an external key-value store, wherein the external key-value store includes bearer details for all current bearers of the cellular network, wherein the bearer details define network flows associated with user equipment using the cellular network; The complete copy of the external key-value store is stored in the local cache; Receive data plane network packets from the load balancer; and The virtualized packet gateway processes the data plane network packets, wherein the processing includes identifying the bearer associated with the data plane network packets in the local cache. 11. The method of claim 10, further comprising: Receive new data plane network packets from the load balancer, the new data plane network packets being associated with new bearers that do not exist in the local cache; In response to receiving the new data plane network packet: Retrieve detailed bearer information for the new bearer from the external key-value store; The detailed bearer information for the new bearer is stored in the local cache; and The new data plane network packets are processed at least in part based on the bearer details for the new bearer. 12. The method of claim 11, further comprising: In response to receiving the new data plane network packet: The virtualized packet gateway is registered as the owner of the new bearer in the external key-value store. 13. The method of claim 10, wherein the virtualized packet gateway is a service gateway/packet gateway for data plane network packets, the SPGW-U operating on a virtual machine running in the cloud computing environment. 14. The method of claim 10, wherein the load balancer directs network services based on Internet Protocol (IP) header information, rather than on encapsulated packet information. 15. A method implemented by a computing device for operating a virtualized packet gateway in a cloud computing environment to process data plane network packets for a cellular network, the method comprising: By the virtualized packet gateway: Receive a first data plane network packet associated with a first bearer; When it is determined that the first bearer is not in the local cache, the bearer details for the first bearer are retrieved from the external key-value store and stored in the local cache. The first data plane network packets are processed based at least in part on the bearer details for the first bearer stored in the local cache; Receive network communication indicating that the first bearer has been updated; In response to receiving the network communication, the updated bearer details for the first bearer are retrieved from the external key-value store and stored in the local cache. Receive a second data plane network packet associated with the first bearer; and The second data plane network packets are processed at least in part based on the updated bearer details for the first bearer stored in the local cache. 16. The method of claim 15, wherein the network communication indicating that the first bearer has been updated is received from a Mobility Management Entity (MME) or from a gateway that processes control plane network services. 17. The method of claim 15, further comprising: The virtualized packet gateway is registered as the owner of the first bearer in the external key-value store. 18. The method of claim 15, wherein the virtualized packet gateway is a Service Gateway/Packet Gateway for Data Plane Network Packets (SPGW-U), the SPGW-U operating on a virtual machine running in the cloud computing environment. 19. The method of claim 15, wherein the first data plane network packet and the second data plane network packet are received from a load balancer, wherein the load balancer guides the network packets based on Internet Protocol (IP) headers and User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) port numbers, rather than based on encapsulated packet information. 20. The method of claim 15, further comprising: Receive third data plane network packets associated with the second bearer; When it is determined that the second bearer is not in the local cache, the bearer details for the second bearer are retrieved from the external key-value store and stored in the local cache. Register the virtualized packet gateway as the owner of the second bearer in the external key-value store; and The third data plane network packets are processed at least in part based on the bearer details for the second bearer stored in the local cache.