HK1262162A1 - Method for transmitting data packet and communication router - Google Patents

Description

Method for transmitting data packet and communication router

The application is a divisional application of the invention application with the application date of 2013, 07, 05 and the application number of 201380077946.1, and is named as 'method and system for transmitting packets through centralized end-to-end connection'.

Technical Field

The present invention generally relates to the field of computer networks. More particularly, the present invention relates to a method and system for establishing a centralized end-to-end connection over a plurality of established end-to-end connections (estabilished end-to-end connections) between two communication routers, and for grouping the plurality of established end-to-end connections into groups. The data packets are sent over one or more sets of established end-to-end connections, depending on the classification and the type of the data packet.

Background

A multi-Wide Area Network (WAN) site-to-site VPN router is a router that is used to support the bandwidth of a centralized plurality of interconnects, such as WAN connections for accessing one or more remote private networks. In some embodiments, each TCP/IP session is routed to only one WAN. In this configuration, a single TCP file transfer session can only utilize the bandwidth of one WAN connection on each terminal. For example, in a session based on site-to-site Virtual Private Network (VPN) connections, VPN traffic is routed to multiple WAN connections between two sites (e.g., sites a and B).

In one embodiment, M × N tunnels are initially formed between WAN connections, where M and N are the number of WAN network connections for site a and site B, respectively. The application TCP/IP session is then routed via a different tunnel. It should be noted, however, that while a site-to-site VPN based session can utilize different tunnels for different sessions, a single download session in this type of connection can utilize only one tunnel.

To improve the throughput of a single data transfer session, routing patterns have been created to attempt to utilize multiple WAN connections for a single TCP/IP session. In some cases, channel bonding is implemented. For example, in a bonded site-to-site VPN connection on a multi-WAN router, data from a single TCP/IP session is distributed to multiple tunnels. In these arrangements, the M × N tunnels are utilized according to their respective upstream or downstream bandwidths, and the amount of data in a single session sent via these tunnels is distributed in proportion to the respective bandwidth ratios of the tunnels. However, these solutions still fail to take into account that some tunnels are not applied to transmitting data due to one or more conditions. Furthermore, these solutions also do not take into account that some network interfaces fail to meet one or more conditions, and therefore those network interfaces are not applied to transferring data.

Disclosure of Invention

Accordingly, the present invention provides an apparatus, system and method for selecting an established end-to-end connection by classifying the established end-to-end connection into a first set of established end-to-end connections and at least one non-first set of established end-to-end connections, wherein the first set of established end-to-end connections complies with all of the one or more conditions, and wherein the at least one non-first set of established end-to-end connections does not comply with all of the one or more conditions. By using multiple conditions, embodiments of the present invention are able to select an established end-to-end connection that has been eligible to transmit a data packet without using an established end-to-end connection that has been ineligible to transmit the data packet.

According to one embodiment of the invention, the router classifies the established end-to-end connections into different groups. The router then transmits different types of packets, including data packets, management packets and error correction packets, over different established end-to-end connections. According to one embodiment of the invention, the first set of established end-to-end connections meets all conditions and is used to transport data packets. The non-first set of established end-to-end connections are used to transport management packets and error correction packets.

According to one embodiment of the invention, the conditions include performance metrics, service providers, usage metrics, location, time, price metrics, security, user identification, internet protocol address range, communication protocols, communication technologies, applications, and devices. Performance metrics such as throughput, error rate, packet delay, packet jitter, symbol jitter, quality of service, security, coverage area, bandwidth, error rate, packet error rate, frame error rate, packet drop rate, queuing delay, round trip time, capacity, signal level, interference level, bandwidth delay product, handover delay time, signal-to-interference ratio, and signal-to-noise ratio are created based on one or more parameters.

According to one embodiment of the invention, when there is more than one set of non-first set of established end-to-end connections, the data packets are distributed among the non-first set of established end-to-end connections according to a hierarchy.

According to one embodiment of the invention, when there are a plurality of conditions for classifying an established end-to-end connection, weights are assigned to the plurality of conditions to indicate priorities of the conditions.

Drawings

FIG. 1A illustrates a network environment in accordance with one of many embodiments of the invention.

Fig. 1B illustrates a system adapted according to an embodiment configured to optimize throughput of a bundled plurality of variable bandwidth connections by adjusting a tunnel bandwidth weighting pattern during a data transfer session.

Fig. 2A illustrates a high-level flow chart of the operation of the system, which depicts a method for improving the throughput of a bonded connection.

Fig. 2B illustrates an embodiment in which packets are distributed based at least in part on the tunnel-modified weights after the weighting method is implemented.

Fig. 3 illustrates the types of information that may be encapsulated in transmitted IP packets in accordance with one of various embodiments.

FIG. 4 is a tree diagram illustrating how established end-to-end connections are grouped according to one of various embodiments.

Fig. 5 is a schematic block diagram of a communication router, in accordance with one of more embodiments.

Fig. 6A is a flow diagram illustrating a process for grouping established end-to-end connections into one of groups according to one of many embodiments.

Fig. 6B illustrates a process for grouping established end-to-end connections into groups based on a plurality of conditions in accordance with one of more embodiments.

Fig. 6C illustrates a process for grouping established end-to-end connections into groups based on multiple conditions, all based on the same metric, in accordance with one of the various embodiments.

Fig. 7 illustrates a hierarchy of sets of established end-to-end connections at a communications router, in accordance with one of multiple embodiments.

FIG. 8 is a flow diagram illustrating a process according to one of more embodiments.

Fig. 9 is a flow diagram illustrating a process for selecting an established end-to-end connection for transmitting a data packet in accordance with one of the various embodiments.

Fig. 10 is a diagram of a preferred embodiment according to the present invention.

Fig. 11 is a flow diagram illustrating a process for selecting a network interface for transmitting a data packet according to one of many embodiments.

Detailed Description

The following description provides preferred exemplary and exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the invention. Indeed, the preferred exemplary embodiments and the following description of the exemplary embodiments will provide those skilled in the art with a useful description of practicing the preferred exemplary embodiments of the invention. It being understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the invention as set forth in the appended claims.

Also, it is noted that the embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process terminates when the operation is completed, but the process may have additional steps not included in the figure. A process may correspond to a method, a function, a step, a subroutine, etc. When a procedure corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

Furthermore, as disclosed herein, the terms "slave storage device" and "master memory" may represent one or more devices for storing data, including Read Only Memory (ROM), Random Access Memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and/or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing or carrying instruction(s) and/or data. The machine-readable medium may be implemented by virtualization and may be a virtual machine-readable medium, including a virtual machine-readable medium in a cloud-based instance.

Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium such as a storage medium. The processing unit may perform the necessary tasks. The processing unit may be a CPU, ASIC semiconductor chip, logic unit, digital processor, analog processor, FPGA, or any processor capable of performing logical and arithmetic functions. A code segment may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc. The processing unit may be implemented by virtualization and may be a virtual processing unit, including a virtual processing unit in a cloud-based instance.

An end-to-end connection may use a connection-oriented protocol (e.g., Transmission Control Protocol (TCP)) or a connectionless protocol (e.g., User Datagram Protocol (UDP)) to transport packets. Well-known protocols for deploying end-to-end connections include layer two tunneling protocol (L2TP), Secure Shell (SSH) protocol, multiprotocol label switching (MPLS), and microsoft's point-to-point tunneling protocol (PPTP). Connections to network interfaces are in the form of fiber, ethernet, ATM, frame relay, T1/e1.ipv4, IPv6, wireless technology, Wi-Fi, WiMax, high speed packet access technology, 3GPP Long Term Evolution (LTE), etc.

The network interface may be a virtual network interface, a virtual network interface included in a cloud-based instance.

FIG. 1A illustrates a network environment in accordance with one of many embodiments of the invention. The communication router 106 establishes a centralized end-to-end connection 103 with the communication router 108. The centralized end-to-end connection 103 comprises a plurality of established end-to-end connections. A plurality of established end-to-end connections are divided into a plurality of groups. For better illustration, group 103A represents a first set of established end-to-end connections, group 103B represents a second set of established end-to-end connections, and group 103C represents all other established end-to-end connections not belonging to the first set of established end-to-end connections or the second set of established end-to-end connections. The grouping of established end-to-end connections into groups 103A, 103B, or 103C will be discussed in more detail in subsequent detail herein. Alternatively, the plurality of established end-to-end connections are divided into a plurality of groups.

According to one of many embodiments, the communication routers 106 and 108 have one or more network interfaces. The communications router 106 establishes a plurality of said established end-to-end connections via its one or more network interfaces with one or more network interfaces of the communications router 108.

In one of the embodiments, after the established end-to-end connections are grouped into groups 103A, 103B, and 103C, data packets are transmitted by the communication router 106 via the different groups of established end-to-end connections according to conditions set by a user, manufacturer, or administrator of the communication router 106.

In one of the embodiments, after the established end-to-end connections are grouped into groups 103A, 103B, and/or 103C, different types of packets are transmitted by the communication router 106 via the different groups of established end-to-end connections depending on conditions set by a user, manufacturer, or administrator of the communication router 106 and the type of packet.

Centralized end-to-end connection

Multiple established end-to-end connections are aggregated, combined, or bound together to form one aggregated end-to-end connection, such as aggregated end-to-end connection 103. Those skilled in the art will appreciate that there are various ways to centralize, combine or bind a plurality of established end-to-end connections to form one centralized end-to-end connection. A centralized end-to-end connection is treated as one end-to-end connection by the session or application that uses it. The centralized end-to-end connection can be seen as a tunnel, a virtual private network or a connected mode or a connection oriented connectionless mode. For example, the centralized end-to-end connection 103 is a TCP connection. In another example, the centralized end-to-end connection 103 is a UDP connection. In another example, the centralized end-to-end connection is a set of multiple tunnels, and each tunnel is linked between communication router 106 and communication router 108. In another example, the centralized end-to-end connection 103 is a VPN tunnel that includes a plurality of established end-to-end connections, and each established end-to-end connection links between the communications router 106 and the communications router 108.

Typically, the communication router 106 encapsulates a data packet originating from a transmitting host into at least one data packet. For ease of reading, the data packets originating from the transmitting host are referred to as raw data packets. The transmitting host may connect to the communication router 106 through a cable, wireless connection, near field connection transmission, or through other network means. When communication router 108 receives a data packet that has encapsulated the original data packet, communication router 108 decapsulates the original data packet from the data packet and then forwards the original data packet to a receiving host or node.

There are various ways to encapsulate the original data packet into a data packet sent by the communication router 106. In particular, the communication router 106 may tunnel the original data packets between its logical endpoints using known layer two and/or layer three tunneling protocols, such as layer two tunneling protocol (L2TP) and Generic Routing Encapsulation (GRE) protocol. In this case, one or more tunnel headers are prepended to the data packet to properly transport the data packet along the virtual circuit. The multiprotocol label switching (MPLS) protocol may be used as a tunneling mechanism for establishing a two-layer virtual circuit or a three-layer network-based VPN over an IP network.

The communication router 106 may use TCP, UDP, or other communication protocols as the communication protocol for the established end-to-end connection. One of the biggest challenges faced when using a converged end-to-end connection is reordering packets received within the established end-to-end connection, as the received packets may arrive out of order at the communications router 108. If the communication router 108 passes received packets to the recipient host too early, the recipient host may treat those packets that have not yet been received as missing packets, even though those packets may eventually arrive. One way to overcome this challenge is for the communication router 106 to assign a sequence number to each packet as the communication router 106 transmits the packet to the communication router 108. The communication router 108 is then able to reorder the data packets according to the sequence number.

There may be one or more centralized end-to-end connections between communications router 106 and communications router 108. Each centralized end-to-end connection has its own established end-to-end connection. For example, an end-to-end connection in a first set may have ten established end-to-end connections, while an end-to-end connection in a second set may have five established end-to-end connections. The end-to-end connection in the first set and the end-to-end connection in the second set cannot use the same established end-to-end connection.

FIG. 5 is a schematic block diagram of a communication router (e.g., communication router 106) in accordance with one of many embodiments of the present invention. The communication router 106 includes a processing unit 502, a main memory 503, a system bus 504, a slave storage device 505, and a plurality of network interfaces 506. The processing unit 502 and the main memory 503 are directly connected to each other. The system bus 504 couples the processing unit 502 directly or indirectly to a slave storage device 505 and to a plurality of network interfaces 506. The use of the system bus 504 may allow the communication router 106 to have a high degree of modularity. The system bus 504 couples the processing unit 502 to a slave storage device 505 and a plurality of network interfaces 506. The system bus 504 may be any of several types of bus structures including a memory bus, a peripheral bus, and a local bus using any of a variety of bus architectures. Program instructions executed by processing unit 502 are stored from storage device 505. The slave storage means 505 further stores conditions wherein the grouping of the established end-to-end connections into different groups depends on whether the established end-to-end connections are eligible or not.

FIG. 4 is a tree diagram illustrating how established end-to-end connections are grouped according to one of various embodiments. Established end-to-end connection 400 represents all established end-to-end connections between two network nodes (e.g., communication routers 106 and 108). Established end-to-end connections that meet the conditions in the first set are classified into a first set of established end-to-end connections 410. Established end-to-end connections that do not meet the conditions to be classified into the first set of established end-to-end connections 410 are classified into non-first set of established end-to-end connections 420. In a variant, the non-first set of established end-to-end connections 420 is further divided into a second set of established end-to-end connections 421 and a non-first/second set of established end-to-end connections 422. Established end-to-end connections that do not meet the conditions to be classified into the second set of established end-to-end connections 421 are classified into non-first/second set of established end-to-end connections 422. In a variant, the non-first set of established end-to-end connections is not further separated into a second set of established end-to-end connections 421 and a non-first/second set of established end-to-end connections 422. The second set of established end-to-end connections 421 is omitted.

To separate the established end-to-end connections into different groups, the communications router determines whether the established end-to-end connections meet certain conditions. According to one of the embodiments, the classification is performed so that certain types of packets are transmitted over established end-to-end connections belonging to a certain group and other types of packets are transmitted using established end-to-end connections in other groups. According to one of the embodiments, once the classification is completed, the groups are ranked such that the highest priority data packets are transmitted using the set of established end-to-end connections having the highest ranking.

In one of the embodiments of the invention, when there are multiple established end-to-end connections in the first set of established end-to-end connections, the data packets are distributed among and transmitted over the multiple established end-to-end connections of the first set using load balancing techniques or any other technique that allows the data packets to be distributed and transmitted among the multiple established end-to-end connections. Those skilled in the art will appreciate that there are many techniques for transmitting data packets using multiple established end-to-end connections. The same technique can also be applied to distribute packets among other groups of established end-to-end connections.

Fig. 1B illustrates a system 100 adapted according to an embodiment configured to optimize throughput of a bundled plurality of variable bandwidth connections by adjusting a tunnel bandwidth weighting pattern during a data transfer session. System 100 comprises a plurality of sites 102 and 104, each of which includes at least one communication router 106 and 108. Communication routers 106 and 108 may be implemented as multi-WAN routers that support a centralized bandwidth for multiple internet connections. The communication routers 106 and 108 are connected via a network 110. Network 110 may include a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a wireless network, the Public Switched Telephone Network (PSTN), the Internet, an intranet, an extranet, and so forth.

Site 102 and router 106 may include M connections 112 and site 104 and router 108 may include N connections 114. Connections 112 and 114 are data connections for transferring information within network 110 between sites 102 and 104. In the illustrated embodiment, M equals 3 and N equals 2; however, these values may vary depending on the desired router and configuration. Connections 112 and 114 may have similar or different bandwidth capabilities. Further, connections 112 and 114 may include different types of WAN connections, such as WiFi, cable, DSL, T1, 3G, 4G, satellite connections, and so forth. It should also be noted that station 102 and station 104 may be considered both transmitters or receivers and that discussion regarding the functionality of either station may be implemented on the other station. In other words, the system 100 may be implemented as a symmetric network.

Fig. 2A illustrates a high-level flow chart of operation of the system 100, which depicts a method 200 for improving throughput of a bonded connection. It is to be understood that the specific functions, order of functions, etc. provided in fig. 2A are intended to be exemplary operations according to the concepts of the present invention. Thus, the concepts herein may be embodied differently than in the illustrated embodiments.

At block 201 of the illustrated embodiment, when a bonded connection is established between routers 102 and 104 (e.g., by a site-to-site VPN connection enforcing the bonding), mxn virtual tunnels 116 may be created. Virtual tunnel 116 corresponds to a unique arrangement of the network connections of station 102 and the network connections of station 104.

The aggregated end-to-end connection 103 is equivalent to M × N virtual tunnels 116 because routers 106 and 108 aggregate M × N virtual tunnels 116 to form aggregated end-to-end connection 103. The M N virtual tunnels 116 are divided into groups 103A, 103B, and 103C.

At block 202 of the illustrated embodiment, a default weight is determined and/or assigned for the tunnel. To determine the default weights, embodiments exchange upstream and downstream bandwidth data of connections 112 and 114 between stations 102 and 104. Using this bandwidth data, a default weight may be calculated according to: suppose the downstream bandwidths of connections 1 to m of station 102 are d1, d2, … …, dm, and the upstream bandwidths of connections 1 to n of station 104 are u1, u1, … …, un; the default weight for the tunnel between connection X of station 102 and connection Y of station 104 may be defined as DW (X, Y), where DW (X, Y) is dx · uy.

Using the above method to calculate the default weights, if the connections 112-1 to 112-3 are WAN connections of multiple WAN routers with respective upstream/downstream bandwidths of 10M/6M, 8M/4M, and 6M/6M, and the connections 114-1 to 114-2 are WAN connections of multiple WAN routers with respective upstream/downstream bandwidths of 7M/5M and 9M/3M, then the respective default weights for each tunnel would be as follows:

TABLE 0001

For site 102	For site 104
		DW(1,1)＝6*7＝42	DW(1,1)＝5*10＝50
DW(1,2)＝6*9＝54	DW(1,2)＝5*8＝40
		DW(2,1)＝4*7＝28	DW(1,3)＝5*6＝30
DW(2,2)＝4*9＝36	DW(2,1)＝3*10＝30
		DW(3,1)＝6*7＝42	DW(2,2)＝3*8＝24
DW(3,2)＝6*9＝54	DW(2,3)＝3*6＝18

It should be noted that other ways of calculating the default weights are conceivable, and the above are only examples of implementations of embodiments of the invention. It should be noted that many different weighting patterns may be used to define the initial bandwidth of the tunnel. For example, it may be desirable to weight a tunnel in one direction using only the downstream capacity of the receiving station and the upstream capacity of the transmitting station. Any weighting pattern used to characterize the tunnel capacity at the mechanism of the bonded connection may be used for purposes of the present invention.

When a packet is routed from station 102 to station 104 according to an embodiment, the packet will be distributed to the tunnel at a rate according to the effective weight EW (x, y). Initially, the effective weight of an embodiment is set equal to the default weight, EW (x, y) ═ DW (x, y), and if the bandwidth of tunnel 116 remains unchanged from the initial setting, then for packet distribution, the effective weight is optimal for packet distribution. However, if a user is downloading a file via a bonded network connection in a TCP session with one or more tunnels with dropped packets, the overall throughput of the session will drop dramatically. This is partly because dropped packets will continue to cause TCP retransmissions and the traffic control of TCP will maintain a lower throughput even if the tunnel without dropped packets is not fully occupied.

An effective way to improve throughput may be to avoid such packet losses. To do so, at block 203 of the illustrated embodiment, embodiments of the present invention discern when a tunnel is experiencing an increase or decrease in packet loss rate. At block 204, embodiments further provide for modifying the effective weights of tunnels that have experienced or have experienced a change in packet loss rate. The packet loss rate information may be continuously monitored or monitored based on a specific time period. Upon determining that the tunnel is experiencing an unacceptable packet loss rate (block 204-1), the illustrated embodiment reduces the effective weight of the tunnel at block 204-2. In some embodiments, unacceptable may mean that the packet loss rate is a non-zero number, while other embodiments may determine that the unacceptable rate is any rate other than a predefined threshold. Embodiments implement these reductions in a stepwise manner, in a continuous manner, in a reduction proportional to a doubling of the packet loss rate, etc. When the reduction is performed in a gradual manner, embodiments may continue to monitor the tunnel to optimize the amount of reduction implemented.

Tunnel 116 may be established or monitored by sending heartbeat packets through each tunnel of router 106 or router 108. In some embodiments, when a receiving terminal fails to receive a heartbeat packet from a tunnel for a period of time, it will treat the tunnel as downstream and the tunnel will not be used to route traffic. If the heartbeat packet starts to be received again, the tunnel may be re-established and weighted along with other tunnels. Thus, embodiments may monitor and re-establish connections using heartbeat packets in the event that all packets are dropped in a tunnel and the effective weight of the tunnel is reduced to zero.

Further, when a tunnel recovers all or a portion of its respective bandwidth, e.g., determines that the packet loss rate is decreasing (block 204-3), the illustrated embodiment operates to increase the effective weight of such tunnel (block 204-4) in order to fully or more fully utilize the bandwidth. Some embodiments increase the effective weight of the tunnel using a predetermined step size until the exact effective weight is restored. Other embodiments increase the effective weight in proportion to the latest measured bandwidth, which may correspond to the latest measured packet loss rate. Further, embodiments may increase the effective weight of the tunnel based on a predetermined linear or exponential ratio.

After the effective weights of the tunnels are adjusted, or after it is determined that no adjustment is needed, the weighting pattern of the system is updated at block 205 of the illustrated embodiment. This updating may include storing any processed information, using such information in further processing, rendering the system inoperative, etc. For example, processing with respect to block 205 may be used to average the weighted patterns over a period of time in order to mitigate errors associated with high transient anomalies. Further, the updated information may be used on the system 100 to modify the packet distribution of the data transfer session, as discussed with respect to fig. 2B. System 100 may continue to implement steps 203 through 205 continuously or periodically throughout the data transfer session.

Fig. 2B illustrates an embodiment in which the modified weight distribution packet is based at least in part on a tunnel after implementing the weighting method 200. In particular, block 206 of the illustrated embodiment is used to distribute packets across tunnels according to a weighting pattern determined by the operation of method 200. In some embodiments, this distribution will change throughout the data transfer session, so the steps of fig. 2B are repeated. Some embodiments change the packet distribution each time the system is updated at block 205. Further, block 205 may cause the change to be implemented periodically in response to a threshold or the like for certain drop rate changes. It should be appreciated that the determination of the weights by the operation of method 200 and the application of the determined weights to the packet distribution at block 206 may have different periodicities. For example, the method 200 may be used to provide updates of weighting pattern information using a relatively short iteration loop, while the distribution of packets is varied based on such weighting pattern information using a longer iteration loop.

To monitor the bandwidth of the various tunnels 116, some embodiments of the present invention encapsulate each transmitted IP packet with various information. Fig. 3 illustrates an example embodiment showing an example embodiment of the type of information 300 that may be encapsulated in a transmitted IP packet. The version field 302 may contain information about the protocol version utilized and the protocol type field 303 may contain the protocol type of the packet for the payload. In general, the value of this field will correspond to the ethernet protocol type of the packet. However, additional values may be defined in other documents. The tunnel ID field 304 may be a 32-bit field and may contain an identifier to identify the current tunnel of the IP packet. Advanced Encryption Standard (AES) initialization vector field 306 may be a 32-bit field and may contain an AES encrypted initialization vector. The global sequence number field 308 may be a 32-bit field and may contain a sequence number that is used to reorder each of the packets for the respective sessions into an appropriate order when they have emerged from their respective tunnels. The sequence number per tunnel field 310 may be a 32-bit field that may represent the sequence number assigned to each packet routed to a particular tunnel. AES encrypted payload field 312 may be used to convey the payload of an IP packet.

The per-tunnel sequence number discussed above may be used to monitor dropped packets in the tunnel. In one embodiment, the router on the receiving terminal calculates the dropped packet rate DR (x, y) per tunnel per f seconds by monitoring the per tunnel sequence number of the received packets. DR (x, y) can be characterized as the number of missing sequence numbers divided by the number of sequence number increments for one period f. The length of period f may vary, and in one embodiment, f equals 5 seconds.

Other methods may also be used to monitor packet loss, such as: the transmitter may periodically inform the receiving terminal how many packets it has transmitted; a transmitter transmits heartbeat packets to a receiving terminal at intervals of a fixed time period; and the receiving terminal may evaluate the overall drop rate by monitoring the drop rate of heartbeat packets, by retrieving a drop rate map from the physical interface/device/layer, etc.

The receiving terminal may feed back to the sending router the drop rate, effective weight, or other bandwidth indicator for a particular tunnel. When the transmitter receives information about packet loss, some embodiments reduce the effective weight EW (x, y) of the tunnel by EW (x, y) · DR (x, y). Other metrics EW (x, y) · DR (x, y) may be used to reduce the effective weight EW (x, y) of the tunnel. Other metric values may also be used to modify the effective weights of the tunnels. In some embodiments, the transmitter may receive feedback and the effective weight may be reduced by a value greater or less than the packet loss rate. Such varying variables may be configured according to the specific needs of the communication system. The above example represents a metric that attempts to reduce the effective weight of the tunnel to a weight to prevent further drop packet loss while maximizing the metric of the weight of the amount of available bandwidth for the tunnel. Any measure of this balance may be found to be preferred.

Fig. 6A is a flow diagram illustrating a process for separating established end-to-end connections into one of a plurality of groups in accordance with one of the various embodiments of the invention. For a better understanding of the embodiments, FIG. 6A is viewed in conjunction with FIG. 5. In step 601, the processing unit 502 retrieves the condition from the storage 505. The classification is based on whether the established end-to-end connection meets the retrieved condition.

In step 602, processing unit 502 determines whether the established end-to-end connection between communication router 106 and communication router 108 is eligible.

If in step 602 the processing unit 502 determines that the established end-to-end connections meet all conditions, then in step 603 the established end-to-end connections belong to a first set of established end-to-end connections.

If in step 602 the processing unit 502 determines that the established end-to-end connection fulfils at least one condition but not all conditions, then in step 604 the established end-to-end connection belongs to a second set of established end-to-end connections.

If in step 602 the processing unit 502 determines that the established end-to-end connection meets one or more specific conditions, but not all, then in step 604 the established end-to-end connection belongs to a second set of established end-to-end connections.

In a variant, if in step 602 the processing unit 502 determines that the established end-to-end connection does not comply with any condition, then in step 605 the established end-to-end connection belongs to a non-first/second set of established end-to-end connections. An established end-to-end connection belonging to a non-first/second group does not belong to either the first or second group of established end-to-end connections.

In a variant, there is no second set of established end-to-end connections, omitting step 604. If the processing unit 502 determines in step 602 that the established end-to-end connection does not meet all conditions, the established end-to-end connection belongs to the non-first/second group in step 605.

In one variation, only one condition is retrieved from the secondary storage 505 in step 601. If in step 602 the processing unit 502 determines that the established end-to-end connection fulfils said condition, the established end-to-end connection belongs to a first set of established end-to-end connections in step 603. If in step 602 the processing unit 502 determines that the established end-to-end connection does not comply with the condition, then in step 605 the established end-to-end connection belongs to a non-first/second set of established end-to-end connections. Step 604 is omitted because there is only one condition.

When the condition relates to a performance metric, the determination in step 602 is made by sending test data to the communications router 108 using the established end-to-end connection. The content of the test data may be based at least in part on the content of the data packet, randomly generated content, or one or more benchmarks. The time period for sending test data is preferably less than ten seconds, since the number of test data sent in ten seconds is sufficient to determine performance in most networks, such as round trip time, latency and packet loss.

Alternatively, instead of using test data, the determination in step 602 is done by sending a data packet, an error correction/detection packet, a management packet, or a health check packet. The health check package is sent for maintaining the established end-to-end connection and in order to check the status of the established end-to-end connection. When the data packet is transmitted via each established end-to-end connection for the determination in step 602, after the classification is completed, the data packet is transmitted only via a specific set of established end-to-end connections. In one of the embodiments, a performance report is received from the communications router 108 and used to determine whether the established end-to-end connection meets performance related conditions. The test data may be transmitted sequentially, one by one, in groups, synchronously or almost synchronously over a plurality of established end-to-end connections. In one variation, the determination in step 602 is based at least in part on information stored in the secondary storage 505 or analysis content of the information. Alternatively, the determination in step 602 is accomplished by using historical data or historical performance data. The history data is stored in the slave storage 505. An example of a scenario in which historical data is used in step 602 to determine whether an established end-to-end connection meets a condition is when the condition is based on a usage metric criterion. For example, an established end-to-end connection is eligible if its usage has not reached a certain limit, and is ineligible if its usage has reached or is close to a certain limit. It is therefore necessary to monitor the usage and store the data of the usage as history data in the slave storage means 505.

In one variation, the test data is sent using OSI two-layer packets via point-to-point protocol (PPP), frame relay protocol, Address Resolution Protocol (ARP), or any other data link layer protocol. Alternatively, the test data is transmitted using Internet Protocol (IP) packets through Transmission Control Protocol (TCP), User Datagram Protocol (UDP) or Internet Control Message Protocol (ICMP). If the established end-to-end connection uses the LTE protocol, then test data is sent using LTE type 1 or type 2 frames. Alternatively, the test data is sent using ethernet frames. Alternatively, the format of the test data is based in part on the retrieved conditions and the communication protocol of the established end-to-end connection.

In one of the embodiments, a score for an established end-to-end connection is calculated based on the compliance. An established end-to-end connection must meet a minimum number of conditions in order to belong to a set of established end-to-end connections. The minimum number may be predefined by a user, manufacturer, or administrator of the communication router 106. For example, there are five conditions, the minimum number of conditions that an established end-to-end connection needs to meet to belong to the first set of established end-to-end connections is three. The established end-to-end connection belongs to a first set of established end-to-end connections if the processing unit 502 determines that the established end-to-end connection fulfils at least three conditions. If the processing unit 502 determines that the established end-to-end connection does not comply with the at least three conditions, the established end-to-end connection belongs to a non-first set of established end-to-end connections. In one variant, the minimum number of conditions that the established end-to-end connection needs to meet to belong to the second set of established end-to-end connections is two. If the processing unit 502 determines that the established end-to-end connection does not comply with the at least three conditions, but that the established end-to-end connection complies with the at least two conditions, the established end-to-end connection belongs to a second set of established end-to-end connections. There are a number of techniques to apply the combination of conditions that an established end-to-end connection needs to meet to belong to a certain set. The combination may be configured by a user, manufacturer, or network administrator of the communication router 106 according to their preferences. It is beneficial for the user to be able to configure different combinations of conditions, since an established end-to-end connection of different characteristics may be required in different situations.

Alternatively, the conditions are assigned weights in order to calculate a score for the established end-to-end connection. The grouping of the established end-to-end connections into groups is done according to the scores. For example, there are three conditions, i.e., a first condition, a second condition, and a third condition. The weights assigned to the first, second, and third conditions are fifty, thirty, and twenty, respectively. If the established end-to-end connection only meets the first condition, its score is fifty. If the established end-to-end connection only meets the second condition, its score is thirty. If the established end-to-end connection only meets the third condition, its score is twenty. If the established end-to-end connection meets the first and second conditions, its score is eighty. If the established end-to-end connection meets the first and third conditions, its score is seventy. If the established end-to-end connection meets the second and third conditions, its score is fifty. If the established end-to-end connection meets all three conditions, its score is one hundred. The minimum score required to belong to the first set of established end-to-end connections is fifty. Thus, if the processing unit 502 determines that the score of the established end-to-end connection is at least fifty according to the weights, the established end-to-end connection belongs to the first set of established end-to-end connections. The weights are used to emphasize certain conditions.

In one variant, a plurality of minimum scores are configured for grouping the established end-to-end connections into a plurality of groups, the scores of the established end-to-end connections being calculated according to the weights of the conditions to which they are accorded. Each group is assigned a corresponding minimum score, wherein the score of an established end-to-end connection must exceed or be equal to the corresponding minimum score assigned to a group in order to belong to said group. For example, seventy, sixty, fifty, and forty are assigned to the first group, the second group, the third group, and the fourth group, respectively, as respective corresponding minimum scores. If an established end-to-end connection has a score of seventy or more, it belongs to the first group. Similarly, if an established end-to-end connection has a score between sixty and seventy, between fifty and sixty, or between forty and fifty, it belongs to the second, third, or fourth group, respectively.

When the compliance of some specific conditions is more important than the compliance of other conditions, the user can assign a higher weight to the specific conditions. Higher priority is given to more important conditions by assigning them a higher weight, while other conditions that are less important are not completely ignored.

In one variation, when there are multiple established end-to-end connections and a classification is made for one of the established end-to-end connections, the communications router 106 does not use the other established end-to-end connections to transfer data. This may reduce interference with other established end-to-end connections and thus make the classification more accurate and efficient. Alternatively, when classifying for one of the established end-to-end connections, the communication router 106 continues to transmit data packets using the other established end-to-end connection. Therefore, transmission of the data packet is not stopped until the classification is completed, and thus the communication router 106 always participates in transmitting the data packet. This is advantageous because it can ensure that there is no interruption in the transmission of data packets each time a classification is made.

In one variant, after classifying the established end-to-end connection, it is reclassified periodically or upon the occurrence of a certain event. An established end-to-end connection from one group may be classified into another group if it has become more or less conditional. This allows the communications router 106 to select the appropriate established end-to-end connection when the network environment changes. This is especially more important for established end-to-end connections carried over wireless technologies, as changes in the physical environment may also change the network performance. In particular, the period of time to determine whether an established end-to-end connection must be moved to another group or can remain in that group is in the range of two seconds to two minutes. The time period for an established end-to-end connection using a wired connection is preferably longer than the time period for an established end-to-end connection using a wireless connection, as established end-to-end connections using wireless connections typically experience more variation. The more often the reclassification is performed, the earlier the communication router 106 can move an established end-to-end connection from one group to another when a change in compliance is detected. The less frequently the reclassification is performed, the less amount of test packets need to be transmitted.

For illustration, an established end-to-end connection in the second group can belong to the first group when it meets the conditions for belonging to the first group. Similarly, an established end-to-end connection in the first group is removed from the first group when the established end-to-end connection fails or ceases to meet the conditions for belonging to the first group. Then, if the established end-to-end connection meets the conditions for belonging to the second group, the established end-to-end connection changes from belonging to the first group of established end-to-end connections to belonging to the second group of established end-to-end connections. If said established end-to-end connection does not comply with the conditions for belonging to the second group, said established end-to-end connection changes from belonging to the first group of established end-to-end connections to belonging to the non first/second group of established end-to-end connections. Similarly, an established end-to-end connection in a non-first/second group becomes belonging to the first group when it meets the conditions belonging to the first group. Similarly, an established end-to-end connection in a non-first/second group becomes belonging to the second group when it meets the conditions for belonging to the second group but does not meet the conditions for belonging to the first group. Similarly, if an established end-to-end connection in the second group fails or stops meeting the conditions required for belonging to the second group, it becomes belonging to the non-first/second group. Thus, the established end-to-end connection belonging to the second group or to the non-first/second group performs a backup or failover (failover) function.

Fig. 6B illustrates a process for grouping established end-to-end connections into groups based on a plurality of conditions, all based on the same metric, in accordance with one of the various embodiments. For example, the plurality of conditions are based on packet latency, which is a performance metric. The plurality of conditions are retrieved from the storage means 505 by the processing unit 502. For better illustration, the plurality of conditions includes a first condition, a second condition, a third condition, a fourth condition, and a fifth condition. The first, second, third, fourth and fifth conditions are met if the packet delay is less than 10ms, 20ms, 30ms, 40ms and 50ms, respectively. The plurality of groups includes a first group, a second group, a third group, a fourth group, a fifth group, and an unsatisfactory group. In step 611 the processing unit 502 determines whether the established end-to-end connection fulfils a fifth condition, more precisely whether the packet delay is less than 50 ms. If the established end-to-end connection does not comply with the fifth condition, more precisely if the packet delay is not less than 50ms, the established end-to-end connection belongs to the non-compliant group in step 621. If the established end-to-end connection complies with the fifth condition, more precisely if the packet delay is less than 50ms, the processing unit 502 determines in step 612 whether the established end-to-end connection complies with the fourth condition, more precisely if the packet delay is less than 40 ms. If the established end-to-end connection does not fit the fourth condition, more precisely if the packet delay is not less than 40ms, then the established end-to-end connection belongs to the fifth group in step 622. If the established end-to-end connection fulfils the fourth condition, more precisely if the packet delay is less than 40ms, the processing unit 502 determines in step 613 whether the established end-to-end connection fulfils the third condition, more precisely if the packet delay is less than 30 ms. If the established end-to-end connection does not meet the third condition, more precisely if the packet delay is not less than 30ms, the established end-to-end connection belongs to the fourth group in step 623. If the established end-to-end connection meets the third condition, more precisely if the packet delay is less than 30ms, the processing unit 502 determines in step 614 whether the established end-to-end connection meets the second condition, more precisely if the packet delay is less than 20 ms. If the established end-to-end connection does not comply with the second condition, more precisely if the packet delay is not less than 20ms, the established end-to-end connection belongs to the third group in step 624. If the established end-to-end connection meets the second condition, more precisely if the packet delay is less than 20ms, the processing unit 502 determines in step 615 whether the established end-to-end connection meets the first condition, more precisely if the packet delay is less than 10 ms. If the established end-to-end connection does not meet the first condition, more precisely if the packet delay is not less than 10ms, then the established end-to-end connection belongs to the second group in step 625. If the established end-to-end connection meets the first condition, more precisely if the packet delay is less than 10ms, the established end-to-end connection belongs to the first group in step 626.

Alternatively, the process of fig. 6B may be implemented using a number of metrics based on conditions other than packet latency, such as other performance metrics, usage prices, IP address ranges, or any other metric that may be used to implement a number of thresholds.

The steps of fig. 6B may be performed sequentially, synchronously, or nearly synchronously. The order of the steps may be different from that illustrated in fig. 6B.

Fig. 6C illustrates a process of grouping established end-to-end connections into multiple groups based on multiple conditions in accordance with one of multiple embodiments. The plurality of conditions are stored in the slave storage device 505. For purposes of illustration, the plurality of conditions includes a first condition, a second condition, a third condition, a fourth condition, and a fifth condition. The plurality of established end-to-end connections are divided into a plurality of groups in step 630. If the processing unit 502 determines that the established end-to-end connection complies with the first condition, the established end-to-end connection belongs to the first group 631. If processing unit 502 determines that the established end-to-end connection complies with the second condition, then the established end-to-end connection belongs to the second group 632. If the processing unit 502 determines that the established end-to-end connection complies with the third condition, the established end-to-end connection belongs to the third group 633. If the processing unit 502 determines that the established end-to-end connection conforms to the fourth condition, then the established end-to-end connection belongs to the fourth group 634. If processing unit 502 determines that the established end-to-end connection conforms to the fifth condition, then the established end-to-end connection belongs to a fifth group 635.

In one variation, the plurality of conditions are based on a plurality of metrics, criteria, and/or parameters. For example, a first condition is based on location, a second condition is based on time, a third condition is based on user identity, a fourth condition is based on security and a fifth condition is based on communication protocol. The group to which an established end-to-end connection may belong may be more than one if it meets more than one condition. Alternatively, the multiple conditions are based on the same metric, criterion, or parameter but multiple thresholds. For example, the plurality of conditions are based on throughput, which is a performance metric. The first condition is fulfilled if a throughput value of the established end-to-end connection is within a first range. Similarly, the second, third, fourth or fifth condition is fulfilled if the throughput value of the established end-to-end connection is within the second, third, fourth and fifth range, respectively. More precisely, the first, second, third, fourth and fifth ranges are all different. In another alternative, the process of fig. 6C may be implemented using a number of conditions based on parts other than throughput, such as other performance metrics, usage prices, IP address ranges, or any other metric, criterion, or parameter that may be used to implement a number of numerical thresholds.

In one of the embodiments, the established end-to-end connections are divided into a plurality of groups based on a combination of conditions met by the established end-to-end connections. For example, established end-to-end connections meeting the first and second conditions belong to a group. The established end-to-end connections meeting the first, second and third conditions belong to another group. Similarly, end-to-end connections meeting other combinations of conditions belong to other groups. Where it is desired to transmit packets of a particular data type using a particular established end-to-end connection that meets a particular combination of conditions, it is advantageous to have multiple sets of established end-to-end connections, where various techniques for packet inspection (including deep packet inspection) can be used to detect packets of a particular data type.

In one of the embodiments, established end-to-end connections are grouped into a certain group based on conditions that the established end-to-end connections are not in compliance. For example, the condition is based on a user identity specifying the first user. Upon determining that the user for which the end-to-end connection has been established is not the first user, the established end-to-end connection belongs to the first group. Upon determining that the user for which the end-to-end connection has been established is the first user, the established end-to-end connection does not belong to the first group.

In one of the embodiments, after the plurality of established end-to-end connections are grouped by the communications router 106, the established end-to-end connections that are not used to transport the data packets will be maintained and not broken. In some communication protocols, an established end-to-end connection is broken if it is not used to transmit packets for some period of time. To avoid this, maintenance, health check or management packets are transmitted through the established end-to-end connection to maintain the established end-to-end connection and to maintain the state in which the established end-to-end connection is established. This ensures that whenever a data packet needs to be transmitted by the communication router using the established end-to-end connection, it is available for use without experiencing slowness.

Description of the conditions:

in one of the embodiments, the established end-to-end connections are divided into different groups based on whether the established end-to-end connections meet certain conditions. The condition is selected from the group consisting of: performance metrics, service providers, locations, time, usage price, security, user identification, Internet Protocol (IP) address range, communication protocols, communication technologies, applications, and devices. The performance metrics may be based on one or more of throughput, error rate, packet delay, packet jitter, symbol jitter, quality of service, bandwidth, error rate, packet error rate, frame error rate, packet drop rate, queuing delay, round trip time, capacity, signal level, interference level, bandwidth delay product, handover delay time, signal-to-interference ratio, and signal-to-noise ratio.

The service provider:

in one of the embodiments, the condition is based at least in part on a service provider that has established the end-to-end connection. For example, service provider a provides more reliable service than service provider B. Then the conditions may be such that: if the established end-to-end connection is from service provider a, it is eligible. Due to reliability, cost, or performance, a user may prefer service provider a over other service providers.

Using the metric:

in one of the embodiments, the condition is based at least in part on a usage metric for a usage limit of the established end-to-end connection. For example, an established end-to-end connection is eligible if its usage has not reached a usage limit, and is ineligible if its usage has reached or is close to the usage limit. Most service providers charge very low prices per gigabit as long as usage is below a certain threshold, and as usage becomes above the certain threshold, the price per gigabit increases. Thus, by making the usage limit equal to the certain threshold, the cost of using the established end-to-end connection is kept within a budget.

Position:

in one of many embodiments, the condition is based at least in part on location. For example, when the communication router 106 is in a certain location, it may be preferable to use some established end-to-end connections over others. These more preferred end-to-end connections are eligible. Having location-based conditions is advantageous because some established end-to-end connections may be more preferable in certain locations, primarily due to the higher signal strength of the established end-to-end connections in those certain locations. In another example, when the communication router 106 is in a certain location using a first established end-to-end connection utilizing LTE protocols, and a second established end-to-end connection utilizing a Wi-Fi connection becomes available in the certain location, based on the location, the communication router 106 can switch from the first established end-to-end connection to the second established end-to-end connection and perform network offloading. Thus, in some positions, the first established end-to-end connection is not eligible and the second established end-to-end connection is eligible.

Time:

in one of many embodiments, the condition is based at least in part on time. For example, some established end-to-end connections may be preferred over other established end-to-end connections at a certain time of day. These more preferred end-to-end connections are eligible. This embodiment takes advantage of the differences in cost, performance and reliability of the established end-to-end connection during peak and off-peak periods.

The price of use:

in one of many embodiments, the condition is based at least in part on a price of use. An established end-to-end connection is not eligible if the price for using the established end-to-end connection exceeds the price limit of the user. The established end-to-end connection is eligible if the price for using the established end-to-end connection is equal to or less than the price limit of the user. Setting a price limit is beneficial for those users who have a cost budget for using an established end-to-end connection. For example, the price limit for a user is $10 per gigabit. Thus, if the price of using an established end-to-end connection exceeds $10 per gigabit, the established end-to-end connection is not eligible, and if the price of using an established end-to-end connection is less than or equal to $10 per gigabit, the established end-to-end connection is eligible.

Safety:

in one of many embodiments, the condition is based at least in part on security. An established end-to-end connection is not eligible if it is not secure enough or if the security protocol used in the established end-to-end connection is not a user preferred security protocol. It is of utmost importance that the established end-to-end connection be secure in order to obtain the authenticity, integrity and confidentiality of data packets transmitted over the established end-to-end connection. The integrity of the data packet can maintain and ensure the accuracy and consistency of the information provided in the data packet. Confidentiality of the data packet is also required in order to prevent information provided in the data packet from being leaked to an unauthorized individual or system. For example, the condition is that the established end-to-end connection must use the encryption standard AES 192. If the established end-to-end connection is encrypted using AES 128, or using any encryption standard other than AES 192, then the established end-to-end connection is not eligible. If the established end-to-end connection is encrypted using AES 192, then the established end-to-end connection is eligible. Alternatively, if the encryption standard used for the established end-to-end connection provides a higher security metric than AES 192, e.g., AES 256, then the established end-to-end connection is also eligible.

And (3) user identity identification:

in one of many embodiments, the condition is based at least in part on a user identity. For example, using an established end-to-end connection requires user authentication. Some established end-to-end connections may be reserved for some users based on the identity of the user. An established end-to-end connection is eligible if it is reserved for a user using the communications router 106. It is advantageous to have a condition based on the identity of the user in a communication router having more than one user, each of which may have different preferences for using the established end-to-end connection.

IP address range:

in one of the embodiments, the condition is based at least in part on an IP address range. For example, a range of IP addresses is specified in the condition, such that an established end-to-end connection is not eligible if it is between the communication router 106 and a network interface whose IP address does not fall within the IP address range. The established end-to-end connection is eligible if it is between the communication router 106 and a network interface whose IP address belongs to the IP address range.

The communication protocol is as follows:

in one of many embodiments, the condition is based at least in part on a communication protocol. The communication protocols include PPP, frame relay protocol, ARP and IP, e.g., TCP, UDP ICMP. For example, a user may want to transmit a certain type of data packet over a particular communication protocol using an established end-to-end connection. The established end-to-end connection is eligible if the established end-to-end connection uses the particular communication protocol. If the established end-to-end connection does not use the particular communication protocol, the established end-to-end connection is not eligible. For example, the user may want to use the established end-to-end connection for transmitting PPP packets. The established end-to-end connection is eligible if the established end-to-end connection uses PPP. If the established end-to-end connection does not use PPP, the established end-to-end connection is not eligible.

The communication technology comprises the following steps:

in one of many embodiments, the condition is based at least in part on the communication technology. Communication technologies include wireless technologies, Wi-Fi, WiMax, high speed packet access technologies, 3GPP Long Term Evolution (LTE), and so on. For example, a user may want to transmit a certain type of data packet over a particular communication technology using an established end-to-end connection. The established end-to-end connection is eligible if the established end-to-end connection uses the particular communication technology. The established end-to-end connection is not eligible if the established end-to-end connection does not use the particular communication technology. For illustration purposes, a user wants to transmit data packets over 3GPP LTE technology using an established end-to-end connection. Thus, if the established end-to-end connection uses 3GPP LTE, the established end-to-end connection is eligible. Setting conditions based on communication technologies is advantageous because different characteristics of different communication technologies, some specific characteristics of certain communication technologies, help to achieve the objective of using an established end-to-end connection.

The application program comprises the following steps:

in one of many embodiments, the condition is based at least in part on the application. For example, a user wants to transmit a data packet to a certain application using an established end-to-end connection. An established end-to-end connection is eligible if it is suitable for use in the certain application. Alternatively, when an established end-to-end connection is known to be suitable for a certain application, the established end-to-end connection is eligible as long as the user wants to transmit data packets to the certain application using the established end-to-end connection.

The device comprises the following steps:

in one of many embodiments, the condition is based at least in part on the device. For example, a user wants to use such an established end-to-end connection: formed between network interfaces using a particular type of modem of a particular model. An established end-to-end connection is eligible if it is formed between network interfaces using a particular type of modem of a particular model. This is advantageous in case the performance, price or reliability of the specific type of modem of the specific model is better and more suitable for transmitting the type of data packets the user wants to transmit over the established end-to-end connection. For example, a user wants to use an established end-to-end connection established through a wilson (Verizon) wireless 4G LTE USB modem 551L. The established end-to-end connection is eligible if it connects the communications router 106 to a wilson (Verizon) wireless 4G LTE USB modem 551L. In another example, a user wants to use an established end-to-end connection with a Subscriber Identity Module (SIM) card that is subscribed to a particular service provider or has a particular service plan. The established end-to-end connection is eligible if it uses a SIM card from the particular service provider or with the particular service plan. This is advantageous in terms of the cost and reliability associated with the SIM card.

Group classification:

in one of the embodiments of the present invention, when the established end-to-end connections are divided into different groups (e.g., a first group, a second group, and a non-first/second group), the different groups are assigned a rank to prioritize the use of established end-to-end connection transport packets in the group having the higher rank. This prioritization is important so that packets are sent using an established end-to-end connection with a higher rank and best characteristics according to the user's preferences. Other types of packets, such as management packets, error correction packets and all non-data packets, are transported using the established end-to-end connection with the lower level. Thus, established end-to-end connections with higher levels are reserved for data packets and their capacity is not filled by other types of packets. Alternatively, an established end-to-end connection with a higher level is also used to transport other types of packets, since some administrators may think that the management packets and error correction packets should have the same or even higher priority than the data packets. A number of established end-to-end connections belonging to a specific group have the same class. In one variant, the first set of established end-to-end connections is ranked highest. The level of the non-first set of established end-to-end connections is lower than the level of the first set of established end-to-end connections. If there is a second set of established end-to-end connections, the level of the second set of established end-to-end connections is the second highest. The level of the non-first/second set of established end-to-end connections is lowest.

Alternatively, there are multiple sets of established end-to-end connections, wherein the multiple sets of established end-to-end connections are created based on conditions met or not met by the established end-to-end connections. Fig. 6B and 6C show in particular how established end-to-end connections may be divided into groups.

In one variation, the communication router 106 transmits the data packet to the communication router 108 using only the first set of established end-to-end connections. The data packets are transmitted later when the capacity of the first set of established end-to-end connections is full. The communication router 106 waits until the first set of established end-to-end connections has sufficient capacity and then transmits the data packets to the communication router 108 using the first set of established end-to-end connections. This may be done for a variety of reasons. For example, a first set of established end-to-end connections is more reliable and faster than a second set or non-first/second set of established end-to-end connections. Thus, due to the reliability and speed of the first set of established end-to-end connections, the user may choose to always use the first set of established end-to-end connections to transmit data packets.

Alternatively, the communication router 106 transmits the data packet to the communication router 108 using only the first set of established end-to-end connections. When the capacity of the first set of established end-to-end connections is full, the communication router 106 transmits the data packet to the communication router 108 using a non-first set of established end-to-end connections.

Alternatively, the communication router 106 transmits the data packet to the communication router 108 using only the first set of established end-to-end connections. When the capacity of the first set of established end-to-end connections is full, the communication router 106 transmits the data packets to the communication router 108 using the second set of established end-to-end connections. When the capacity of the second set of established end-to-end connections is also full, no data packets are transmitted until the capacity of the first or second set of established end-to-end connections is available. The communication router 106 waits until either the first set or the second set of established end-to-end connections have sufficient capacity and then transmits the data packet to the communication router 108 using the first set of established end-to-end connections. It should be noted that the second set of established end-to-end connections is only used when the capacity of the first set of established end-to-end connections is still full.

Alternatively, when the capacity of the first set of established end-to-end connections is full, the communication router 106 transmits the data packet to the communication router 108 using the second set of established end-to-end connections. When the capacity of the second set of established end-to-end connections is also full, the communication router 106 transmits the data packets to the communication router 108 using the non-first/second set of established end-to-end connections. The non-first/second set of established end-to-end connections includes the non-first set of established end-to-end connections and the non-second set of established end-to-end connections.

In one of the embodiments, the established end-to-end connections in the second group are ranked according to the conditions to which they are eligible. The first set of established end-to-end connections fulfils all conditions and the non-first/second set of established end-to-end connections do not fulfil any conditions. However, the second set of established end-to-end connections meets some but not all conditions. Thus, the second group comprises established end-to-end connections having some characteristics, wherein the respective established end-to-end connections may or may not be in compliance for the same conditions. Since the established end-to-end connections in the second group have different characteristics, it is advantageous to assign a class to each established end-to-end connection, so that packets can be transmitted using established end-to-end connections with higher classes according to the preferences of the user.

Fig. 7 illustrates a hierarchy of sets of established end-to-end connections at a communications router according to one of the embodiments. The hierarchy is stored in a slave storage 505 and implemented by the processing unit 502. The ranking table 700 lists a number of groups and their corresponding ranks. In one variation, hierarchy 720 is considered. According to the ranking 720, the first group 711, the second group 712, the third group 713, the fourth group 714, and the fifth group 715 are assigned a first rank, a second rank, a third rank, a fourth rank, and a fifth rank, respectively. The hierarchy may be configured by a user of the communication router or a network administrator. Thus, in another variation, when considering the ranking 730, the first 711, second 712, third 713, fourth 714, and fifth 715 groups are assigned a second rank, a third rank, a first rank, a fourth rank, and a fifth rank, respectively. It will be appreciated by those skilled in the art that in order to prioritize the use of established end-to-end connection transport packets in groups having a higher rank, it is advantageous to assign a rank to each group of established end-to-end connections. It should also be appreciated that different users may have different preferences when assigning ratings to groups, and thus, two types of ratings, rating 720 and rating 730, are shown in the rating table 700 for illustrative purposes only.

Transmission of

According to one of the embodiments of the present invention, when an established end-to-end connection fails to meet a condition for a data type, the communication router 106 does not transmit packets belonging to the data type over the established end-to-end connection, but still transmits packets not belonging to the data type over the established end-to-end connection. The established end-to-end connections used to transport a certain type of data packets, but not all types of data packets, belong to a second set of established end-to-end connections.

The manner in which packets are transmitted using the second set of established end-to-end connections may help improve packet transmission performance because not all established end-to-end have the same network performance. For example, some established end-to-end connections have greater latency and some established end-to-end connections have less packet drops. Depending on the conditions that are met by the established end-to-end connection, the established end-to-end connection may belong to a set of established end-to-end connections that are applicable to one type of packet but not another type of packet. For example, the second set of established end-to-end connections with high latency is suitable for data packets belonging to file transfer applications, but not for data packets belonging to voice communications.

According to one of the embodiments of the present invention, the communications router 106 continuously monitors whether any of the second set of established end-to-end connections has become eligible for belonging to the first set of established end-to-end connections due to all conditions being met or has become eligible for belonging to a non-first/second set of established end-to-end connections due to failure to meet other conditions. Moving the established end-to-end connection from the second set of established end-to-end connections to the first set of established end-to-end connections may improve overall network performance of the aggregated end-to-end connection because the communication router 106 has more established end-to-end connections available in the first set to transmit data packets. Moving the established end-to-end connection from the second set of established end-to-end connections to the non-first/second set of established end-to-end connections may also improve the overall network performance of the centralized end-to-end connection, as the communication router 106 does not transmit packets over the established end-to-end connections in the non-first/second set, thereby reducing packet loss, packet delay, packet jitter, and other undesirable effects.

Although it is important that for some applications, the communication router 106 does not transmit packets from the communication router 106 to the communication router 108 over non-first/second set of established end-to-end connections, the communication router 106 should be allowed to use the second set of established end-to-end connections for other applications, such that the strict requirements of not using non-first/second set of established end-to-end connections are relaxed, thereby improving packet performance.

For applications using TCP for communication, packet loss on one of a plurality of established end-to-end connections in a centralised end-to-end connection belonging to the first group may adversely affect the overall throughput of the centralised end-to-end connection. This is because the packets of a TCP session transmitted by communications router 106 to communications router 108 are dispersed among the first set of established end-to-end connections. Thus, when an established end-to-end connection belonging to the first group becomes an established end-to-end connection belonging to a non-first/second group, the communication router 106 should not transmit packets over the established end-to-end connection belonging to the non-first/second group.

Packet detection

According to one of many embodiments, the packets are distributed according to three categories (i.e., data packets, management packets, and error correction packets). In one alternative, the distribution of packets is performed by packet detection, where the contents of the packets are detected to determine whether they are data packets, management packets, or error correction packets. In another alternative, for packets originating from the communication router 106, such as management packets, packet detection is not required to determine the class to which the packet belongs.

A data packet is a packet carrying information that needs to be transmitted from communications router 106 to communications router 108 via at least one established end-to-end connection. The data packet may originate from the communication router 106 or may originate from a host or node connected to the communication router 106. The destination of the data packet is the communication router 108 or a host and/or node connected to the communication router 108.

A management packet is a packet created by the communication router 106 and transmitted via an established end-to-end connection in order to manage the end-to-end connection in the set. Management packets include routing packets, handover packets, health check packets, maintenance packets, performance reporting packets, and other types of packets needed to manage a centralized end-to-end connection. The management information is used to assist the communications router 106, including managing established end-to-end connections, managing centralized end-to-end connections, testing the performance of end-to-end connections, and communicating with the communications router 108 to coordinate management of established end-to-end connections. The management information may be piggybacked with data in one or more data packets, or may be transmitted as one or more separate data packets.

An error correction packet is a packet transmitted by the communication router 106 to correct or detect any errors in the data packet transmitted from the communication router 106 to the communication router 108. The error correction packet includes a Forward Error Correction (FEC) packet, an automatic repeat request (ARQ) packet, a parity packet, and the like.

According to a preferred embodiment, the data packets are transmitted using a first set of established end-to-end connections. The first set of established end-to-end connections has the highest ranking among the other established end-to-end connections. Thus, the first set of established end-to-end connections is most suitable for transporting data packets according to the user's preferences. Management packets, error correction packets are transmitted using either the second set of established end-to-end connections or the non-first/second set of established end-to-end connections. The benefit of using a non-first set and/or a non-first/second set of established end-to-end connections for transport of management packets and error correction packets is that the first set of established end-to-end connections is reserved for data packets and that a higher capacity is available for transport of data packets. Furthermore, it is preferred that the error correction packet is transmitted using an established end-to-end connection different from the established end-to-end connection used for transmitting the data packet, so that the same error does not exist in the error correction packet.

In one of the embodiments, the data packets are transmitted using the second set of established end-to-end connections if the capacity of the first set of established end-to-end connections is full. In one of the embodiments, the data packets are transmitted using a non-first/second set of established end-to-end connections if the capacity of the first set of established end-to-end connections and the second set of established end-to-end connections are full.

In one of the embodiments, the management packet is transmitted using the first set of established end-to-end connections if the first set of established end-to-end connections has sufficient capacity for transmitting non-data packets other than data packets. In one of the embodiments, the management packets for user authentication are transmitted using a first set of established end-to-end connections. For example, an established end-to-end connection using File Transfer Protocol (FTP) requires user authentication, and therefore authentication information is transmitted using the first set of established end-to-end connections.

In one of many embodiments, the data packets are distributed among multiple established end-to-end connections within a group and are transmitted over the established end-to-end connections using the techniques described in fig. 2A and 2B.

Failover

According to one of the embodiments of the invention described in fig. 8 and the classification illustrated in fig. 4, the communication router 106 transmits the data packets using one or more established end-to-end connections belonging to the second group when one or more established end-to-end connections belonging to the first group fail and other established end-to-end connections belonging to the first group do not have sufficient capacity to transmit the data packets. Thus, the established end-to-end connection belonging to the second group performs a backup or failover function. There are many possibilities for the reason that the established end-to-end connection belonging to the first group does not have sufficient capacity. For example, the number of packets is too large. In another example, an established end-to-end connection belonging to the first group fails due to a broken connection or a failure of the device, and thus the established end-to-end connection is broken and can no longer be classified into the first group. Although the performance of established end-to-end connections belonging to the second group may not be as good as those in the first group, the established end-to-end connections belonging to the second group can provide additional capacity for transporting data packets.

According to one of the embodiments of the invention, the first established end-to-end connection is the only one established end-to-end connection belonging to the first group. In step 801 a data packet is transmitted over a first established end-to-end connection. If in step 802 the processing unit 502 determines that the number of data packet drops or losses detected in the first established end-to-end connection within the first predefined time period is less than the first predefined number of data packet drops or losses, then in step 801 the transmission of data packets over the first established end-to-end connection is continued. When the processing unit 502 detects in step 802 that at least a first predefined number of data packet drops or packets are dropped in the first established end-to-end connection within a first predefined time period, the communication router 106 transmits in step 803 a "warm-up" packet over a second established end-to-end connection belonging to the second group. If in step 804 the processing unit 502 determines that the number of data packet drops or losses detected in the first established end-to-end connection within the second predefined time period is less than a second predefined number of data packet drops or losses, then in step 801 the transmission of data packets over the first established end-to-end connection is continued. When the processing unit 502 detects at least a further second predefined number of packet drops or losses in the first established end-to-end connection within a second predefined time period in step 804, the communication router 106 starts transmitting data packets over the second established end-to-end connection in step 805. After the third predefined time period, the communication router 106 tests the performance of the first established end-to-end connection in step 806 to check if there is still at least a third predefined number of dropped packets or packets within a fourth predefined time period. If the first established end-to-end connection has not experienced at least a third predefined number of packet drops or losses within a fourth predefined time period, the communication router 106 starts to repeatedly transmit data packets using the first established end-to-end connection in step 807 and stops transmitting data packets using the second established end-to-end connection in step 808. If in step 806 the processing unit 502 detects at least a third predefined number of data packet drops or losses in the first established end-to-end connection within a fourth predefined time period, the transmission of data packets over the second established end-to-end connection is continued in step 805. It is important that a third predefined time period is allowed to elapse before step 806 is performed, so that handovers between using the first established end-to-end connection and using the second established end-to-end connection are not too frequent, as too frequent handovers may reduce the overall performance. In one variation, step 808 is omitted and the communications router continues to transmit data packets (more precisely, "warm-up" packets) over the second established end-to-end connection so that the second established end-to-end connection remains functional. The process of fig. 8 is performed periodically.

A "warm-up" packet is used in step 803 to help fully enable the second established end-to-end connection in a timely manner. As different access service providers deploy different mechanisms to handle data packets from inactive connections, these connections may experience delays as the data packets begin to be transported over these established end-to-end connections using inactive connections. Likewise, different network adapter manufacturers may deploy different technologies when managing inactive network interfaces. For example, a wireless modem used as or as part of a network interface may be switched to a power-saving mode or a sleep mode after a long period of no reception or transmission of packets. To reduce the probability of slowness experienced by the data packets, the communication router 106 transmits a "warm-up" packet to cause the connection to become functional. This is especially important for wireless connections used to establish end-to-end connections (e.g., connections using satellite, 3G, LTE, and cellular technologies). The "warm-up" package may contain data or may be a health check package. The size and frequency of sending "warm-up" packets should be adjusted for different communication technologies and protocols to establish the end-to-end connection.

Another purpose of transmitting the "warm-up" packet in step 803 is to test whether the second established end-to-end connection is still in operation and still belongs to the second group. If the second established end-to-end connection is not in operation or should not remain in the second group, a third established end-to-end connection belonging to the second group is selected and tested with a "warm-up" packet in step 803.

The first predefined number of data packet drops or losses, the second predefined number of data packet drops or losses, and the third predefined number of data packet drops or losses are set by the manufacturer, administrator, or user of the communication router 106. These predefined numbers may be the same or different. In a preferred embodiment, the first predefined first number of packet drops or losses is less than five, the second predefined first number of packet drops or losses is less than ten, and the third predefined first number of packet drops or losses is less than five. The smaller the predefined number, the more stringent the performance requirements of the established end-to-end connection are, since the smaller the number of packets allowed to be dropped or lost.

The first, second, third, and fourth predefined time periods are set by a manufacturer, administrator, or user of the communication router 106. These predefined time periods may be the same or different. The longer the first, second, third and fourth predefined time periods, the more stringent the performance requirements of the established end-to-end connection are, since the lower the allowed packet drop or packet loss rate.

According to one of the embodiments of the invention and the classification illustrated in fig. 4, the communication router 106 transmits the data packet over the first established end-to-end connection. When the probability that the first established end-to-end connection cannot remain in the first group reaches a predefined threshold or the first established end-to-end connection is to be removed from the first group, the communication router 106 transmits a "warm-up" packet over a second established end-to-end connection belonging to the second group. When the first established end-to-end connection is no longer retained in the first group, the communication router 106 begins transmitting data packets over the second established end-to-end connection. After a predefined time period, when the first established end-to-end connection is again in the first group, the communication router 106 again transmits data packets using the first established end-to-end connection and stops transmitting data packets using the second established end-to-end connection.

The communication router 106 determines the probability of whether the first established end-to-end connection can remain in the first group based on the proximity to violating the conditions of the first group. For example, the conditions of the first group are that the average delay is below 50ms, the average packet loss rate is below one packet in two thousand packets, and there is no data transmission between five and six pm, respectively. For purposes of illustration, when the first established end-to-end connection experiences an average latency that rises from 40 milliseconds to 48 milliseconds within ten seconds, the probability may rise to extremely high values depending on the methodology by which the probability is calculated. In this case, the communication router 106 transmits the "warm-up" packet over the second established end-to-end connection, since the first established end-to-end connection may be removed from the first group in the near future. Also for purposes of illustration, when the time is fifty-eight pm, the communication router 106 transmits a "warm-up" packet over the second established end-to-end connection to test whether the second established end-to-end connection still belongs to the second group. If the second established end-to-end connection still belongs to the second group, the communication router 106 starts transmitting data packets over the second established end-to-end connection at just about five pm in time, since the first established end-to-end connection will not meet the conditions of the first group at five pm.

According to one of the embodiments of the invention, the second established end-to-end connection is either broken or classified into non-first/second groups when the second established end-to-end connection no longer belongs to the second group and also does not belong to the first group. Thus, the communication router 106 transmits another "warm-up" packet over the other one or more established end-to-end connections belonging to the second group to determine which established end-to-end connection can be used in case the first established end-to-end connection no longer remains in the first group. In the event that no more established end-to-end connections belong to the second group, the communication router 106 transmits another "warm-up" packet over one or more established end-to-end connections belonging to the non-first/second group. Since it has been determined that the second established end-to-end connection is broken or is in a non-first/second group, the time window to determine which established end-to-end connection may be used may be reduced when it takes time to use the "warm-up" packet to determine the quality and/or status of the other established end-to-end connection.

A first set of established end-to-end connections may be determined by using the embodiments described above.

FIG. 9 is a flow diagram illustrating a process according to one of the embodiments. In step 901, a centralized end-to-end connection is established. The centralized end-to-end connection comprises a plurality of established end-to-end connections. The plurality of established end-to-end connections includes a first established end-to-end connection grouped into a first set of established end-to-end connections and a second established end-to-end connection grouped into a second set of established end-to-end connections. In step 902, the processing unit 502 determines whether a first condition is met. If the first condition is met, the data packet is transmitted over the first established end-to-end connection in step 905. If the first condition is not met, the data packet is not transmitted using the first established end-to-end connection in step 904. In step 903, the processing unit 502 determines whether a second condition is met. If the second condition is met, the data packet is transmitted over the second established end-to-end connection in step 907. If the second condition is not met, the data packet is not transmitted using the second established end-to-end connection in step 906.

According to one of the embodiments of the present invention, a plurality of established end-to-end connections are grouped after step 901 and before determining whether the first and second conditions are met. The classification is to determine which established end-to-end connections should be grouped together. One condition applies to all established end-to-end connections belonging to the same group. Furthermore, the same set of established end-to-end connections may be managed together.

In one variation, the first condition and the second condition are mutually exclusive. For example, the first condition is met when the second condition is not met, and the second condition is met when the first condition is not met. Thus, the data packet is transmitted either over the first established end-to-end connection in step 905 or over the second established end-to-end connection in step 907.

Alternatively, the first condition and the second condition are not related. It is possible to simultaneously meet both the first condition and the second condition. When both the first condition and the second condition are met, data packets are distributed and transmitted through both the first and second established end-to-end connections.

Alternatively, the first condition and the second condition are related. The probability of meeting a condition is therefore influenced by whether another condition is met or not.

In a variant, the data packet is transmitted in step 905 over all established end-to-end connections belonging to the first group when the first condition is fulfilled in step 902, and the data packet is not transmitted in step 904 over any established end-to-end connection belonging to the first group when the first condition is not fulfilled in step 902. Similarly, when the second condition is met in step 903, the data packet is transmitted in step 907 over all established end-to-end connections belonging to the second group, and when the second condition is not met in step 903, the data packet is not transmitted in step 906 over any established end-to-end connections belonging to the second group.

In a variant, the first and second conditions may be based on the performance of certain established end-to-end connections when used, the service provider of certain established end-to-end connections, the location of the communication router 106, the time of the day, the price of use of certain established end-to-end connections, the security protocol of certain established end-to-end connections, the user identity, the IP address range of the destination of the centralised end-to-end connection, the communication protocol of certain established end-to-end connections, the communication technology of certain established end-to-end connections, the application using certain established end-to-end connections, and the means using certain established end-to-end connections.

Network interface failure

Network interface with end-to-end connection

According to one of the embodiments of the invention and the classification illustrated in fig. 4, the communication router 106 has a plurality of network interfaces and some of the network interfaces are connected with end-to-end connections. The communication router 106 establishes a first established end-to-end connection over the first connection using the first network interface. The first established end-to-end connection is classified into a first group by the processing unit 502. The communication router 106 has no other established end-to-end connections, but has possible end-to-end connections that may be made available through at least one non-first network interface. When processing unit 502 determines that the probability that the first established end-to-end connection cannot be retained in the first group has reached a predefined threshold, the first established end-to-end connection is to be removed from the first group, or the network interface has reported at least one error that may result in the first established end-to-end connection not being retained in the first group, communication router 106 enables the second network interface and attempts to establish a second end-to-end connection with communication router 108. If the second end-to-end connection can be successfully established and the conditions of the second group are met, the processing unit 502 groups the second end-to-end connection into the second group. If the second end-to-end connection cannot be successfully established and/or fails to meet the second set of conditions, the communication router 106 enables the other non-first/second network interfaces and the processing unit 502 determines which network interface is capable of establishing an end-to-end connection meeting the second set of conditions.

Once there is an established end-to-end connection that can meet the conditions of the second set, the communications router 106 is ready to transmit data packets using the established end-to-end connection when the first established end-to-end connection no longer belongs to the first set. This embodiment does not require the second established end-to-end connection to be established in advance. Instead, the second established end-to-end connection is established when the first end-to-end connection has a high probability of not being retained in the first group or will not be retained in the first group. This embodiment reduces the need to send packets over the non-first network interface and makes less use of connections to the non-first network interface.

Those skilled in the art will appreciate that there are a variety of reasons for a network interface to report errors, including connection failures, modem failures, operating system failures, degradation in the signal-to-noise ratio detected by the modem, and the like. When a network interface reports an error, it is likely that it will not be able to transmit and/or receive data packets and affect the connection over which the established end-to-end connection is established.

According to one of the embodiments of the invention, since in this embodiment there is only one established end-to-end connection, there is no need to have a centralized end-to-end connection to maintain the established end-to-end connection, since this may reduce the size of the available payload without providing a significant improvement. Furthermore, since only one network interface is used at a time, the conditions used to determine whether the first established end-to-end connection can remain in the first set may also be used to determine whether all data packets are transmitted using a non-first network interface, regardless of whether the destination of the data packets is the communication router 108.

For example, it is determined that a first network interface of the communication router 106 will be in a first group for transmitting and receiving data packets because it meets all conditions, while a second network interface is determined to be in a second group. When the first network interface is to be removed from the first group, the communication router 106 begins sending a "warm-up" packet over the second network interface to prepare the second network interface for transmission of the data packet. When the first network interface is removed from the first group, the second network interface can begin transmitting and receiving data packets. When the first network interface again belongs to the first group, the communication router 106 will use the first network interface to transmit and receive data packets, and stop using the second network interface to transmit and receive data packets. The network interfaces may be divided into a first group and a second group in a similar manner as the division of established end-to-end connections into different groups as discussed in the embodiments above.

When the communication router 106 does not transmit and receive data packets using the second network interface, the communication router 106 is still able to transmit error correction packets, management packets, health check packets, and test packets using the second network interface.

Network interface without end-to-end connection

According to one of the embodiments of the invention, the classification is performed on the network interface of the communication device instead of on the end-to-end connection. The communication device may be a router, switch, multiplexer, host, or node. This is illustrated in fig. 11, and for better understanding, fig. 11 is viewed in conjunction with fig. 5, where fig. 5 shows a block diagram of a communication device. In step 1101, the data packet is ready for transmission by the communication device. When the processing unit 502 determines in step 1102 that the first condition is met, the communication device transmits the data packet using the first network interface in step 1105. When the processing unit 502 determines in step 1103 that the second condition is met, the communication device transmits the data packet using the second network interface in step 1107. In one variation, when the processing unit determines in step 1102 that the first condition is not met, the communication device may still transmit the data packet using the first network interface; when the first condition is met, the communication device must transmit the data packet using the first network interface. Similarly, when the processing unit 502 determines in step 1103 that the second condition is not met, the communication device may still transmit the data packet using the second network interface; when the second condition is met, the communication device must transmit the data packet using the second network interface. In a variant, when the processing unit 502 determines in step 1102 that the first condition is not met, the communication device stops transmitting the data packet using the first network interface in step 1104, and the same applies for the second condition and the second network interface in step 1106.

The first condition the second condition may be independent, dependent, or dependent. The first condition and the second condition are configured by an administrator, a manufacturer, or a user of the communication device.

In the event that the communication device ceases to use the first network interface to transmit data packets in step 1104 when the first condition is not met, the overall network performance of the communication device will be adversely affected if the first network interface suddenly ceases to operate. Thus, the communication device should cease using the first network adapter to transmit data packets without adversely affecting overall network performance significantly. According to one of the embodiments of the invention, the communication device will prepare the second network interface for transmitting and receiving data packets when it is about to stop using the first network interface. Thus, the communication device prepares the second network interface when the first condition is not to be met. Assuming that the probability that the first condition is still met after a predefined period of time or that the first condition is not met has decreased, the communication device does not attempt to keep the second network interface functional, since keeping the second network interface increases power consumption and results in higher data usage.

The process of fig. 11 is similar to the process of fig. 9, the only difference being that fig. 9 shows the process of selecting an established end-to-end connection for transporting a data packet, while fig. 11 shows the process of selecting a network interface for transporting a data packet.

The previously described conditions for classifying an end-to-end connection or network interface, including performance metrics, service providers, usage metrics, location, time, usage price, security, user identification, internet protocol address range, communication protocol, communication technology and application, may be used as conditions for the network interface. For example, the first condition is based on a performance metric, so the communication device does not further use the first network interface if the first network interface or a network connection carried over the first network interface does not comply with the first condition. In another example, the first condition is that only an authenticated user can use the first network interface based on the user identity. Thus, the communication device does not use the first network interface to transmit data packets originating from non-authenticated users.

According to one of the embodiments of the invention, the third condition is to start an enabling procedure of the second network interface. During the enabling process, the communication device begins to enable or "warm up" the second network interface. There are various ways to enable the second network interface. For example, the second network interface may be enabled by supplying power to the second network interface. In another example, the second network interface is enabled by transmitting a packet to the communication network via the second network interface, the packet may encapsulate data, another packet, management information, randomly generated content, or predefined content. The purpose of enabling the second network interface is to allow the second network interface to transmit data packets when the first condition is met or when the first network interface is stopped transmitting data packets.

The third condition should be more stringent than the second condition, otherwise a situation will occur in which the third condition is met but the second condition has not been met resulting in a failure to timely enable the second network interface.

According to one of the embodiments of the present invention, when the communication device has determined that it is able to transmit the data packet again using the first network interface, the communication device will stop transmitting the data packet using the second network interface and start transmitting the data packet using the first network interface. The fourth condition is used to evaluate whether the first network interface can be used again to transmit the data packet.

According to one of the embodiments of the present invention, the conditions, including the first condition, the second condition, the third condition, and the fourth condition, may follow the conditions described in the previous section of the present invention.

According to one of the embodiments of the present invention, when the communication device transmits data through one of the network interfaces, the communication device may receive the data packet using the same network interface or another network interface.

Fig. 10 is a diagram of a preferred embodiment according to the present invention. At step 1001, a communication device transmits a data packet over a first network interface. At step 1002, the processing unit 502 determines whether the first network interface meets a third condition, the third condition being a third predefined number of data packets to drop or discard within a third predefined time period. If the third condition is met, the communication device begins to initiate the second network interface at step 1003. The third predefined number of data packet drops or losses is in the range of one packet to one hundred packets. The third predefined time period is in the range of one second to one minute. The larger the third predefined number of packet drops or losses, the more packet drops or losses are tolerated. On the other hand the larger the third predefined time period, the less packet drops or packet losses are tolerated. For purposes of illustration, the third predefined number of data packet drops is five packets and the third predefined time period is ten seconds. Therefore, the third condition is satisfied when there are six or more discarded packets within five seconds.

At step 1004, the processing unit 502 determines whether a second condition, i.e. a second predefined number of dropped packets or packet within a second predefined time period detected at the first network interface, is fulfilled. If the second condition is met, the communication device begins transmitting the data packet using the second network interface at step 1005. In one variation, the communication device stops transmitting data packets using the first network interface when the communication device transmits data packets using the second network interface. Thus, the second condition may be configured based on the first condition such that the second condition is opposite to the first condition. Thus, when the first condition is met, the second condition is not met, and when the second condition is met, the first condition is not met. For purposes of illustration only, the second predefined number of packets dropped at the first network interface is ten packets and the second predefined time period is ten seconds. Thus, the second condition is met when there have been ten abandoned packets in less than ten seconds. The first condition should be configured to be less than ten packet losses in ten seconds. Thus, the second network interface is capable of transmitting data packets substantially simultaneously when the communication device ceases to use the first network interface to transmit data packets.

At step 1006, the processing unit 502 determines whether the data packet should be transmitted again using the first network interface. If the fourth condition is met, steps 1007 and 1008 are executed to again transmit the data packet over the first network interface and stop transmitting the data packet using the second network interface. The fourth condition is a fourth predefined number of packets lost within a fourth predefined time period detected at the first network interface. For purposes of illustration only, the fourth predefined number of packets dropped at the first network interface is five packets and the fourth predefined time period is ten seconds.

In one variation, the communications device should not perform step 1006 almost immediately after step 1005. This is because it is possible that the network connection of the first network interface is not yet stable. If step 1006 is performed when the network connection or network interface of the first network interface is unstable, the communication device may frequently switch between using the first network interface and using the second network interface. This may result in significant adverse effects on performance.

In one variation, the fourth condition should not be too relaxed compared to the first condition. Otherwise, the communication device may also experience frequent transitions between using and decommissioning the first network interface. This may also result in significant adverse effects on performance.

The conditions are stored in the slave storage 505 and the conditions are retrieved from the slave storage 505 by the processing unit 502 to determine whether the network interface meets the conditions. The first network interface and the second network interface are two of the plurality of network interfaces 506.

According to one of many embodiments of the invention, the processing unit 502 determines whether a network interface is eligible by determining whether an access link, connection, network, or end-to-end connection with which the network interface is associated is eligible. For example, if the condition is that the network latency experienced by the network interface must be less than two milliseconds, then the condition is equivalent to the network latency observed at the access link connected through the network interface being less than two milliseconds.

Warning message

According to one of the embodiments of the present invention, when an established end-to-end connection is broken out of or joined into a group, or when an established end-to-end connection stop is eligible, the communication router 106 generates at least one warning message and the warning message is sent to a predefined destination. The warning message may be an email, instant message, Short Message Service (SMS), telephone call, message displayed in a web page, pop-up message at a web page, alarm, sound, flashing light, open Light Emitting Diode (LED), and other indicators that may be used to indicate that an event has occurred. The predefined destination may be an email address, an IP address, an LED, a speaker, a display screen, a console, a network node, a host, a mobile phone, a laptop, and any electronic device capable of receiving a warning message. For example, the SMS is sent to an administrator of the communication router 106 when a first established end-to-end connection at the communication router 106 has left the first group. In another example, at least two warning messages are generated by the communication router 106 when a first established end-to-end connection at the communication router 106 has departed from the first group. The at least two warning messages include pop-up messages generated on the web page and instant messages sent to the user and/or administrator of the communication router 106. In another example, when the first established end-to-end connection at the communication router 106 has rejoined the first group, an LED light on the communication router 106 will flash for a few seconds to indicate that the first established end-to-end connection has rejoined the first group.

Entry of conditions

The conditions may be entered into the communication router 106 through a console, a connection device, and/or a removable computer readable storage medium. Specifically, the console is a serial console such that, for example, a laptop is connected to the communication router 106 through the serial console to input, configure, manage, and remove conditions. The connection device is connected to the communication router 106 by an ethernet cable. One end of the ethernet cable is connected to the communication router 106 and the other end of the ethernet cable is connected to a network node that provides connectivity to other networks, including the internet. Thus, for example, an administrator of the communication router 106 can log onto the communication router 106 to enter, configure, manage, and remove conditions. Alternatively, the connection means is a wireless connection implementing a wireless communication standard including LTE, WiFi, High Speed Downlink Packet Access (HSDPA) communication standards, Zigbee, UWB, 3G, WiMAX, bluetooth, GPRS, and the like. Removable computer-readable storage media include flash memory cards, Subscriber Identity Module (SIM) cards, Secure Digital (SD) cards, or any removable computer-readable storage media that allows instructions to be uploaded onto the communication router 106.

According to one of the various embodiments of the invention, a web interface, a command line interface, and/or an Application Programming Interface (API) are provided to an administrator when the administrator accesses the communication router 106 over a connection. Thus, conditions may be entered remotely or locally through a web interface, command line interface, and/or API and may be stored in the secondary storage 505.

Claims (20)

1. A method of transmitting data packets to a second communication router over a plurality of network interfaces of a first communication router, comprising the steps of:

a. establishing a plurality of centralized end-to-end connections;

b. dividing the plurality of aggregated end-to-end connections into a first aggregated end-to-end connection and a second aggregated end-to-end connection;

c. when the end-to-end connection in the first set does not meet a first condition:

not transmitting data packets over any established end-to-end connection belonging to the end-to-end connection in the first set;

d. when the end-to-end connection in the first set meets the first condition:

transmitting data packets over any established end-to-end connection belonging to the end-to-end connection in the first set;

e. when a predefined number of packet losses are detected within a predefined time period:

allowing a second network interface of the first communication router to transmit and accept data packets;

wherein the first set of end-to-end connections are established over at least two of the plurality of network interfaces of the first communication router and at least two of the plurality of network interfaces of the second communication router;

wherein packet loss rate information for transmitting said data packets is continuously monitored;

wherein each of the plurality of end-to-end connections has a different bandwidth capacity;

wherein the first condition is selected from the group consisting of: performance metrics, service providers, locations, time, usage price, security, user identification, Internet Protocol (IP) address range, communication protocols, communication technologies, applications, and devices.

2. The method of claim 1, further comprising: allowing end-to-end connections in a second set to prepare to transmit and accept data packets, wherein the end-to-end connections in the second set are established over the second network interface and at least one of the plurality of network interfaces of the second communications router; wherein the plurality of end-to-end connections belong only to end-to-end connections in the first set or to end-to-end connections in the second set.

3. The method of claim 2, wherein the end-to-end connection in the second set is established using a wireless communication standard.

4. The method of claim 2, further comprising: transmitting a warm-up packet over a second established end-to-end connection of the end-to-end connections in the second set.

5. The method of claim 1, further comprising: transmitting a warning message when the end-to-end connection stop in the first set meets the first condition.

6. The method of claim 1, further comprising: stopping transmission of data packets belonging to a Transmission Control Protocol (TCP) session over the end-to-end connection in the first set when the end-to-end connection stop in the first set meets the first condition.

7. The method of claim 1, further comprising: when the end-to-end connection in the first set is a virtual private network tunnel.

8. The method of claim 1, further comprising: error correction packets are transmitted over the end-to-end connection in the non-first set.

9. The method of claim 2, further comprising: transmitting an error correction packet over a third end-to-end connection of the end-to-end connections in the second set.

10. The method of claim 1, further comprising: transmitting error correction packets or management packets over the plurality of end-to-end connections.

11. A first communications router for transmitting data packets to a second communications router over a plurality of network interfaces, the first communications router comprising:

a plurality of network interfaces;

at least one processing unit;

at least one non-transitory computer-readable storage medium stores program instructions executable by the at least one processing unit for:

a. establishing a plurality of centralized end-to-end connections;

b. dividing the plurality of aggregated end-to-end connections into a first aggregated end-to-end connection and a second aggregated end-to-end connection;

c. when the end-to-end connection in the first set does not meet a first condition:

not transmitting data packets over any established end-to-end connection belonging to the end-to-end connection in the first set;

d. when an end-to-end connection in a first set of end-to-end connections in the first set conforms to the first strip:

transmitting packets over any established end-to-end connection belonging to the end-to-end connection in the first set;

e. when a predefined number of packet losses are detected within a predefined time period:

allowing a second network interface of the first communication router to transmit and accept data packets;

wherein the first set of end-to-end connections are established over at least two of the plurality of network interfaces of the first communication router and at least two of the plurality of network interfaces of the second communication router;

wherein packet loss rate information for transmitting said data packets is continuously monitored;

wherein each of the plurality of end-to-end connections has a different bandwidth capacity;

wherein the condition is selected from the group consisting of: performance metrics, service providers, locations, time, usage price, security, user identification, Internet Protocol (IP) address range, communication protocols, communication technologies, applications, and devices.

12. The first communication router of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: allowing end-to-end connections in a second set to prepare to transmit and accept data packets, wherein the end-to-end connections in the second set are established over the second network interface and at least one of the plurality of network interfaces of the second communications router; wherein each of the plurality of end-to-end connections belongs only to end-to-end connections in the first set or to end-to-end connections in the second set.

13. The first communications router of claim 12, wherein the end-to-end connections in the second set are established using a wireless communications standard.

14. The first communication router of claim 12, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: transmitting a warm-up packet over a second established end-to-end connection of the end-to-end connections in the second set.

15. The first communication router of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: transmitting a warning message when the end-to-end connection stop in the first set meets the first condition.

16. The first communication router of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: stopping transmission of data packets belonging to a Transmission Control Protocol (TCP) session over the end-to-end connection in the first set when the end-to-end connection stop in the first set meets the first condition.

17. The first communications router of claim 11, wherein the end-to-end connection in the first set is a virtual private network tunnel.

18. The first communication router of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: error correction packets are transmitted over the end-to-end connection in the non-first set.

19. The first communication router of claim 12, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: transmitting an error correction packet over a third end-to-end connection of the end-to-end connections in the second set.

20. The first communication router of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit for: transmitting error correction packets or management packets over the plurality of end-to-end connections.