CN1410905A - Full distribution type aggregation network servicer system - Google Patents

Abstract

A fully distributed cluster network server system. The front-end shared media hub serves as the single entry point of the cluster, and the client broadcasts its request to the nodes in the back-end cluster through the hub; each node in the cluster includes a homogeneous network card , packet filtering module, delay equalization module and TCP stack processing module; the isomorphic network card is used to configure the network cards of each node in the cluster to the same IP address and MAC address; the packet filtering module is used to selectively filter out isomorphic network cards Incoming data packets; the delay balance module is responsible for intercepting all request packets and delaying the request packets. After the delay time is over, the delay balance module submits the request packets to the TCP stack processing module for processing; the TCP stack processing module sends back All data packets are sent to the gateway machine through the real network card and sent to the client. The invention has the following technical effects: 1) no single point of failure; 2) no bottleneck effect; 3) applicable to services based on dynamic protocols; 4) fine-grained load balancing.

Description

A Fully Distributed Cluster Network Server System

technical field

The invention belongs to the field of computer applications, and in particular relates to a fully distributed cluster network server system.

Background technique

Today's computer technology has entered the network-centric computing period, and a large number of applications are carried out around the network, which puts forward higher and higher requirements for the performance and reliability of the server. For example, with the rapid development of the Internet and the rapid growth of users, the more popular Web sites will not be able to process user requests in time due to the sharp increase in the number of visits, causing users to wait for a long time or even be rejected, which greatly reduces the quality of service . For CPU-intensive applications, such as Web services with CGI (Common Gateway Interface, Common Gateway Interface) and database operations, the server performance bottleneck problem is even more prominent. In addition, with the promotion of key applications such as e-commerce on the Internet, any exceptional service interruption will cause immeasurable losses, so the reliability of the server is becoming more and more important.

In order to solve the performance problem of servers, many companies and research institutes have proposed scalable cluster systems. These cluster systems transfer external network service requests to a group of back-end real servers through a front-end machine. Scheduling and load balancing technology can be implemented in multiple places such as link layer, network layer or application layer. For example, EDDIE, Reverse-Proxy and pWEB all use the method based on application layer scheduling to build a scalable WEB cluster server. They forward incoming HTTP requests to different web servers, and after obtaining the results, return them to the user. IBM's TCP Router and Network Dispatcher are both schedulers implemented at the network layer. Linux Virtual Server is a cluster scheduling software based on Linux operating system. It provides a variety of request scheduling schemes for different network services by extending the IP protocol stack.

In addition to software implementation, some companies have also proposed hardware solutions, such as Cisco's LocalDirector. It rewrites the import and export packages of cluster servers through hardware, and achieves high performance. It is said that it can schedule hundreds of thousands of TCP connection requests at the same time, but the price of this specialized hardware product is so expensive that ordinary users cannot afford it.

The above solutions all use centralized scheduling. Although it solves the overall performance bottleneck problem of the cluster, they all have a common disadvantage, that is, the reliability of the front-end machine in this centralized scheduling is seldom considered. As the single entry point of the entire cluster server, once the front-end machine fails, the entire server will be paralyzed, which may cause huge economic losses. A simple solution is to set up a backup machine for the front-end machine. When the front-end machine breaks down, the backup machine will continue to work instead of its IP address. An important shortcoming of this fault-tolerant processing method is that all established TCP connections will be lost, which will cause inconvenience or even loss to users. In addition, the above-mentioned centralized scheduling schemes all use connections as the load measurement unit, so they cannot achieve ideal load balancing effects. Especially in a heterogeneous cluster where each node provides multiple different services, each service consumes different network resources, so it will become extremely unreasonable to use the number of connections to weigh the load of each node.

To sum up, the traditional cluster server based on centralized scheduling has the following disadvantages: 1. The front-end machine becomes a single point of failure; 2. Taking the connection as the load balancing unit will lead to inaccurate load positioning; 3. Since the same IP is directed to the same node 4. In the increasingly widely used end-to-end encryption applications, the front-end machine will become a bottleneck point, resulting in poor scalability of the cluster for encryption applications.

Contents of the invention

Aiming at the shortcomings of centralized scheduling, the present invention proposes a fully distributed cluster network server system, which eliminates the concept of "front-end machine" in the centralized scheduling scheme, has no single point of failure, no bottleneck effect, and is suitable for Technical effects of dynamic protocol-based services and fine-grained load balancing.

In order to achieve the above-mentioned purpose of the invention, a fully distributed cluster network server system is characterized in that:

The front-end shared media hub acts as a single entry point for the cluster, through which clients broadcast their requests to the nodes in the back-end cluster;

Each node in the cluster includes isomorphic network card, packet filter module, delay balance module and TCP stack processing module;

The homogeneous network card is used to configure the network cards of each node in the cluster to the same IP address and MAC address;

The packet filtering module is used to selectively filter out the data packets from the isomorphic network card;

The delay balance module is responsible for intercepting all request packets and delaying the request packets. After the delay time is over, the delay balance module submits the request packets to the TCP stack processing module for processing;

All data packets sent back by the TCP stack processing module are sent to the gateway machine through the real network card and sent to the client.

The packet filtering module includes a packet processing module, a PASS table, a timing trigger and a rule management module;

The packet processing module resides in the link layer, and is responsible for inquiring about the rules of the PASS table and checking the protocol number, source IP address, target IP address, source port of these packets when receiving the data packets from the isomorphic network card. , the target port to match, and complete the corresponding processing action, the corresponding action is divided into three categories: discarding the packet, submitting the packet interception sub-module in the delay balance module and submitting the kernel TCP/IP protocol stack processing module;

The PASS table is a rule set described by tuples, which is responsible for defining the data packets that the current packet processing module allows to pass;

The timing trigger is responsible for triggering the rule management module to maintain the PASS table at regular intervals;

The rule management module is responsible for updating and deleting rule items in the PASS table.

The delay balance module includes a timing trigger, a load statistics module, a delay calculation module, a packet interception module and a packet queue table;

The timing trigger is responsible for triggering the load statistics module to collect system resources, and simultaneously triggering the packet interception module to obtain the data packets suspended in the packet queue table;

The load statistics module is responsible for collecting the utilization of various system resources, it communicates with each device of the system and collects the resource utilization of each device, and passes these utilizations to the delay calculation module as a vector;

The delay calculation module calculates the delay value according to the load vector sent by the load statistics module;

Described packet interception module receives the SYN packet that transmits from packet processing module, and hangs this SYN packet in the packet queue according to the delay value that delay calculation module transmits; In addition, when receiving the timing trigger sent When the signal is triggered, it will traverse the packet queue table to determine whether the delay time of each packet is over, and if it is over, the datagram will be retrieved and sent to the TCP stack processing module;

The packet queue table is used to store delayed SYN packets, and each packet carries a delay variable, which is retrieved by the packet interception module after the delay time is over.

The present invention has the following technical characteristics:

Isomorphic network card technology realizes a global single MAC image, that is, the network card receiving service requests in each node can be bound to a unified virtual IP address and virtual MAC address, so it can be guaranteed that each node will be able to receive the information forwarded by the front-end hub service request package;

Based on the distributed balancing technology of delay competition, this technology can delay all connection requests for a certain period of time according to the local load (CPU load, memory usage, disk I/O, network traffic, etc.). The longer the delay;

Stateful link layer packet filtering technology, which can remember the data streams belonging to the same connection, allow those data packets that have been established to pass through, and block the datagrams that have not been established to pass through, so as to avoid some meaningless datagrams going up layer protocol stack flows.

Based on the above technical characteristics, the present invention has the following technical effects: 1) there is no single point of failure

Since the cluster server system proposed by the present invention adopts a fully distributed architecture, when one of the nodes fails, the remaining nodes will continue to take over the work, and can automatically redistribute service requests to the server with the lightest load. on the node. 2) No bottleneck effect

In the centralized scheduling method, when the cluster provides encrypted applications, the front-end machine needs to frequently encrypt and decrypt the incoming and outgoing packets before scheduling to the back-end node, which will cause the front-end machine to become a bottleneck point and reduce the scalability of the system. worse. However, the distributed network scheduling scheme of the present invention uses a hub to replace the front-end scheduling machine in the centralized scheduling, thereby eliminating this bottleneck effect. 3) Suitable for services based on dynamic protocols

Many services now use dynamic protocols (such as FTP, RTSP). Since service nodes will temporarily open some new data ports dynamically during the service process, in centralized scheduling, the front-end machine needs to know these temporary ports and open them. Therefore, frequent communication between the service node and the front-end machine is required, and when the service node temporarily opens a port in the distributed network scheduling of the present invention, only the PASS table of the local packet filtering module needs to be updated, thereby reducing network overhead. 4) Fine-grained load balancing

In the existing centralized scheduling scheme, the connection is used as the load balancing unit, so the load of each service node will be unbalanced, but the present invention adopts a load-based scheduling method, which can accurately locate the node with the lightest load and achieve excellent The load balancing effect, so it can be better applied to heterogeneous cluster systems.

Description of drawings

Fig. 1 is a kind of fully distributed cluster network server system structure schematic diagram of the present invention;

Fig. 2 is the workflow diagram of each node;

Fig. 3 is a schematic diagram of the structure of the packet filtering module in Fig. 1;

Figure 4 is a schematic diagram of the structure of the delay equalization module in Figure 1.

Detailed ways

The system structure of the present invention is shown in Figure 1. The front-end shared media hub acts as a single entry point for the cluster. With the help of the shared media feature of the hub, all data packets accessing the cluster can be received by the homogeneous network cards of internal nodes and sent to their respective link layers. All service nodes in the cluster are equipped with a homogeneous network card, and these network cards are bound to a public external IP address to realize a single IP image of the cluster. It can be seen from Figure 1 that when the client sends a connection request, a request packet (SYN packet) will be generated, and the destination IP address of the SYN packet is the external IP address of the cluster. When the request packet arrives at the hub at the front end of the cluster server, the hub will broadcast the request packet to the service nodes at the back end.

The workflow of each node is shown in Figure 2. The homogeneous network cards of all nodes will receive the request packets broadcast from the hub and submit them to the packet filtering module residing in the link layer, which will allow all request packets and Those packets that have established connections are passed, and those packets that have not yet established connections are rejected. The packet filter module submits the request packet to the upper layer IP protocol stack, and the delay balance module residing in the TCP layer will intercept all request packets, and perform a certain analysis on the request packet according to the load information collected by the load statistics sub-module in this module. Time delay, the delay time is proportional to the load of the machine (the heavier the load, the longer the delay), when the delay time is over, the delay balance module will submit the request packet to the TCP stack processing module for processing, according to the processing of the TCP protocol stack The process sends back a request acknowledgment packet (SynAck packet) to the client. Therefore, the request confirmation packet sent by the node with the lightest load is the earliest, and the request confirmation packets of all nodes will arrive at the client in turn at the end, that is, the request confirmation packet of the light-loaded node arrives first, and the request confirmation packet of the heavy-loaded node arrives last . All data packets sent back by the TCP stack processing module are sent to the gateway machine through the real network card.

As shown in Figure 1, the output data packets of all nodes are returned to the client through the single exit point of the gateway machine. Due to the inherent first-come-first-served feature of the TCP protocol stack, the client will only receive the request confirmation packet that arrives first (the request confirmation packet is sent by the node with the lightest load) and feed back a connection confirmation packet (ACK packet), while Subsequent arrival packets are discarded and a reset packet (RST packet) is fed back. After the packets sent back by these clients (one ACK packet, multiple RST packets) are received by all nodes in the cluster, each node will perform certain processing: the node receiving the ACK packet will complete the establishment of the connection, and in the filtering module Add a rule to allow the subsequent data packets belonging to the connection to pass; the node receiving the RST packet will release the socket buffer opened, that is, the node will not be able to establish a connection with the client. Finally, each time the client accesses the cluster server, it can always establish a connection with the node with the lightest load to complete fine-grained dynamic network load balancing.

Each module is described in detail below. 1. Isomorphic NIC

The isomorphic network card 4 is realized by the isomorphic network card technology, which configures the network cards of each node in the cluster to have the same IP address (virtual IP address) and MAC address (virtual MAC address). This virtual IP address is the public access address of the cluster server.

Each node has an ARP cache. It stores recent mapping records between IP addresses and hardware MAC addresses. When a network data packet is routed, it can get the address of the next router or the IP address of the destination host, and then search for the corresponding MAC address in the ARP cache. If it is not found, broadcast a message in the LAN. ARP query packet, which will query the MAC address corresponding to the address of the next router or the address of the destination host, and add the new IP address-MAC address mapping relationship to its own ARP cache, and then send an IP address based on this MAC address. data pack. In this technology, in order to ensure the consistency of the mapping relationship between the virtual IP address and the virtual MAC address in the ARP cache on each node and avoid IP address conflicts, it is necessary to shield the response of the ARP broadcast inquiry packet in the kernel. The implementation method is to modify / The code about the ARP query packet in the net/ipv4/arp.c file. In addition, because the network cards of each node are configured with the same MAC address and the same IP address, the ICMP protocol UDP port unreachable message sent by a certain node will cause a network storm. In order to suppress this ICMP message storm, the implementation of this technology also includes To modify the udp_recv() function in /net/ipv4/udp.c.

Finally, use the ifconfig command to configure the virtual MAC address of the network card. For example, if you want to configure the MAC address of the network card of all nodes as the virtual address 00:03:61:83:2a:8b, you need to perform the following configuration steps (take the network card of node 1 eth1 as an example): Step 1: [root@node1/]#ifconfig eth1 down Step 2: [root@node1/]#ifconfig eth1 ether HW 00:03:61:83:2a:8b Step 3: [ root@node1/]#ifconfig eth1 up2 packet filter module

The packet filtering module 5 is realized by using stateful link layer packet filtering technology, which is used to selectively filter out the data packets transmitted from the homogeneous network card. Because each data packet flowing into the cluster must be broadcast to each node through the hub, the homogeneous network card of each node will receive all these packets and submit them to the upper kernel protocol stack, which will cause the kernel protocol stack of most nodes to lose It consumes a lot of resources to process some irrelevant data packets, especially in the case of intensive service requests, the negative effect of this broadcast will be more obvious. Therefore, the present invention discards irrelevant data packets at the link layer by using stateful packet filtering technology. The structural representation of this module is as shown in Figure 4, specifically comprises: 1) packet processing module (10): this module resides in the link layer, when receiving the data packet that isomorphic network card transmits, it will query the PASS table The rules match the protocol number, source IP address, destination IP address, source port, and destination port of these packets, and complete the corresponding processing actions. The corresponding actions are divided into three categories: discarding this packet, submitting the delay balance module Packet intercepting submodule (17) and submitting kernel TCP/IP protocol stack processing module (7). 2) PASS table (11): The PASS table is a rule set described by tuples, which defines the data packets allowed by the current packet processing module. In this table, all data packets related to control connections are allowed to pass through, such as: SYN packets, SynAck packets, ACK packets, RST packets, etc.; all established connection data packets will be allowed to pass; specified by the member, otherwise it will be discarded by default. 3) Timing trigger (12): responsible for triggering the rule management module to maintain the PASS table at regular intervals. 4) Rule management module (13): responsible for updating and deleting rule items in the PASS table. When a TCP connection is established or withdrawn, the TCP stack processing module will notify the module, and then the module will perform appropriate processing: when a connection is established, it sends the connection information (such as: source IP, source port, destination IP , target port) is converted into a rule and added to the PASS table; when a connection is withdrawn, it will delete the corresponding rule in the PASS table. 3 delay equalization module

The delay balance module 6 is realized by the delay competition balance technology, which uses the three-way handshake mechanism of the TCP protocol to complete the connection establishment between the client and the node with the lightest load. As the client, because the natural response delay of the protocol stack cannot reflect all the performance conditions of the system, in order to generate an appropriate delay value, it is necessary to collect the load conditions of all devices. The structural diagram of this module is shown in Figure 3, specifically includes: 1) timing trigger (14): responsible for triggering the load statistics module to collect system resources, and simultaneously triggering the packet interception module to obtain the data packets suspended in the packet queue table . 2) load statistics module (15): responsible for collecting the utilization of various system resources, it communicates with each system equipment such as CPU, disk, memory and network card and collects the resource utilization of each equipment, and uses these utilization as vector ( LOAD _CPU , LOAD _Disk , LOAD _Mem , LOAD _NIC ) are passed to the delay calculation module. 3) Delay calculation module (16): calculate an appropriate delay value according to the load vector transmitted from the load statistics module, and the specific calculation formula is provided by the following function.

Delay=f(Load)=RTT _MAX ×(Load/Load _MAX );

Load＝LOAD _CPU × Weight _CPU + Load _Disk × Weight _Disk + LOAD _Mem × Weight _Mem +

Load _NIC × Weight _NIC ;

Among them, RTT _MAX represents the Round-Trip Time value that can be achieved when the cluster communicates with the farthest client (that is, the number of Hops in the middle is the largest); Load _MAX is the peak load value, which is related to the specific service provided; Weight _CPU , Weight _Disk , Weight _Mem , and Weight _NIC all adopt normalized values, and have different weights for different applications or content, and can be combined with content scheduling to dynamically use different definitions, such as:

For static web pages, Weight _CPU = Weight _Mem = Weight _NIC = 0, and Weight _Disk = 1;

For dynamic web pages, Weight _CPU = 0.5, and Weight _Disk = 0.5, Weight _Mem = Weight _NIC = 0;

To video service, Weight _CPU =Weight _Mem =Weight _NIC =Weight _Disk =0.25. 4) packet interception module (17): packet interception module receives the SYN bag that transmits from packet processing module (8), and calculates module according to delay The transmitted delay value hangs the SYN packet into the packet queue; in addition, when receiving the trigger signal from the timing trigger, it will traverse the packet queue list to determine whether the delay time of each packet is over, if After finishing, the datagram is retrieved and sent to the TCP stack processing module (7). 5) Packet queue table (18): This table stores delayed SYN packets, and each packet carries a delay variable, which is retrieved by the packet interception module after the delay time ends. 4TCP stack processing module

The TCP stack processing module 7 is realized by adopting security-enhanced TCP protocol stack processing technology, which is a modified protocol module with some simple security mechanisms added. The cluster system is vulnerable to DoS attacks because the potential broadcast overhead will easily cause those "lost" nodes to open up too much socket memory space in response to the three-way handshake. In this module, it is only necessary to enable the syn-cookie technology in the proc file system.

The hardware and operating system configurations of each node in the present invention can be selected from the configuration shown in Table 1 for specific implementation, but is not limited to this configuration.

During specific implementation, the external public network is connected to the shared media hub at the outer end of the cluster server, and at the same time connected to the internal switch with a network cable. Then configure each node accordingly to start the homogeneous network card. CPU Memory motherboard operating system network PIII866 256M Embedded with two 100Mbps network interface cards Linux kernel 2.4.5 100M switch

Table 1 The hardware and software configuration of the distributed adaptive scheduling cluster system

During specific implementation, the external public network is connected to the shared media hub at the outer end of the cluster server, and at the same time connected to the internal switch with a network cable. Then configure each node accordingly to start the homogeneous network card.

In conjunction with the accompanying drawings, the configuration of the entire system is described as follows: 1) PASS table

The PASS table is used to specify which packets can be received by this node. The initial configuration is shown in Table 2. The meaning of this table is: receive all data packets whose flags are SYN, SynAck, and RST, and receive all packets sent from this node , and discard all other packets by default.

The fields are explained as follows:

Protocol number: divided into TCP, UDP, ICMP, ANY, ANY refers to any protocol;

Flag bit: divided into SYN, SynAck, RST, ANY;

Source IP/Source Port: the source IP address/source port of the packet;

Destination IP/destination port: the destination IP address/destination port of the data packet;

Measures: Refers to the measures to be accessed for the matched data packets, divided into two types: PASS (receive) and DROP (reject). agreement number flag bit source IP source port Destination IP destination port measure TCP SYN ANY ANY 17.0.0.2 twenty one PASS TCP SynAck ANY ANY 17.0.0.2 twenty one PASS TCP RST ANY ANY 17.0.0.2 twenty one PASS ANY ANY 17.0.0.2 ANY ANY ANY PASS ANY ANY ANY ANY ANY ANY DROP

Table 2 provides a node configuration example of FTP service (assuming that the node IP is 17.0.0.2) 2) delay calculation module

The delay calculation module needs to calculate the corresponding delay according to the service type and the load weights of CPU, memory, network card and disk. The corresponding relationship is shown in Table 3. Service type _WeightCPU Weight _Mem Weight _NIC Weight _Disk HTTP 0.5 0.5 0 1 FTP 0 0.5 0 0.5 RSTP 0.25 0.25 0.25 0.25

Table 3 Correspondence between service types and load weights

Claims (3)

1, a kind of full distributed cluster network service device system is characterized in that:

The shared medium type hub of front end is as the single entrance of cluster, and the client is broadcast to its request on the node in the cluster of rear end by this hub;

Each node includes isomorphism network interface card, packet filtering module, delay balance module and TCP stack processing module in the cluster;

The isomorphism network interface card is used for the network interface card of inner each node of cluster is configured to identical IP address and MAC Address;

The packet filtering module is used for selectively filtering out the packet that the isomorphism network interface card transmits;

The delay balance module is responsible for intercepting all request package, and request package is postponed, and after finishing time delay, the delay balance module submits to TCP stack processing module to handle request package;

All packets that TCP stack processing module is beamed back send to gateway machine by true network interface card and deliver to client.

2, system according to claim 1 is characterized in that:

Described packet filtering module comprises packet handing module, PASS table, clocked flip-flop and rules administration module;

Described packet handing module resides at link layer, be responsible for when receiving the packet that the isomorphism network interface card transmits, mate with the rule of inquiry PASS table and to protocol number, source IP address, target ip address, source port, the target port of these bags, and finish corresponding processing action, this corresponding actions is divided three classes: abandon this bag, submit the bag intercepting submodule in the delay balance module to and submit kernel ICP/IP protocol stack processing module to;

Described PASS table is a rule set of describing with tuple, is responsible for the current packet handing module of definition and allows the packet that passes through;

Described clocked flip-flop is responsible at set intervals, and the triggering rule administration module carries out attended operation to the PASS table;

Described rules administration module is responsible for upgrading, delete the regularization term in the PASS table.

3, system according to claim 1 and 2 is characterized in that:

Described delay balance module comprises clocked flip-flop, load statistics module, time-delay computing module, bag interception module and bag queue table;

Described clocked flip-flop is responsible for triggering the load statistics module system resource is collected, and triggers the bag interception module simultaneously and obtain the packet of hanging up in the bag queue table;

Described load statistics module is responsible for collecting the situation of utilizing of various system resources, it and each devices communicating of system and the resource utilization of collecting each equipment, and these utilization factors are passed to the time-delay computing module as vector;

Described time-delay computing module calculates delay value according to the load vector that the load statistics module transmits;

Described bag interception module receives the SYN bag that transmits from packet handing module, and according to the delay value that the time-delay computing module transmits this SYN bag is suspended in the bag formation; In addition, when receiving the trigger pip that clocked flip-flop is sent, it will travel through the bag queue table, judge whether finish the time delay of each bag, if finish then fetch this datagram and send TCP stack processing module to;

Described bag queue table is used to deposit the SYN bag that is delayed, and each bag all carries a time-delay variable, is just fetched again by the bag interception module after finishing time delay.

Images