HK1204726B - System and method for restricting requests to access - Google Patents

Description

System and method for limiting access request

Technical Field

The present invention relates to a system for restricting access requests from a network and a corresponding method.

Background

A conventional Load Balancer (Load Balancer), for example, configured by HaProxy under linux, is configured to read cookies or URL interpretations included in each HTTP request from the network, rewrite a header based on the information, and send the HTTP request to the backend server cluster, so that each server in the backend server cluster reaches a balanced state of traffic and resource occupation. However, conventional load balancers do not automatically filter network traffic and cannot throttle or drop attack traffic.

In the prior art, it is known to protect against ICMP/TCP/UDP stream attacks by detecting TCP packets based on traffic flushing, and to retransmit TCP/UDP packets. This known solution is only effective for packets at the TCP/UDP layer and it is not effective for preventing HTTP streaming attacks at the application layer 7 of the open systems interconnection reference model (OSI) that requires decryption.

One idea is to limit access requests according to the amount of access per unit time, such as QPS, based on statistics on access URLs. Statistics for a large website visit URL typically consume a lot of memory. Generally, for any one combination of data item IP address, user identification (USERID), Uniform Resource Locator (URL), etc., a timestamp (timestamp) for each access associated with the combination needs to be recorded. When the QPS needs to be calculated, each time point is filtered or sequenced, which is time-consuming and memory-consuming.

In the prior art, when the QPS needs to be calculated, each time point is filtered or sequenced, which is time-consuming and memory-consuming.

Disclosure of Invention

It is an object of the present invention to provide a system and method that at least partially solves the above mentioned problems. The invention is particularly suitable for solving the field of network DOS attack (denial of service attack) and DDOS attack (distributed denial of service attack). The technique is particularly suitable for protection against HTTP streaming attacks (HTTP streaming). The invention finds out the attack flow which is matched with the mode through the statistics and analysis of the user flow and limits the flow or discards the attack flow, thereby protecting the back-end server of the attack flow.

According to one aspect of the present invention, a system for restricting access requests from a network includes a load balancer, an aggregator, and an aggregator. The load balancer receives access requests from the network, parses each received access request into a UDP message, and passes the UDP message to the aggregator; the aggregator counts the received UDP messages according to the predefined statistical data item combination and the predefined statistical duration, and sends statistical results to the aggregator according to the request of the aggregator; the aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request, wherein the blacklist defines processing actions for respective access requests. Wherein the load balancer determines a processing action for each access request received based on the current blacklist.

According to another aspect of the invention, there is provided a method performed in a system for restricting access requests from a network, the system comprising a load balancer, an aggregator and an aggregator, the method comprising: the load balancer receives the access request from the network, analyzes the received access request into a UDP message, and transmits the UDP message to the aggregator; the load balancer determines a processing action for the received access request according to a current blacklist, wherein the blacklist defines the processing action for the access request. The aggregator receives the UDP messages sent by the load balancer, counts the received UDP messages according to a predefined statistical data item combination and a predefined statistical duration to generate statistical results, and sends the statistical results to the aggregator according to the request of the aggregator; the aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request.

According to the invention, the statistical result gives the number of UDP packets accumulated over the predefined statistical duration and containing the combination of the predefined statistical data items

According to the invention, the network is the internet and the access request is an HTTP request from the internet.

According to the invention, the aggregator comprises a plurality of aggregators operating independently of one another, and the load balancer transmits each UDP packet to one aggregator, each aggregator performing the statistics separately. The aggregator receives the statistics from the plurality of aggregators and generates aggregated statistics. And the blacklist is generated according to the summarized statistical result and the predefined rule.

The system according to the invention further comprises a predefined white list defining access requests that are specified to be released. And the load balancer determines the processing action of the access request on the received access request as follows: when the access request is matched with the white list or not matched with the current black list, the access request is released; and when the access request does not match the white list and the current black list, operating the access request according to the processing action specified by the corresponding record in the black list.

The system according to the invention, wherein said processing action specified in said blacklist comprises one of: interception, redirection, and latency.

Drawings

Embodiments of the invention will be described in detail below with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of a system for restricting access requests in accordance with the present invention;

FIG. 2 is a timing diagram of a load balancer processing a received access request according to the present invention;

fig. 3 is a block diagram of an exemplary composition of an aggregator in accordance with the present invention.

Detailed Description

The present invention provides a system and method for restricting access requests from a network using statistical information.

An access request, such as an HTTP request, includes the following data items: the query user is about the destination domain name (host) to access; a universal resource identifier (uri) requested by the user; a user name (uid); client IP address (cip), etc.

In accordance with the present invention, a system for restricting access requests from a network includes a load balancer, an aggregator, and an aggregator.

The load balancer receives access requests from the network, parses each received access request into a UDP message, and passes the UDP message to the aggregator.

The aggregator makes statistics on the received UDP messages according to the predefined statistical data item combination and the predefined statistical duration, and sends statistical results to the aggregator according to the request of the aggregator. The statistical result gives the number of the UDP messages which are accumulated in the current predefined statistical duration and contain the predefined statistical data item combination.

The aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request, wherein the blacklist defines processing actions for respective access requests.

The load balancer determines a processing action for each access request received based on the current blacklist.

A system according to an embodiment of the invention is described in detail below with reference to specific embodiments. It is to be understood that the specific embodiments are merely illustrative of the spirit and specific implementations of the invention, which are not to be construed as limiting the invention to the specific embodiments.

Fig. 1 shows a system 100 according to an embodiment of the invention, the system 100 comprising 3 modules:

the load-balancer 101 is provided with a load-balancer,

an aggregator 102, and

an aggregator 103.

Load balancer 101 receives an access request from a network, such as an HTTP request from the internet, parses the received access request into a UDP packet, and transmits the UDP packet to aggregator 102.

The additional overhead of the load balancer can be greatly reduced due to the advantages of small resource consumption, high processing speed and the like of the UDP message. The UDP protocol does not belong to a connection type protocol, so that a certain message loss phenomenon may occur. But the loss is within a controlled error range and has negligible impact on the robustness of the system.

The aggregator 102 may include multiple aggregators. The plurality of aggregators are horizontally and independently expanded, namely, each aggregator is independently deployed and operated, and the aggregators have no influence and dependence on each other. If the aggregator processing capacity is not sufficient, machines may be added and new aggregators may be deployed. Therefore, the requirement of large traffic flow can be better met.

The load balancer 101 transmits each UDP packet to a selected aggregator. Each aggregator performs the statistics separately.

According to one embodiment of the invention, load balancer 101 sends UDP messages to multiple aggregators in a round-robin fashion. For example, a first UDP packet is sent to the first aggregator, a second UDP packet is sent to the second aggregator, and the generated UDP packets are sequentially sent to the aggregators. And after the UDP packet is sent to the last aggregator, the data packet is sent to the first aggregator again.

Each aggregator performs statistical analysis on the received UDP packets according to predefined statistical rules or conditions, such as predefined statistical data item combinations, predefined statistical time intervals, and the like.

The aggregator receives the UDP message and parses the UDP message. The UDP message includes information of the HTTP access request of the user: a destination domain name (host) accessed by the user, a universal resource identifier (uri) requested by the user, a user name (uid), a client IP address (cip), and the like.

Then, the aggregator makes statistics on the number of access requests of the same destination domain name (same host) and the same source (same uid, same cip), and can calculate the number of access requests of the source.

If the above statistical access request amount is examined over a defined period of time, such as the last 60 seconds or the last 5 minutes, the request amount per unit time for this source can be derived, such as by dividing the access request amount by the duration of the time period to obtain the QPS value. QPS is the query rate per second, which is a measure of how much traffic a particular query server processes within a specified time, and the performance of a machine as a domain name system server is usually measured by the query rate per second on the internet.

The aggregator then sends the statistics to it upon request by aggregator 103.

The aggregator 103 requests "statistics" from the aggregator at regular intervals, for example, every 10 seconds, such requests being made, for example, in the form of HTTP.

When the aggregators of the aggregator receive the request from the aggregator 103, the current statistics are constructed into response messages, for example, response messages in the form of HTTP, and the response messages containing the current statistics are sent to the aggregator 103.

The aggregator 103 receives the response messages from the aggregators, processes the statistical results contained in the messages received from all the aggregators according to a specific classification, generates aggregated statistical analysis results, and generates a blacklist according to predefined rules.

According to one embodiment of the invention, a blacklist record in the blacklist contains 4 parameters:

the domain name of the destination (host),

the source user name (uid),

source client IP (cip), and

processing action (action).

The meaning of the record is understood to mean that, for a request sent to the destination domain name (host), if the user name and client IP are identical to the corresponding values in the record (source user name, source client IP), the corresponding processing action is taken. The processing action is for example intercepting or redirecting or delaying the request.

When the destination domain name, the user name and the client IP address of an access request and the destination domain name, the user name and the client IP address recorded in a blacklist are used, the access request is called to be matched with the blacklist.

A threshold amount of requests per unit time and processing actions for a particular destination domain name are set in the configurator 1036 of aggregator 103.

The summarizing device 103 performs a determination according to a summarizing statistical result of the access request combined with some statistical data items, and when a unit time access amount (QPS) value of such an access request exceeds a unit time access amount threshold value corresponding to its destination domain name, generates a corresponding blacklist record, which includes the destination domain name, the user name, the client IP, and a corresponding processing action of the request. The generated blacklist record is added to the blacklist, thereby generating a current blacklist.

In addition, a white list is maintained in the aggregator 103. The white list may have the same structure as the black list, but the white list implements a preferential processing strategy, and for an access request matching the white list, the processing action is uniformly released, i.e. the access request is sent to the destination back-end server (host), and the back-end server responds to the request.

Aggregator 103 asynchronously loads the blacklists and predefined whitelists to load balancer 101. According to one embodiment of the invention, the listening thread of aggregator 103 is used to provide load balancer 101 with a blacklist. When load balancer 101 initiates a request, the listening thread obtains the blacklist and sends the blacklist to load balancer 101 together with the whitelist.

Load balancer 101 is the actual actor of a "process action" in the blacklist.

Of course, the white list need not necessarily be maintained in aggregator 103, and may be maintained in load balancer 101, for example.

If an access request matches a whitelist, load balancer 101 performs a processing action on the query that is passed, regardless of whether the access request matches a blacklist, since load balancer 101's processing principle is whitelist first. If the access request does not match the white list but the black list, load balancer 101 takes a corresponding processing action as specified by the black list. If the access request does not match either the white list or the black list, the access request may be passed.

According to the present invention, load balancer 101 processes access requests from the network according to the black list and the white list as follows:

when the access request matches the white list or does not match the black list, the access request is released, and a corresponding UDP message is constructed and sent to the aggregator (because only the released traffic is the traffic really sent to the corresponding back-end server, the aggregator needs to continue to calculate the 'statistical result' of the access request);

and when the access request does not match the white list and the black list, operating the access request according to the processing action specified by the corresponding record in the black list.

Fig. 2 illustrates a method for a load balancer to restrict access requests from a network according to the present invention.

At step S11, the load balancer receives the access request from the network and parses the received access request into a UDP packet.

In step S12, the UDP message is transmitted to the aggregator.

The load balancer then determines the processing action for the received access request based on the current whitelist and blacklist, where the blacklist defines the processing action for the access request, as follows.

At step S13, the load balancer determines whether the access request matches a white list. If so, go to S14, i.e., release the access request and send it to the corresponding backend server. Otherwise, it proceeds to step S15.

At step S15, the load balancer determines whether the access request matches a blacklist. If not, then go to S14, i.e., the access request is released and sent to the corresponding backend server. Otherwise, it proceeds to step S16.

In step S16, the load balancer processes the access request according to the corresponding processing action specified in the blacklist.

According to the invention, the aggregator receives the UDP packets sent by the load balancer, counts the received UDP packets according to a predefined combination of statistical data items and a predefined statistical duration to generate statistical results, and sends the statistical results to the aggregator according to its request. The statistical result gives the number of the UDP messages which are accumulated in the current statistical duration and contain the predefined statistical data item combination.

The aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request.

The aggregator and its process of performing statistics are described in detail below.

The aggregator generates statistics for particular statistics items from the received UDP packet. According to one embodiment of the invention, an aggregator comprises:

a receiving part, configured to receive a UDP packet sent by the load balancer, where the UDP packet is generated by the load balancer parsing a query request from a network;

a calculation part for counting the received UDP message according to the predefined statistical data item combination and the predefined statistical duration to generate a statistical result;

and a transmission section to which the statistical result is transmitted according to a request of the aggregator.

For each HTTP request from the network, it can only be parsed into one UDP message by the load balancer 101 and transmitted to one machine on the aggregator 102's cluster of aggregators. Each aggregator makes statistics on the number of access requests according to predefined statistical conditions, such as according to a specific combination of data items in the UDP message, such as the user name userid, the IP address IP, and the universal resource identifier uri, and a specified statistical period.

For the received UDP message, the aggregator extracts information items such as userid, ip, uri and the like from the UDP message, takes a specified combination (various combinations can exist) of the information items as a statistical data item, and carries out statistics on the number of requests according to the set statistical time period duration (such as short time period and long time period).

Common statistical data items include the following three:

userid+ip+uri；

userid+uri；

ip+uri。

after the statistics are complete, the aggregator generates statistics, for example in the form of web pages in the JSONS data format. Upon receiving the request of the aggregator 103, the statistics are transmitted to the aggregator 103.

The detailed process of calculating the flow statistics by one aggregator is as follows:

after starting the statistical process, when receiving a first upd message of a given statistical data item, initializing a corresponding variable, taking the current time when the first upd message is received as an effective calculation starting time, setting the access times as 1, setting the starting time as the current time, and setting the duration as 0:

total_count＝1；

start_time＝time(NULL)；

last_length＝0；

in the subsequent statistical calculation, last _ length is the duration of the duration from the start calculation time start _ time to the current _ time at the current time.

For the subsequently received message of the statistical data item, the access times are calculated according to the relationship between the receiving time of the message and the set statistical time period, and the access times are divided into three conditions:

(1) when the receiving time falls within the first statistical time period, the number of accesses is increased by 1 on the original basis, and the duration is the difference between the current time and the effective starting calculation time (assuming that the set statistical time period duration is set to 60 seconds, then there are 60 statistical time periods in one hour, and the "first statistical time period" can be regarded as a certain 60-second time).

(2) When the reception time falls between a second statistical time period of equal length after said first statistical time period, the effective start calculation time will be re-recorded. And taking the receiving time as the end point of the new statistical time period, and recording the corresponding starting point time as effective calculation starting time. The access times at this time are the access times in a new statistical time period, and the calculation method is to calculate the QPS in the original statistical time period, multiply the QPS by the time of the original statistical time period in the new statistical time period, and increase by 1, where the duration is the statistical time period duration.

(3) When the receiving time falls after the second statistical time period, the current receiving time is set as the effective starting calculation time, the access times are reset to be 1, the duration is 0, and the flow is calculated again after the initialization.

Taking the set statistical time period as 60 seconds as an example, the pseudo code of the corresponding algorithm is as follows:

diff＝current_time-start_time；

if (diff < 60X) { # when the receive time falls within a first statistical time period

total_count++；last_length＝diff；

} else if (diff > 2X 60X) { # when the receive time falls after a second statistical time period

total_count＝1；

start_time＝current_time；

last_length＝0；

Else when the reception time falls within a second statistical time period

start_time＝current_time-60*x；

total_count＝total_count–total_count/60/X*(diff-60*X)+1；

last_length＝60*X；

}

As for the setting of the statistical time, both short-period statistics and long-period statistics may be set. The short-time statistics are to prevent dos fast attacks, that is, a large number of connection requests impact the server in a short time, so that all available resources are consumed, and eventually, the computer cannot process the requests of the legitimate users any more. And long-time statistics are used for preventing dos slow attack.

The summary statistical processing and blacklist generation by the summarizer 103 are described in detail below.

The aggregator 103 reads the statistical results of the number of requests of all aggregators according to the specific classification, aggregates the statistical results, generates a blacklist according to a predefined policy, and asynchronously loads the blacklist to the load balancer 101.

The structure and the specific operation principle of the aggregator 103 are shown in fig. 3.

The aggregator 103 includes a receiving portion 1031, a generating portion 1032, a listening portion 1033, a configuration file 1036, and a database 1037.

According to an embodiment of the present invention, the receiving part 1031 may create a receiving thread for each aggregation machine, and the receiving thread reads the statistical data of the number of requests for different data item combinations from the Web page output of each aggregation machine at regular intervals (e.g., one minute). The statistical data is, for example, data in JSONSG format.

The generation section 1032 of the totalizer 103 totalizes the data received from the plurality of aggregation machines for predetermined data statistical items.

The different combinations of statistical data items include userid + ip + uri, userid + uri.

The request numbers of the different data item combinations are respectively stored in the corresponding hashmaps. Hashmap is the implementation of a Map interface based on a hash table, and is used for functions such as storage, searching and the like in computer programming.

When the number of requests for a certain combination of statistical data items exceeds a predefined blacklist threshold in the configuration file, such as 0.5QPS (query per second), userid or ip is added to the blacklist, resulting in a current blacklist.

According to one embodiment, the binning thread in the generation section 1032 stores the summarized data in the hashmap into the database 1037. According to an embodiment of the invention, in order to not lock the shared hashmap between the warehousing thread and the receiving thread, a dual hashmap is used.

In addition, the monitoring terminal 200 of the system can directly acquire the data of the database 1037 through an HTTP protocol, and the data is exported in a JSONS format and does not pass through the summarizer 103.

Listening portion 1033 of aggregator 103 is used to provide blacklists to load balancer 101. When load balancer 101 makes a request, snooping section 1033 acquires the blacklist and sends the blacklist together with the whitelist to load balancer 101.

According to an embodiment of the present invention, the aggregator 103 may adopt a mode in which a master server and a slave server work together, where one is the master server as a normal working machine and the other is the slave server as a data backup machine, and when the master server fails, the slave server will automatically take over all the work of the master server, thereby ensuring high reliability of the system.

The invention has been described in connection with the drawings and the specific embodiments. The embodiments in the drawings and description are illustrative of the invention and are not to be construed as limiting the invention. The scope of the invention is defined by the claims.

Claims (8)

1. A system for restricting access requests, the system comprising a load balancer, an aggregator, and an aggregator,

the load balancer receives the access requests from the network, analyzes each received access request into a UDP message, and transmits the UDP message to the aggregator;

the aggregator makes statistics on the received UDP messages according to the predefined statistical data item combination and the predefined statistical duration, and sends statistical results to the aggregator according to the request of the aggregator;

the aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request, wherein the blacklist defines processing actions for respective access requests,

wherein the load balancer determines a processing action for each received access request according to a current blacklist;

the aggregator comprises a plurality of aggregators operating independently of each other, the load balancer transmits each UDP message to one aggregator, each aggregator performs the statistics respectively,

the aggregator receives the statistics from the plurality of aggregators, generates aggregated statistics, and

and the blacklist is generated according to the summarized statistical result and the predefined rule.

2. The system according to claim 1, wherein the statistical result gives the cumulative number of UDP packets containing the predefined combination of statistical data items during the predefined statistical duration.

3. The system of claim 1, wherein

The system further comprises a predefined white list defining access requests that are to be granted, and

the load balancer determines the processing action of the access request on the received access request as follows:

when the access request matches the white list or does not match the current black list, then the access request is released,

and when the access request does not match the white list and the current black list, operating the access request according to the processing action specified by the corresponding record in the black list.

4. The system of claim 1, wherein the processing action specified in the blacklist includes one of: interception, redirection, and latency.

5. A method performed in a system for restricting access requests, the system comprising a load balancer, an aggregator, and an aggregator, the method comprising:

the load balancer receives the access request from the network, analyzes the received access request into a UDP message, and transmits the UDP message to the aggregator;

the load balancer determines a processing action for the received access request according to a current blacklist, wherein the blacklist defines processing actions for the access request,

wherein the content of the first and second substances,

the aggregator receives the UDP messages sent by the load balancer, counts the received UDP messages according to a predefined statistical data item combination and a predefined statistical duration to generate statistical results, and sends the statistical results to the aggregator according to the request of the aggregator;

the aggregator requests statistics from the aggregator at predetermined request time intervals, receives the statistics from the aggregator, generates a blacklist of access requests based on the received statistics and predefined rules, and sends the blacklist to the load balancer based on the load balancer's request;

the aggregator comprises a plurality of aggregators operating independently of each other, the load balancer transmits each UDP message to one aggregator, each aggregator performs the statistics respectively,

the aggregator receives the statistics from the plurality of aggregators, generates aggregated statistics, and

and the blacklist is generated according to the summarized statistical result and the predefined rule.

6. The method according to claim 5, wherein the statistical result gives the cumulative number of UDP packets containing the predefined statistical data item combination during the predefined statistical time period.

7. The method of claim 5, wherein

The system further comprises a predefined white list defining access requests that are to be granted, and

the load balancer determines the processing action of the access request on the received access request as follows:

when the access request matches the white list or does not match the current black list, then the access request is released,

and when the access request does not match the white list and the current black list, operating the access request according to the processing action specified by the corresponding record in the black list.

8. The method of claim 5, wherein the processing action specified in the blacklist includes one of: interception, redirection, and latency.