CN110233774A - A kind of Distributed probing method and system of Socks proxy server - Google Patents

Abstract

The invention discloses a distributed detection method and system of a Socks proxy server, belonging to the field of network resource space detection, including: if the host to be tested opens the service, then perform subsequent steps; otherwise, the detection ends; The GET request message is sent to the host to be tested through the preset ports, and the response information of each port is obtained; the response information of each port is analyzed, and if the first characteristic string is not included and the second characteristic string is included, then it is determined The host has opened the Socks proxy service through the corresponding port; otherwise, it is determined that the host has not opened the Socks proxy service through the corresponding port; if the host has opened the Socks proxy service through any port, then identify the host to be tested as a Socks proxy server; otherwise, identify the host to be tested The host is a non-Socks proxy server; the probe ends. The invention can effectively solve the problem of low recognition rate of the existing Socks proxy server detection method, and accelerates the detection speed through a distributed detection system.

Description

Distributed detection method and system for a Socks proxy server

technical field

The invention belongs to the field of network resource space detection, and more specifically relates to a distributed detection method and system of a Socks proxy server.

Background technique

With the rapid development of the network society, various industries are relying on the Internet to handle related businesses, and our daily activities such as chatting, shopping, entertainment and learning are closely integrated with the Internet, and the Internet is closely related to our lives. Although the network facilitates our life, but at the same time, the security problems cannot be ignored.

The Internet is a highly open field. At present, both enterprises and users can build their own proxy servers. Through proxy servers, users can obtain resources that are difficult to access under normal circumstances, such as Google Scholar and YouTube. In addition, proxy servers generally have relatively large The buffer zone can speed up access to network resources, and can also hide your private information. Based on the above purposes, proxy servers such as HTTP proxy, Telnet proxy, PPTP proxy, L2TP proxy, and Socks proxy have been widely used in the Internet. While proxy servers bring convenience to Internet users, they also have certain security risks. Many network attacks and illegal information transmission are realized through illegal agents. For example, hackers use proxy servers to carry out DDoS attacks to hide their information, criminals use proxy servers for gambling and fraud information transmission. Therefore, the study of proxy server identification method is not only of great significance for further use, but also can provide technical guarantee for network security. Among many proxy servers, Socks proxy server is currently the most widely used proxy server, and its detection is relatively difficult. Therefore, it is of great significance to study Socks proxy detection methods in production and life.

At present, mainstream detection tools, such as Zmap, ProxyBroker, etc., can detect Socks proxy servers. However, these detection tools are based on the Socks protocol request response packet analysis method, and can only identify Socks proxy servers without account passwords. The Socks proxy server with the account number and password, when actually detecting, the detection data packets sent are often directly discarded by the host to be tested, and no response information can be obtained, which leads to the inability to effectively identify the Socks proxy server. Generally speaking, the recognition rate of existing Socks proxy server detection methods is low.

Contents of the invention

Aiming at the defects and improvement needs of the prior art, the present invention provides a distributed detection method and system for a Socks proxy server, aiming at solving the problem of low recognition rate of the existing Socks proxy server detection method.

For achieving the above object, according to the first aspect of the present invention, a kind of detection method of Socks proxy server is provided, is used to identify whether a known host to be tested of an ip is a Socks proxy server, comprising:

(1) Judging whether the host computer to be tested is open for service, if so, then proceed to step (2); otherwise, proceed to step (6);

(2) multiple ports for establishing communication connections are preset on the host computer to be tested;

(3) Send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

(4) parse the response information of each port, if it does not contain the first characteristic character string, and contains the second characteristic character string, then it is determined that the host computer to be tested has opened the Socks proxy service through the corresponding port; otherwise, it is determined that the host computer to be tested has not Open the Socks proxy service through the corresponding port;

(5) if the host to be tested has opened the Socks proxy service through any port, then identify the host to be tested as a Socks proxy server; otherwise, identify the host to be tested as a non-Socks proxy server;

(6) The detection ends;

Wherein, both the first characteristic string and the second characteristic character string are pre-collected character strings according to the proxy characteristics, the first characteristic character string is used to eliminate interference from other proxy servers, and the second characteristic character string is used to identify the Socks proxy server.

The present invention detects the Socks proxy server based on the GET request message of the HTTP protocol, and identifies whether the host to be tested is a Socks proxy server according to whether the response information of the port of the host to be tested contains a characteristic character string. A Socks proxy server or a Socks proxy server without an account password will generate response information correspondingly to the GET request message of the HTTP protocol. Therefore, the present invention can identify both the Socks proxy server without an account password and the one with an account password. The Socks proxy server effectively solves the problem of low recognition rate of the existing Socks proxy server detection method.

Further, the first characteristic string is 'SSH', 'FTP', '\xff\xfd\x18\xff\xfd\xff\xfd#\xff\xfd', 'SMTP' or 'HTTP'.

Further, the second feature string is '\x05\x00', '\x00[\x00\x00\x00\x00\x00\x00' or 'www.herokucdn.com'.

The above-mentioned first characteristic character string is that other proxy servers such as SSH, FTP, SMTP, HTTP produce the response information for the GET request message of HTTP protocol, the unique character string; The unique character string in the GET request message generation response information; Since other proxy servers such as SSH, FTP, SMTP, HTTP can interfere with the detection of the Socks proxy server, the present invention will only not include the above-mentioned first characteristic character string And the host to be tested containing the second characteristic character string is identified as the Socks proxy server, and the interference of other proxy servers is avoided during the process of identifying the Socks proxy server, so the identification accuracy of the present invention is relatively high.

Further, the detection method of the Socks proxy server provided by the present invention also includes: if the identification host to be tested is a Socks proxy server, then obtain the spatial location of the host to be tested according to its ip, thereby realizing the physical location of the Socks proxy server;

The invention further realizes the positioning of the Socks proxy server according to the IP, and provides convenience for the supervision of the Socks proxy server.

According to the second aspect of the present invention, a kind of distributed detection method of Socks proxy server is provided, is used to identify whether the known host to be tested of large-scale ip is a Socks proxy server, comprising:

Divide all hosts to be tested into multiple host sets to be tested;

For each set of hosts to be tested, utilize the detection method of the Socks proxy server provided by the first aspect of the present invention to detect the hosts to be tested in turn to identify the Socks proxy server therein; to the detection of all host sets to be tested in parallel implement.

The above-mentioned distributed detection method of the Socks proxy server divides a plurality of hosts to be tested into a set of hosts to be tested, so that the detection of the hosts to be tested in the set is executed serially, and the detection of multiple sets is executed in parallel, thereby effectively improving detection rate.

According to the third aspect of the present invention, a kind of detection system of Socks proxy server is provided, is used to identify whether a host to be tested known by ip is a Socks proxy server, comprising: pre-judgment module, port setting module, transceiver module, Analysis module and identification module;

A pre-judgment module is used to judge whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection;

The port setting module is used to preset a plurality of ports for establishing communication connections on the host to be tested when the prediction module determines that the host to be tested is open for service;

The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

Parsing module, for parsing the response information of each port, and when the response information does not contain the first characteristic character string, and when the second characteristic character string is included, it is determined that the host to be tested has opened the Socks proxy service through the corresponding port; and In other cases, it is determined that the host to be tested has not opened the Socks proxy service through the corresponding port;

The identification module is used to identify the host to be tested as a Socks proxy server when the analysis module determines that the host to be tested has opened the Socks proxy service through any port; and in other cases, identify the host to be tested as a non-Socks proxy server;

Wherein, both the first characteristic string and the second characteristic character string are pre-collected character strings according to the proxy characteristics, the first characteristic character string is used to eliminate interference from other proxy servers, and the second characteristic character string is used to identify the Socks proxy server.

According to the fourth aspect of the present invention, a kind of distributed detection system of Socks proxy server is provided, is used for identifying whether the host computer to be tested known in large-scale ip is a Socks proxy server, including: database, central scheduling node and multiple parallel task execution node;

The central scheduling node is used to divide all the hosts to be tested into multiple host sets to be tested, and distribute the detection tasks of each host set to be tested to multiple different task execution nodes, so that the detection tasks of each host set to be tested parallel execution;

The task execution node is used to detect the hosts to be tested in sequence by using the detection method of the Socks proxy server provided by the first aspect of the present invention to identify the Socks proxy server therein. , and send the task execution result back to the central scheduling result, and store the detection result persistently in the database; the execution result is used to display the success or failure of the task execution, and the detection result includes the ip address, port, and spatial location information of the Socks proxy server;

The central scheduling node is also used to receive the execution results returned by each task execution node, and transfer the failed tasks from the original task execution node to other task execution nodes to continue execution according to the execution results.

In the distributed detection system of the above-mentioned Socks proxy server, the central scheduling node and the task execution node constitute the Master-Slave (master-slave) model, through scheduling, each task execution node is made to perform the detection task of the Socks proxy server in parallel, by This can effectively improve the detection rate of a large-scale host to be tested.

Further, the central scheduling node includes: a task distribution module and a failover module;

The task distribution module is used to divide all the hosts to be tested into multiple host sets to be tested, and distribute the detection tasks of each host set to be tested to a plurality of different task execution nodes, so that the detection tasks of each host set to be tested parallel execution;

The failover module is used to receive the execution result returned by each task execution node, and transfer the failed task from the original task execution node to another task execution node to continue execution according to the execution result.

The above distributed detection system implements a failover mechanism by transferring failed detection tasks from the original task execution node to another task execution node, thereby ensuring that the detection of each host to be tested is normally performed.

Further, the task execution node includes: a pre-judgment module, a port setting module, a transceiver module, an analysis module, an identification module, a storage module, and a return module;

A prediction module, used to judge whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection of the host to be tested;

The port setting module is used to preset a plurality of ports for establishing communication connections on the host to be tested when the prediction module determines that the host to be tested is open for service;

The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

Parsing module, for parsing the response information of each port, and when the response information does not contain the first characteristic character string, and when the second characteristic character string is included, it is determined that the host to be tested has opened the Socks proxy service through the corresponding port; and In other cases, it is determined that the host to be tested has not opened the Socks proxy service through the corresponding port;

The identification module is used to identify the host to be tested as a Socks proxy server when the analysis module determines that the host to be tested has opened the Socks proxy service through any port; and in other cases, identify the host to be tested as a non-Socks proxy server;

The storage module is used to persistently store the detection results in the database;

The return module is used to return the execution result to the central scheduling node;

Wherein, both the first characteristic string and the second characteristic character string are pre-collected character strings according to the proxy characteristics, the first characteristic character string is used to eliminate interference from other proxy servers, and the second characteristic character string is used to identify the Socks proxy server.

Generally speaking, through the above technical solutions conceived by the present invention, the following beneficial effects can be obtained:

(1) The detection method of the Socks proxy server provided by the present invention and the detection system of the Socks proxy server utilize the GET request message based on the HTTP protocol to carry out the detection of the Socks proxy server, and according to whether the response information of the host port to be tested contains A specific characteristic string is used to identify whether the host to be tested is a Socks proxy server, because whether it is a Socks proxy server with an account password or a Socks proxy server without an account password, it will generate response information for the GET request message of the HTTP protocol accordingly Therefore, the present invention can identify both the Socks proxy server without account password and the Socks proxy server with account password, thereby effectively solving the problem of low recognition rate of the existing Socks proxy server detection method.

(2) the detection method of the Socks proxy server provided by the present invention and the detection system of the Socks proxy server accurately set the first characteristic character string for eliminating interference and the second characteristic character string for identifying the Socks proxy server, And do not contain above-mentioned first characteristic character string and the host computer to be tested that contains second characteristic character string is identified as Socks proxy server, make the present invention can in the process of identifying Socks proxy server, avoid the interference of other proxy servers, therefore, The recognition accuracy of the present invention is relatively high.

(3) The detection method of the Socks proxy server provided by the present invention and the detection system of the Socks proxy server can also carry out spatial positioning to the Socks proxy server according to ip, which provides convenience for the supervision of the Socks proxy server.

(4) the distributed detection method of the Socks proxy server provided by the present invention and the distributed detection system of the Socks proxy server, based on the detection method of the Socks proxy server provided by the present invention, and execute in multiple tasks in a distributed manner The nodes detect the Socks proxy server at the same time, so that tasks can be executed in parallel. Therefore, the invention can effectively improve the detection rate on the basis of ensuring a high recognition rate.

(5) the distributed detection system of the Socks proxy server provided by the present invention, when its central dispatching node fails to the detection of any host computer to be tested, the detection task of this host computer to be tested will be transferred from the original task execution node to Another task execution node is used to restart the detection task of the host to be tested, thereby implementing a failover mechanism to ensure that the detection of each host to be tested is performed normally.

Description of drawings

Fig. 1 is the detection method flowchart of the Socks proxy server that the embodiment of the present invention provides;

Fig. 2 is the distributed detection method schematic diagram of the Socks proxy server that the embodiment of the present invention provides;

Fig. 3 is the distributed detection system module schematic diagram of the Socks proxy server that the application example of the present invention provides;

Fig. 4 is a schematic diagram of the execution flow of the distributed detection system of the Socks proxy server provided by the application example of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

In order to realize the detection of Socks proxy servers with account passwords, to solve the problem of low recognition rate of existing Socks proxy server detection methods, the invention provides a detection method of Socks proxy servers, which is used to identify a known IP Whether the host to be tested is a Socks proxy server, as shown in Figure 1, the detection method includes:

(1) Judging whether the host computer to be tested is open for service, if so, then proceed to step (2); otherwise, proceed to step (6);

Optionally, the ip of the host to be tested can be combined with network test commands such as ping and tracert to detect the host to be tested, for example, to send the test command "ping-c 3-w 6"+ip to the host to be tested; if the host to be tested If the corresponding information is returned, it means that the host to be tested has opened the service; if the host to be tested does not return any information, it means that the host to be tested has not opened the service;

Only when the host to be tested opens the service, continue to perform the subsequent Socks proxy server detection task, which can avoid invalid operations and greatly save the time required for detection;

(2) multiple ports for establishing communication connections are preset on the host computer to be tested;

In the Internet, not all port numbers of 1-65535 can be used for Socks to open the proxy. By setting common ports, the detection speed can be saved and the detection efficiency can be improved. In the embodiment of the present invention, the preset ports include 27274, 443,2333,80,8080,4145,6667,9999,8082,9050,3128,8388,8000,8888,8088,1080,9000,53281,54566,808,8081,8118,65103,21071,1080,64312, 53281, 54321;

It should be understood that in other application scenarios, it can also be set according to the actual port opening status of the detected host;

(3) Send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

The constructed GET request message can be a GET request message with any content conforming to the HTTP protocol. For example, the content of a GET request message constructed according to a blank website is as follows:

'GET/HTTP/1.1\r\nHost:hm.baidu.com\r\n\r\n';

In the request message, hm.baidu.com is the URL of the blank website; the above content is only an exemplary description of the GET request message, and should not be understood as the only limitation of the present invention;

(4) parse the response information of each port, if it does not contain the first characteristic character string, and contains the second characteristic character string, then it is determined that the host computer to be tested has opened the Socks proxy service through the corresponding port; otherwise, it is determined that the host computer to be tested has not Open the Socks proxy service through the corresponding port;

Wherein, the first characteristic character string and the second characteristic character string are character strings pre-collected according to the agent characteristics, the first characteristic character string is used to eliminate the interference of other proxy servers, and the second characteristic character string is used to identify the Socks proxy server;

Specifically, the first characteristic character string and the second characteristic character string are pre-defined character strings according to the response characteristics of the proxy server to the above GET request message;

In an optional implementation, the first feature string is 'SSH', 'FTP', '\xff\xfd\x18\xff\xfd\xff\xfd#\xff\xfd', 'SMTP' or ' HTTP';

The first characteristic character string is that other proxy servers such as SSH, FTP, SMTP, HTTP produce in the response information for the GET request message of HTTP protocol, the peculiar character string; These proxy servers can interfere with the detection of Socks proxy server;

In an optional implementation, the second feature string is '\x05\x00', '\x00[\x00\x00\x00\x00\x00\x00' or 'www.herokucdn.com';

The second characteristic character string is that the Socks proxy server produces the unique character string in the return information for the GET request message of the HTTP protocol;

By not including the above-mentioned first characteristic string and including the host to be tested of the second characteristic character string identified as the Socks proxy server, in the process of identifying the Socks proxy server, the interference of other proxy servers can be avoided, so the present invention High recognition accuracy;

It should be noted that the first characteristic character string and the second characteristic character string are not limited to the above-mentioned ones, and other character strings that can identify relevant interfering servers can also be included in the first characteristic character string; similarly, other The string that can identify the Socks proxy server can also be included in the second feature string; in addition, the above-mentioned first feature string and second feature string can also be updated according to actual application scenarios;

(5) if the host to be tested has opened the Socks proxy service through any port, then identify the host to be tested as a Socks proxy server; otherwise, identify the host to be tested as a non-Socks proxy server;

(6) Detection ends.

The present invention utilizes the GET request message based on the HTTP protocol to carry out the detection of the Socks proxy server, and according to whether the response information of the port on the host computer to be tested contains a specific characteristic character string to identify whether the host computer to be tested is a Socks proxy server, because whether it is A Socks proxy server with an account password or a Socks proxy server without an account password will generate return information correspondingly to the GET request message of the HTTP protocol. Therefore, the present invention can identify both the Socks proxy server without an account password and the The Socks proxy server with the account password effectively solves the problem of low recognition rate of the existing Socks proxy server detection method.

In order to further realize the spatial location of the Socks proxy server, optionally, the detection method of the above-mentioned Socks proxy server may also include:

If the host to be tested is identified as a Socks proxy server, the spatial location of the host to be tested is obtained according to its ip, so as to realize the physical location of the Socks proxy server, which can provide convenience for the supervision of the Socks proxy server;

Optionally, after identifying the host to be tested as the Socks proxy server, obtain the corresponding latitude and longitude according to its ip bar with the API of Baidu map, the API of Google Maps or other APIs, thereby obtaining the geographic location of the Socks proxy server; further, According to application requirements, the city or other administrative divisions to which the Socks proxy server belongs can be judged based on the obtained latitude and longitude of the Socks proxy server. The specific method can also be obtained through the relevant map API.

Based on the detecting method of above-mentioned Socks proxy server, the present invention also provides a kind of distributed detecting method of Socks proxy server, is used for identifying whether For the Socks proxy server, the distributed detection method includes:

Divide all hosts to be tested into multiple host sets to be tested;

For each host set to be tested, the detection method of the above-mentioned Socks proxy server is used to detect the host to be tested in turn, to identify the Socks proxy server wherein; the detection of all host sets to be tested is performed in parallel;

The above-mentioned distributed detection method of the Socks proxy server divides a plurality of hosts to be tested into a set of hosts to be tested, so that the detection of the hosts to be tested in the set is executed serially, and the detection of multiple sets is executed in parallel, thereby effectively improving Probing rate; usually, in order to maximize parallelism, the number of hosts to be tested contained in each set of hosts to be tested is equal or similar.

The present invention also provides a detection system of a Socks proxy server, which is used to realize the detection method of the above-mentioned Socks proxy server. The detection system includes: a pre-judgment module, a port setting module, a transceiver module, an analysis module and an identification module;

A pre-judgment module is used to judge whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection;

The port setting module is used to preset a plurality of ports for establishing communication connections on the host to be tested when the prediction module determines that the host to be tested is open for service;

The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

Parsing module, for parsing the response information of each port, and when the response information does not contain the first characteristic character string, and when the second characteristic character string is included, it is determined that the host to be tested has opened the Socks proxy service through the corresponding port; and In other cases, it is determined that the host to be tested has not opened the Socks proxy service through the corresponding port;

The identification module is used to identify the host to be tested as a Socks proxy server when the analysis module determines that the host to be tested has opened the Socks proxy service through any port; and in other cases, identify the host to be tested as a non-Socks proxy server;

Wherein, the first characteristic character string and the second characteristic character string are character strings pre-collected according to the agent characteristics, the first characteristic character string is used to eliminate the interference of other proxy servers, and the second characteristic character string is used to identify the Socks proxy server;

In the embodiment of the present invention, for the specific implementation manner of each module, reference may be made to the description in the foregoing method embodiment, which will not be repeated here.

According to the fourth aspect of the present invention, a kind of distributed detection system of Socks proxy server is provided, for realizing the distributed detection method of above-mentioned Socks proxy server, as shown in Figure 2, this distributed detection system comprises: database, center Scheduling nodes and multiple parallel task execution nodes;

The central scheduling node is used to divide all the hosts to be tested into multiple sets of hosts to be tested, and distribute the detection tasks of each set of hosts to be tested to a plurality of different task execution nodes, so that each set of hosts to be tested The detection tasks are executed in parallel;

The task execution node is used to detect the hosts to be tested by using the detection method of the above-mentioned Socks proxy server to identify the Socks proxy server therein, and return the task execution result to the set of hosts to be tested. Pass it to the central scheduling result, and store the detection result persistently in the database; the execution result is used to display the success or failure of the task execution, and the detection result includes the IP address, port, and spatial location information of the Socks proxy server;

The central scheduling node is also used to receive the execution results returned by each task execution node, and transfer the failed tasks from the original task execution node to other task execution nodes to continue execution according to the execution results;

As shown in Figure 3, in the embodiment of the present invention, the central scheduling node specifically includes: a task distribution module and a failover module;

The task distribution module is used to divide all the hosts to be tested into multiple sets of hosts to be tested, and distribute the detection tasks of each set of hosts to be tested to a plurality of different task execution nodes, so that each set of hosts to be tested The detection tasks are executed in parallel;

The failover module is used to receive the execution result returned by each task execution node, and transfer the failed task from the original task execution node to another task execution node to continue execution according to the execution result;

Task execution nodes include: pre-judgment module, port setting module, transceiver module, analysis module, identification module, storage module and return module;

A prediction module, used to judge whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection of the host to be tested;

The port setting module is used to preset a plurality of ports for establishing communication connections on the host to be tested when the prediction module determines that the host to be tested is open for service;

The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port;

Parsing module, for parsing the response information of each port, and when the response information does not contain the first characteristic character string, and when the second characteristic character string is included, it is determined that the host to be tested has opened the Socks proxy service through the corresponding port; and In other cases, it is determined that the host to be tested has not opened the Socks proxy service through the corresponding port;

The identification module is used to identify the host to be tested as a Socks proxy server when the analysis module determines that the host to be tested has opened the Socks proxy service through any port; and in other cases, identify the host to be tested as a non-Socks proxy server;

The storage module is used to persistently store the detection results in the database;

The return module is used to return the execution result to the central scheduling node;

Wherein, the first characteristic character string and the second characteristic character string are character strings pre-collected according to the agent characteristics, the first characteristic character string is used to eliminate the interference of other proxy servers, and the second characteristic character string is used to identify the Socks proxy server;

In the embodiment of the present invention, for the specific implementation manner of each module, reference may be made to the description in the foregoing method embodiment, which will not be repeated here.

In the distributed detection system of the above-mentioned Socks proxy server, the central scheduling node and the task execution node constitute the Master-Slave (master-slave) model, through scheduling, each task execution node is made to perform the detection task of the Socks proxy server in parallel, by This can effectively improve the detection rate of a large-scale host to be tested.

In an optional embodiment, in the distributed detection system of the above-mentioned Socks proxy server, the task execution node is also used to re-execute all detection tasks distributed therein after each preset time interval; time interval It can be comprehensively set according to the execution time of the detection task and the change of the agent to be detected. The execution time of the detection task is determined according to the actual task execution time. A constant can be set for the change of the agent to be detected. The time interval is the sum of the execution time of the detection task and the constant; for example, For example, it takes 3 hours for a task execution node to execute a task, and the proxy host to be detected is updated every 3 hours, because the agent will often change the address and port, then the corresponding time interval is 6 hours, and the task execution time of the task execution node can be predicted through experiments , but the proxy change is changed by the proxy provider, so a constant can be set in the actual detection, such as a few days, a week or a month;

The ip and port list of the Socks proxy server will be updated, and the task execution node will re-execute all the detection tasks distributed to it after each preset time interval, which can ensure the validity and real-time performance of the detection results.

Applications

Fig. 3 shows an application example of the distributed detection system of the Socks proxy server provided by the present invention, based on this detection system, the execution schematic diagram of the distributed detection method of the Socks proxy server provided by the present invention, as shown in Figure 4 ;

Among them, the Master node is the central scheduling node, and the Slave node is the task execution node; the original data is stored in the database, and the corresponding execution results are stored in the log;

Based on the distributed detection system shown in Figure 3, the distributed detection method of the Socks proxy server provided by the present invention can generally be simplified into the following steps:

1. The user submits the task to the Master node -> 2. The Master node receives the task and distributes the task to the Slave node -> 3. The Slave node starts the detection task -> 4. The Slave node obtains the Socks proxy data and records the log -> 5. The Slave node stores the Socks proxy data in the database -> 6. The Slave node returns the execution result to the Master node -> 7. The Master node stores the execution result as a disk file, and determines whether to adopt a failover mechanism according to the execution result.

The following is a specific description in conjunction with Fig. 3 and Fig. 4:

(S1) The user collects the target server that needs to be detected, mainly collects the open ip segment collection of some cloud service vendors;

(S2) The user Client submits the list of hosts to be detected to the Master node. The Master node queries the number of online Slave nodes and fragments the hosts to be tested. The fragmentation strategy can dynamically use various customized strategies, including polling, randomness, consistency HASH, least frequently used, least recently used, failover, busy transfer and other strategies, generally in order to achieve load balancing, you can choose a consistent HASH strategy, and the Master node will distribute the fragmentation tasks to each Slave according to the selected fragmentation strategy node;

(S3) The Master node communicates with the Slave node through heartbeat information. After receiving the fragment issued by the Master node, it starts the proxy detection task, detects these IPs according to the preset port, and mainly uses network I/O to realize data transmission. The detected information includes ip, the port where the Socks proxy service is open, the latitude and longitude information and the city where the Socks proxy server is located;

(S4) Slave node proxy detection can directly obtain information such as ip, open port, etc., where the geographical location of the website, that is, the latitude and longitude cannot be directly detected and obtained. It needs to call the API of Baidu map or the API of Google map according to the specific ip to obtain, and the website belongs to Similarly, the city needs to call the relevant map API to obtain;

(S5) During the execution of the detection task, if the Slave node fails to execute the detection task for any host to be tested, the task execution node will retry a certain number of times. information;

(S6) The Slave node maintains communication with the Master node while executing the detection task. Once the Slave node fails to perform the task, the Master can monitor the abnormal situation, and the Master node will transfer the failed detection task from the original execution node to another execution node , to implement a failover mechanism;

(S7) Each Slave node stores the collected data in the database. The database needs to be enabled to allow remote connections. In order to ensure security, the database only allows Slave nodes to log in, and the Master node also needs its own database to store task execution logs. , to record the execution status of the task. Once the task fails, you can analyze it by querying the log, and then determine which specific host to be tested failed the detection task, and restart the detection task, thereby ensuring the integrity of data collection.

Those skilled in the art can easily understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (9)

1. a kind of detection method of Socks proxy server, is used to identify whether a known host to be tested of ip is Socks proxy server, it is characterized in that, comprises: (1) Judging whether the host computer to be tested is open for service, if so, then proceed to step (2); otherwise, proceed to step (6); (2) preset a plurality of ports for establishing communication connections on the host computer to be tested; (3) sending the pre-configured GET request message based on the HTTP protocol to the host to be tested respectively through preset ports, and obtaining the response information of each port; (4) parse the response information of each port, if it does not include the first characteristic character string, and includes the second characteristic character string, then determine that the host to be tested has opened the Socks proxy service through the corresponding port; otherwise, determine the The host to be tested has not opened the Socks proxy service through the corresponding port; (5) if the host to be tested has opened the Socks proxy service through any port, then identify the host to be tested as a Socks proxy server; otherwise, identify the host to be tested as a non-Socks proxy server; (6) The detection ends; Wherein, both the first characteristic character string and the second characteristic character string are character strings pre-collected according to agent characteristics, the first characteristic character string is used to exclude interference from other proxy servers, and the second characteristic character string String used to identify the Socks proxy server. 2. the detection method of Socks proxy server as claimed in claim 1, is characterized in that, described first character string is ' SSH ', ' FTP ', '\xff\xfd\x18\xff\xfd\xff\ xfd#\xff\xfd', 'SMTP' or 'HTTP'. 3. the detecting method of Socks proxy server as claimed in claim 1, is characterized in that, described second character string is '\x05\x00', '\x00[\x00\x00\x00\x00\x00\ x00' or 'www.herokucdn.com'. 4. the detection method of the Socks proxy server as described in any one of claim 1-3, is characterized in that, also comprises: if identifying described host computer to be tested is Socks proxy server, then obtains described host computer to be tested according to its ip spatial location, so as to realize the physical location of the Socks proxy server. 5. a kind of distributed detection method of Socks proxy server, is used to identify whether the known mainframe to be tested of large-scale ip is Socks proxy server, it is characterized in that, comprises: Divide all hosts to be tested into multiple host sets to be tested; For each set of hosts to be tested, utilize the detection method of the Socks proxy server described in any one of claims 1-4 to detect the hosts to be tested in turn, to identify the Socks proxy server therein; to all guessed host sets The probes are executed in parallel. 6. a detection system of a Socks proxy server, used to identify whether a known host to be tested of an ip is a Socks proxy server, it is characterized in that, comprising: pre-judgment module, port setting module, transceiver module, analysis module and identification module; The prediction module is used to judge whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection; The port setting module is configured to preset a plurality of ports for establishing a communication connection on the host to be tested when the predicting module determines that the host to be tested is open for service; The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port; The parsing module is used for parsing the response information of each port, and when the response information does not contain the first characteristic character string and contains the second characteristic character string, it is determined that the host to be tested has opened the Socks proxy through the corresponding port service; and in other cases, determine that the host to be tested does not open the Socks proxy service through the corresponding port; The identification module is used to identify the host to be tested as a Socks proxy server when the parsing module determines that the host to be tested has opened a Socks proxy service through any port; and in other cases, identify the host to be tested The test host is a non-Socks proxy server; Wherein, both the first characteristic character string and the second characteristic character string are character strings pre-collected according to agent characteristics, the first characteristic character string is used to exclude interference from other proxy servers, and the second characteristic character string String used to identify the Socks proxy server. 7. a distributed detection system of a Socks proxy server, used to identify whether the known host to be tested of large-scale ip is a Socks proxy server, it is characterized in that, comprising: database, central scheduling node and a plurality of parallel task execution nodes ; The central scheduling node is used to divide all the hosts to be tested into multiple sets of hosts to be tested, and distribute the detection tasks of each set of hosts to be tested to a plurality of different task execution nodes, so that in each set of hosts to be tested The detection tasks are executed in parallel; The task execution node is used to detect the hosts to be tested in sequence by using the detection method of the Socks proxy server described in any one of claims 1-4 for the hosts to be tested. The Socks proxy server wherein returns the task execution result to the central scheduling node, and stores the detection result persistently in the database; the execution result is used to display task execution success or failure, and the detection result includes Socks proxy server ip address, port, spatial location information; The central scheduling node is also used to receive the execution results returned by each task execution node, and transfer the failed task from the original task execution node to another task execution node to continue execution according to the execution result. 8. the distributed detection system of Socks proxy server as claimed in claim 7, is characterized in that, described central scheduling node comprises: task distribution module and failover module; The task distribution module is used to divide all the hosts to be tested into multiple host sets to be tested, and distribute the detection tasks of each host set to be tested to a plurality of different task execution nodes, so that each host set to be tested The detection tasks are executed in parallel; The failover module is configured to receive the execution result returned by each task execution node, and transfer the failed task from the original task execution node to another task execution node to continue execution according to the execution result. 9. the distributed detection system of Socks proxy server as claimed in claim 7 or 8, it is characterized in that, described task execution node comprises: pre-judgment module, port setting module, transceiver module, parsing module, identification module, storage module and return module; The prediction module is used to determine whether the host to be tested is open for service, and when the host to be tested is not open for service, end the detection of the host to be tested; The port setting module is configured to preset a plurality of ports for establishing a communication connection on the host to be tested when the predicting module determines that the host to be tested is open for service; The transceiver module is used to send the pre-configured GET request message based on the HTTP protocol to the host to be tested through each preset port, and obtain the response information of each port; The parsing module is used for parsing the response information of each port, and when the response information does not contain the first characteristic character string and contains the second characteristic character string, it is determined that the host to be tested has opened the Socks proxy through the corresponding port service; and in other cases, determine that the host to be tested does not open the Socks proxy service through the corresponding port; The identification module is used to identify the host to be tested as a Socks proxy server when the parsing module determines that the host to be tested has opened a Socks proxy service through any port; and in other cases, identify the host to be tested The test host is a non-Socks proxy server; The storage module is configured to persistently store detection results in the database; The return module is used to return the execution result to the central scheduling node; Wherein, both the first characteristic character string and the second characteristic character string are character strings pre-collected according to agent characteristics, the first characteristic character string is used to exclude interference from other proxy servers, and the second characteristic character string String used to identify the Socks proxy server.