TW201815142A - Method for detecting domain flux botnet through proxy server log - Google Patents

Abstract

The invention relates to a method for detecting a domain flux botnet through a proxy server log. A website analysis module performs the following steps, including: accessing proxy server log data; and then filtering the log data according to a node filtering algorithm. The conventional user-agent connection is filtered out; and according to a website grouping algorithm, grouping is performed in the remaining records to obtain the associated set of external websites in the log content; the connection topology of the external website is obtained, and a The connection feature algorithm determines whether the collection conforms to the connection characteristics of domain flux; and filters out the domains connected to normal applications or CDNs in the remaining records, and the remaining domains are judged to be domain flux botnets.

Description

   Method for detecting domain flux botnet through proxy server logs   

The present invention relates to a method for detecting a malicious network through a proxy log, and more particularly, to a method for detecting a domain flux botnet through a proxy server log.

The collation and analysis of the log data of the proxy server is the basis of today's information security control. With the advent of the huge amount of data, how to extract valuable information from the huge amount of log data is the responsibility of technical personnel in the field. The main issue, if we can develop a cluster that can screen out malicious domains, and further lock the victim hosts within the enterprise or company, it will have obvious benefits for enterprises or companies that need strict security control.

Such as the Republic of China Patent Bulletin No. 1455546, which is a method and system for detecting malicious domains with rapidly changing domain technologies. It mainly uses router host names and network address autonomous system numbers included in router information. To determine whether it is a malicious domain based on the same name of a specific part of the router host, or if the network packet transmission time is greater than a preset check value, however, its accuracy is incorrect and it has normal applications Possibility of reporting a programmatic error as a malicious domain.

In addition to the previous technology, the field has also developed a set of domain name-related domain results obtained through a web search engine and a WHOIS website query to find suspicious websites that contain only a small number of search results. A collection of related networking domains and the number of search results to determine whether it is a relay domain name of a suspicious botnet. However, it still cannot accurately screen out the domain flux botnet. If the domain flux technology is applied to a botnet, it can Existing DNS services or domain generation algorithms to achieve the technology of associating multiple domain names with the same IP address and evading URL detection.

According to the various malicious website protection methods that are extended to the proxy server log data, there are still many possibilities in response to the changing technology. Each can be improved according to the problem. A screening method for the domain flux botnet, It is the urgent need of people in the field.

The present invention proposes a method for detecting a domain flux botnet through a proxy server log. In order to improve the survival rate of a botnet, attackers often use the domain flux technology to avoid being easily detected and then blocked. However, since the behavior of a malicious program connecting to a specific external website will be recorded in detail in the log data of the proxy server, the idea of the present invention is to analyze the log data of the proxy server and use the result of the network domain collection. After obtaining the union set that conforms to the domain flux connection behavior, a method that may be a domain flux botnet is obtained.

A method for detecting a domain flux botnet through a proxy server log according to the present invention is mainly to execute multiple steps through a website analysis module. First, the website analysis module accesses the proxy server log data, and according to a The node filtering algorithm filters the connection status represented by the user-agent information in each of the log data to filter the log content that belongs to the connection with the conventional website; among them, the node filtering algorithm uses the user This feature of the node's node dimension (degree) is used to filter the famous websites. In detail, this step uses the MapReduce architecture to use the terminal URL of the external website as the key and the user agent as the value. A list of the number of times each external website is linked by different user agents can be efficiently obtained. By filtering websites with a large dimension value in the list, you can initially filter out relatively well-known websites. The present invention retains long-term traffic information External websites whose median dimensional value is less than a preset threshold (for example, the threshold is 10) are intended to protect only the more well-known websites Only relatively small number of user agent (User-agent) of the external connection site usually indicates its specific purpose, these sites have a high probability of external malicious relay station.

The next step is that the website analysis module uses a website grouping algorithm to match the remaining log content of the log content filtered in the previous step to the connection records of external websites that have the same Client IP and user agent connection records. Grouping to find the collection of associations between external websites in the log content; the purpose of this step is to provide information through the traffic information in the logs to establish associations between external websites, which is intended to be used by the same program All external websites linked to are considered as related websites. In detail, the present invention is to group external websites with the same Client IP and user agent connection records in the same collection, which means that these websites are related websites. Because of the characteristics of establishing an external website collection (Client IP and user agent), it can be judged by whether the content of the fields in the log records exactly match (Exact Match), so this website grouping algorithm can also pass Key-value architecture in MapReduce.

Among them, the website grouping algorithm can be grouped by the method of joint set search, that is, after the website analysis module establishes several collections with the connection records of the remaining log content, the following steps are performed: individual connections in each collection Record data is used as an element. If the external website connected to the connection record has the same Client IP and user agent, it is judged as an intersection and merges the collection of the intersection of the elements; deletes the collection whose number of elements is greater than the preset threshold; A set that does not intersect with the remaining sets is determined as an independent set; and the above three steps are repeated until all sets are determined as independent sets.

Following the steps, the website analysis module obtains the client IP connection record in the log content and the client IP and the connection topology of the connected external website; wherein, the present invention uses the above steps to collect the disjoint sets of URLs. After classifying the connection records, the connection records of the same collection will only be connected to the domain of the same collection. The meaning of the connection records of the same collection is generated by programs with the same purpose. Out of the network behavior, this step will obtain the client IP and the connection topology of the external URL, and obtain the client IP connection information in the proxy server log, for example, hash the URL string of disjoint sets ) Method to obtain its group code, with the corresponding connection information of each URL recorded in the connection record, using MapReduce's Multiple Input mechanism to use the URL as the key, you can compare the connection record collection and group corresponding to the group code. Connected topology illustration of group.

Next, the website analysis module presents the relationship between the external website and the associated collection through a connection topology, and uses a connection characteristic algorithm to determine whether the collection meets the connection characteristics of domain flux. Among them, the so-called domain The connection feature of flux refers to the connection through a few Client IPs and the same user-agent to connect to many different external URL domains. In practice, the connection feature algorithm is It is used to calculate the ratio of the number of terminal URLs to the number of client IPs in a collection. If the calculated ratio exceeds a preset threshold, it means that the website of the collection meets the domain flux behavior of domain flux connection.

Finally, the website analysis module extracts a set of domain flux connection characteristics, and filters out the domains that are connected to normal network applications (such as antivirus software) or CDNs. The website analysis module The remaining domains can be judged as a botnet of domain flux and alert.

The above is the method for detecting a domain flux botnet through a proxy server log according to the present invention. Through the possible characteristics of the domain flux botnet, a suspected domain can be mined in a huge amount of long-term log data. An external website of the flux botnet for further precautions.

A‧‧‧Computer

B‧‧‧Computer

C‧‧‧Computer

E‧‧‧Computer

F‧‧‧Computer

G‧‧‧Computer

H‧‧‧Computer

1‧‧‧ attacker

2‧‧‧ attacker

10‧‧‧ cluster

11‧‧‧ relay station

12‧‧‧ relay station

13‧‧‧ relay station

20‧‧‧ cluster

21‧‧‧ relay station

22‧‧‧ relay station

23‧‧‧ relay station

30‧‧‧ relay station

S201 ~ S205‧‧‧Method steps

FIG. 1 is a schematic diagram of a scenario of detecting a domain flux botnet through a proxy server log according to the present invention.

FIG. 2 is a flowchart of a method for grouping external websites by proxy logs according to the present invention.

FIG. 3 is a diagram illustrating an example of a statistical list of the number of user agent connection times according to the present invention.

FIG. 4 is a schematic diagram of an example of converting a simple log data into a collection of external websites according to the present invention.

FIG. 5 is a schematic diagram of an exemplary pushback network connection group according to the present invention.

FIG. 6 is an exemplary schematic diagram of a network behavior topology diagram of the domain flux of the present invention.

FIG. 7 is a schematic diagram of an example of a connection behavior of an antivirus software through a domain flux method.

FIG. 8 is a schematic diagram of an example of proxy server log data.

FIG. 9 is a schematic diagram of an embodiment of screening a botnet that conforms to domain flux by the method of the present invention.

FIG. 10 is a schematic diagram of an embodiment of screening a botnet that conforms to domain flux through the method of the present invention.

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not intended to limit the present invention.

FIG. 1 is a schematic diagram of a scenario of detecting a domain flux botnet through a proxy server log according to the present invention. An attacker can insert a malicious program into a victim host to make it a bot, and the bot can steal. Malicious behaviors such as confidential or sensitive data, and today, in order to improve the survival rate of botnets, attackers will use domain flux technology to avoid checks, but the behavior of malicious programs connecting to specific external websites will be recorded in the proxy In the server log data, the present invention can still parse it out. In Figure 1, the attacker 1 uses a malicious program to change the computer A, computer B, and computer C in the target enterprise's intranet into zombies. Computer A and Malicious programs in computer B and computer C will be connected to relay station 11, relay station 12, and relay station 13, and computer C will connect to a well-known website while performing a normal connection; in addition, attacker 2 uses a malicious program to connect the target enterprise's intranet Computer D and computer E and computer H become zombie computers, and the malicious programs in computer A and computer B and computer C will link to relay station 21, relay station 22, and China Station 23, and computer D is connected to the famous website when the normal connection is performed; among them, computer C and computer D will be connected to the relay station and also to the famous website; in addition, the famous website 30 is connected to the computer C and computer running the normal application program. D, computer E, computer F, and computer G are connected, and the computer and the relay station have a staggered connection relationship, and the botnet bypasses the general inspection through the domain flux. The method of the present invention will be required to be able to efficiently The proxy server's log data uses the records of computer A, computer B, computer E, and computer H that became bots to track down the suspected botnet relay domain.

Next, FIG. 2 is a flowchart of a method for grouping external websites by proxy logs according to the present invention. First, step S201 is to filter famous websites by using user agent dimension values. Through the MapReduce architecture, the website analysis module obtains each external website in the log. A list of the number of times a website has been linked by different user agents. A website with a large dimension value in the filtering list can filter out famous websites first, as shown in Figure 3, which is an example diagram of a statistical list of the number of user agent links. It can be observed that most websites have only been connected by a group of user agents (accounting for 92.77%), and famous websites such as www.google.com are connected by 22825 different user agents. Through the above observations, it can be known from the above observations that by checking the connection records of external websites by different user agents, the present invention can make a preliminary screening for famous websites. According to the example shown in FIG. 1, the famous website 30 will be eliminated in step S201.

Then, please continue to refer to FIG. 2. In step S202, the association set of the external website is obtained through the joint search, which is mainly presented by transforming the proxy server log data into an external website collection. The present invention is based on a website analysis model. The group divides external websites with the same Client IP and user-agent connection records into the same collection, which indicates that these websites are related and can be implemented through the MapReduce Key-value architecture; please refer to Figure 4, which It is a schematic diagram of an example of converting a simple log data into a collection of external websites. As can be observed in Figure 4, it uses Client IP and user agent as keys, and the URL (Dest Url) of the terminal as the value to perform joint set search. Finally, the results of the example log data through the joint set search are two disjoint sets: “CnC1, CnC2, CnC3” and “CnC4, CnC5, CnC6”. For the detailed method of how the joint set search is implemented, This will be explained in the following paragraphs.

Next, step S203 pushes back the network connection group by collection. The website analysis module groups the connection records by disjoint sets of URLs. The connection records of the same group are only connected. To the domain of the same set, the meaning of the connection records of the same group is the network behavior generated by programs with the same purpose; in this step, the website analysis module will obtain the Client IP The topology of the connection with the external website, and obtain the connection information of the Client IP in the proxy server log; as shown in the example diagram of the pushback network connection group in Figure 5, the website analysis model of the present invention After the two disjoint sets of "CnC1, CnC2, CnC3" and CnC4, CnC5, CnC6 "are obtained in a hash manner, the Group-ID is given, and then CnC1 ~ CnC6 of the terminal URL are given according to the set to which they belong. Group-ID, combined with the CnC1 ~ CnC6 connection information recorded in the connection record, and using MapReduce Multiple Input with the URL as the key, you can compare the connection record group and group corresponding to the Group-ID The connection topology is shown below.

Next, step S204 is to determine that the behavior conforms to the characteristics of domain flux. The website analysis module presents each connected group with its network connection topology diagram, and then determines whether the group meets the connection characteristics of domain flux. Among them, the connection characteristic of domain flux is that a small number of Client IPs connect to many external URLs, as shown in Figure 6, which is a schematic diagram of a network behavior topology diagram of domain flux. In a single set, only a few The Client IP uses the same User-agent to connect to multiple different domains. As can be seen in the figure, the Client IP uses the same User Agent to connect to the beginning of sp-install and c-sp at 10.107.56.20. -storage, sp-download, sp-alive, sp-setting, sp-storage, Orbtr-install, spms-download, c-api.sec, and so on, which conform to a typical use of the same The user-agent (User-agent) connection characteristics of the domain flux of multiple different domains.

Then, step S205 is to filter the domains of legitimate behaviors. After the website analysis module takes out the collection that meets the domain flux connection characteristics, it will filter out the representative and normal network applications (for example, antivirus software) or CDN and other connected domains; as shown in Figure 7, it is an example schematic of anti-virus software's connection behavior through the domain flux method, but this known normal network application behavior will not be listed The botnet behavior to be detected by the present invention will be eliminated at this step, and the remaining domains will be judged as botnets with domain flux. For example, in the example in FIG. 7, "iavs9x" .u.avast.com "regular expression is added to the filtering list to filter out this normal web application.

Next, as shown in FIG. 8, it is a schematic diagram of an example of log data of a proxy server, in which each line of data represents a created time stamp (Timestamp), client IP (Client IP) of a log record, respectively. , Terminal URL (Dest Url), terminal port (Dest Port), user-agent (User-agent) and other information, can also include the amount of transmission (Sent Byte), the amount of reception (Receive Byte), method (Method), path ( Path) and other information, and the present invention mainly uses only the client IP, terminal URL, and user agent information to detect the domain flux botnet, and judges whether it is a famous website by the size of the node dimension value. The dimension value A low representative of an external website that is only connected by a small number of user agents has a higher chance of being a malicious relay station.

In succession, in step S202, a website clustering algorithm is used to merge the sets that have intersections in the collection of external websites, and then generate disjoint sets. The steps of the method for searching for a connected set are as follows: (1) using the elements in the set as the unit, find Set the intersections of the sets with each other, and merge the sets that have intersections into one set (the size of the set exceeds the preset threshold is filtered in this step); (2) determine which sets are disjoint sets (there is no intersection with other sets) And separate them, and the remaining sets return to step (a) for execution; (c) repeat steps (a) and (b) until all sets are independent. Taking Figure 4 as an example, using the MapReduce architecture, in Item 1 ~ Item 8, with Client IP and user agent as keys, and terminal URL as the value, four sets can be clustered first, which are Item 1 and Item 2 The values of CnC1 and CnC2 are clustered. The values of CnC2 and CnC3 are clustered at Item 3 and Item 4. The values of CnC4 and CnC5 are clustered at Item 5 and Item 6. The values of CnC4 and CnC6 are clustered at Item 7 and Item 8. These four sets can be used as the input set of the rejoining set, and the final result of the rejoining set is two disjoint sets of URLs (CnC1, CnC2, CnC3) and "CnC4, CnC5, CnC6"; then, According to the example shown in FIG. 5, after step S203 is performed in the present invention, a connection topology diagram of the two sets is obtained.

The present invention is based on the same Client IP connecting to many different terminal URLs, and the one-to-many relationship presented conforms to the connection topology of the domain flux connection to determine the botnet. For example, it can be based on the The value of the terminal URL in a set divided by the value of the client IP exceeds a certain preset threshold, as a judgment basis to determine the domain change behavior of the set that meets the domain flux connection characteristics.

Finally, as shown in FIG. 9 and FIG. 10, it is a schematic diagram of an embodiment of screening a botnet that conforms to domain flux through the method of the present invention. After the process shown in FIG. 2 of the present invention, a group 739663648 is obtained. In line with the characteristics of the domain flux botnet collection, further, the domain on this group is queried on the Virus Total website. As shown in FIG. 10, the entire group of networks obtained through the present invention can be observed. Domains are registered as malicious web addresses, so it is known that the method of detecting domain flux botnets through proxy server logs of the present invention is indeed effective.

The above detailed description is specifically for the preferred embodiment of the present invention, but this embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or change without departing from the technical spirit of the present invention should be included in Within the scope of the patent in this case.

In summary, the present invention is technically innovative and has multiple effects that are inferior to the previous technology. It has fully met the novel and progressive statutory invention patent requirements. It has filed a patent application in accordance with the law and urges your office to approve this invention. The patent application encourages invention, and it is a matter of virtue.

Claims (4)

    A method for detecting domain flux botnets through proxy server logs, which uses a website analysis module to perform the following steps, including: accessing at least one log data stored by an external proxy server; The connection status represented by the user-agent information in the log data is filtered, and the log content belonging to the connection with the conventional website is filtered; according to a website group algorithm, the connection log of the remaining log content is filtered. In order to match the external client with the same Client IP and user-agent connection record, perform grouping to obtain the associated set of external websites in the log content; obtain the client IP connection record and Client IP and connection in the log content The connection topology of external websites; the relationship between external websites and associated collections is presented through the connection topology, and a connection feature algorithm is used to determine whether the collection meets the connection characteristics of domain flux. Among them, domain flux Connection characteristics refers to the use of the same user agent with Client IP to connect to multiple dissimilar domains; Collection feature, which was filtered off and the domain lines connected to the normal web application or the CDN, it is determined that the remaining domain zombie network domain flux.         The method for detecting a domain flux botnet through a proxy server log as described in item 1 of the scope of the patent application, wherein the node filtering algorithm is performed by the website analysis module based on long-term log data. The terminal URL is used as the key, and the user agent information is used as the value. The MapReduce framework calculates and obtains a list of the number of times that each external website in the log data has been connected by a different user agent. The number of times is the same as the dimension value on the node. The size of the dimension value can be used to determine whether the external website represented by the node is a famous website. Only external websites whose dimension value is less than the preset threshold will be retained. .         The method for detecting domain flux botnets through proxy server logs as described in item 1 of the scope of the patent application, wherein the grouping algorithm of the website is grouped by the method of joint set search, that is, the website analysis module After establishing several collections of connection records in the log content, the following steps are performed: using individual connection record data in each collection as elements, if the external website connected to the connection records has the same Client IP and user agent, it is determined as Intersect and merge sets with elements that have intersections; delete sets that have more elements in the set than a preset threshold; judge sets that do not intersect with other sets as independent sets; and repeat the above three steps until all sets are judged as independent set.         The method for detecting a domain flux botnet through a proxy server log as described in item 1 of the scope of the patent application, wherein the connection feature algorithm is used to calculate the ratio of the number of terminal URLs to the number of client IPs in a set If the ratio exceeds the preset threshold, it means that the aggregated website conforms to the domain change behavior of the domain flux connection characteristic.