CN114257565B - Method, system and server for mining potential threat domain names - Google Patents

Abstract

The application discloses a method, a system and a server for mining potential threat domain names, relates to the field of communication, and aims to solve the problem that analysis efficiency is low for mass data by adopting an existing threat domain name identification method. The method comprises the following steps: acquiring a second feature set based on a first feature set acquired in advance, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set and comprises dynamic scene features and static scene features; and obtaining a suspected threat domain name set by carrying out association calculation on the features in the second feature set. The present application threatens the mining of domain names.

Description

Method, system and server for mining potentially threatening domain names

technical field

The present application relates to the communication field, and in particular to a method, system and server for mining potentially threatening domain names.

Background technique

With the advancement of science and the development of communication technology, mobile phones have become an indispensable part of people's daily life. In recent years, more and more users use the mobile Internet to browse the web, watch video information, and use various mobile phone applications (APP) to socialize, entertain, study, live, etc., resulting in huge mobile Internet access data. Many criminals will attack users' mobile phones through network servers, which poses a great threat to network security. Therefore, it is an urgent task to effectively mine potential threat domain names.

In related technologies, for the mining of potential threat domain names, a domain name content identification engine is often established. This method is based on the similarity recognition technology of web page content, and performs feature mining on the elements of the web page, then classifies, predicts and recognizes them. However, because this method needs to successfully visit the website page to obtain content elements, the analysis efficiency is low in the face of massive data, and it is difficult to achieve effective analysis of the entire network log.

It can be seen that there is an urgent need for a method for mining potential threat domain names to improve the efficiency of mining threat domain names.

Contents of the invention

The embodiment of the present application provides a method for mining potentially threatening domain names, which is used to solve the problem of low analysis efficiency for massive data using existing threatening domain name identification methods.

The embodiment of the present application also provides a system for mining potentially threatening domain names, which is used to solve the problem of low analysis efficiency and difficulty in realizing the effective analysis of the entire network logs in the face of massive data using the existing threatening domain name identification methods.

The embodiment of the application adopts the following technical solutions:

In the first aspect, a method for mining potentially threatening domain names is provided, including:

Obtaining a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set, and The second feature set includes dynamic scene features and static scene features;

A set of suspected threat domain names is obtained by performing correlation calculation on the features in the second feature set.

In the second aspect, a mining system for potentially threatening domain names is provided, including:

An acquisition module, configured to acquire a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set set and said second feature set includes dynamic scene features and static scene features;

A processing module, configured to obtain a set of suspected threat domain names by performing correlation calculation on the features in the second feature set.

In a third aspect, a server is provided, including:

An acquisition module, configured to acquire a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set set and said second feature set includes dynamic scene features and static scene features;

A processing module, configured to obtain a set of suspected threat domain names by performing correlation calculation on the features in the second feature set.

In a fourth aspect, a computer-readable storage medium is provided, wherein a program is stored in the computer-readable storage medium, and when the program is executed, the following process is implemented:

Obtaining a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set, and The second feature set includes dynamic scene features and static scene features;

A set of suspected threat domain names is obtained by performing correlation calculation on the features in the second feature set.

The above at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects:

In the embodiment of the present invention, the second feature set is obtained based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set A subset of a feature set, and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing correlation calculation on the features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in this application. Those skilled in the art can also obtain other drawings based on these drawings without any creative effort.

Fig. 1 is a flow chart of a method for mining potentially threatening domain names provided by an embodiment of the present application;

Fig. 2 is a schematic diagram of an example feature association relationship link in the embodiment of the present application;

FIG. 3 is a flow chart of determining a set of suspected threat domain names based on association analysis in an embodiment of the present application;

FIG. 4 is a flow chart of a method for mining potentially threatening domain names provided by an embodiment of the present application;

Fig. 5 is a structural block diagram of the system provided by the embodiment of the present application;

FIG. 6 is a structural block diagram of a server provided by an embodiment of the present application.

Detailed ways

Embodiments of the present application provide a method and system for mining potentially threatening domain names.

In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

FIG. 1 is a flow chart of a method for mining potentially threatening domain names provided by an embodiment of the present application. As shown in Figure 1, a method for mining potentially threatening domain names provided in the embodiment of the present application may include the following steps:

Step 110: Obtain a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is a feature set of the first feature set and the second feature set includes dynamic scene features and static scene features.

In the embodiment of the present application, the first feature set may specifically include but not limited to: the number of domain name IPs, the number of visiting users, the number of visits, the number of Uniform Resource Locator (Uniform Resource Locator, URL) URLs, the maximum number of visiting users, The average number of URLs per user, the average number of URL users, the average number of URL visits, the standard deviation of the number of users, the standard deviation of the number of visits, the dispersion coefficient of the number of URL visits, the dispersion of the number of visiting users, the number of frequent users visiting the domain name by a single user on the day, and the domain name At least two of the proportion of the most visited URLs, the number of URLs accessed by a single user, the survival time of the domain name, the terminal operating system attribute, the access carrier attribute, the domain name return code attribute, the IP region attribute, and the URL return code attribute.

In this embodiment of the present application, the second feature set may specifically include dynamic scene features and static scene features.

The dynamic scene feature includes: at least one of a communication feature and an associated feature, wherein the communication feature includes a Domain Generation Algorithm (Domain Generation Algorithm, DGA) attribute, and the associated feature includes: a domain name return code attribute, a domain name IP At least one of regional attributes and URL return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: number of visiting users, dispersion of visiting users, domain name At least one of the proportion of the most visited URLs, the number of URLs visited by a single user, and the survival time of the domain name. The fingerprint feature includes: at least one of the attributes of the terminal operating system and the attributes of the access carrier.

The DGA attribute specifically refers to the calculation of the degree of random dispersion of domain name characters, that is, the DGA attribute is the ratio of the domain name entropy value to the domain name vowel value, wherein the entropy value refers to the randomness value of characters, and the domain name vowel value refers to the number of vowel letters in the domain name. The ratio of the number to the total length of the domain name.

The domain name return code attribute specifically refers to the jump status code attribute of the URL when accessing the domain name, such as: the domain name returns a 4xx error, but the URL returns 200 Request Success or 204 No Content. Domains with these return codes are highly suspicious.

The IP geographical attribute specifically refers to the attribution of the IP address resolved by the domain name. Generally, overseas IPs are more dangerous than domestic IPs.

The URL return code attribute specifically refers to the status code attribute of the URL when accessing the URL under the domain name. For example, if the URL returns a 302 status code, it means that the URL has been redirected and is suspicious.

The number of visiting users mentioned specifically refers to the more visiting users of a domain name on the same day, except for whitelisted domain names, it means that the domain name has a wide range of influence and a high degree of danger.

The dispersion of the number of visiting users specifically refers to the subjective randomness of the whitelisted domain names, and the number of visiting users will be relatively balanced; if the number of visiting users of a certain domain name and its URL is concentrated, it means that the domain name may be in the stage of explosive visits. High risk.

The proportion of the most URLs visited by the domain name refers to the URLs under the domain name. If there is a URL that is accessed intensively, the higher the proportion of visits, the higher the possibility of the URL being used and the higher the risk.

The number of URLs accessed by a single user specifically means that if the access behavior of a single user is controlled, it will not interact frequently with the domain name of the malicious server. This behavior is very different from uncontrolled normal user behavior. The smaller the number, the higher the risk.

The survival time of the domain name specifically refers to malicious domain names and gray domain names that will be regularly replaced with new domain names in order to avoid monitoring. The shorter the survival time, the higher the degree of danger.

In this embodiment of the present application, the second feature set is a subset of the first feature set. That is, any feature in the second feature set is from the first feature set. For example, the first feature set is, for example, the standard deviation of the number of users and the DGA attribute, and the second feature set is, for example, the DGA attribute.

In the embodiment of the present application, before step 110, the method for mining potentially threatening domain names provided in the embodiment of the present application may further include the step of obtaining the first feature set. Specifically, the process of acquiring the first feature set may include: acquiring operator-side data; and obtaining the first feature set related to the potentially threatening domain name based on the acquired operator-side data.

The operator side has a wide coverage communication network, which generates massive amounts of data all the time. The data on the operator's side includes not only the basic information of the user, but also multi-dimensional information such as the user's communication data, social activity data, consumption behavior data, and location information data. The data on the carrier side has unique data integrity, continuity, and richness, which is unmatched by any other industry data. Therefore, the embodiment of the present application takes advantage of the massive data on the operator side to obtain a large amount of user information, such as information that does not involve user privacy, and then summarizes the massive data to obtain the first feature set related to the potential threat domain name, which can ensure that The accuracy of the first feature set of .

In step 110, the obtaining the second feature set based on the pre-acquired first feature set includes: using the random forest algorithm to train the feature data in the pre-acquired first feature set to obtain information related to potential threat domain names. the second feature set.

The second feature set is obtained after the first feature set is trained by the random forest algorithm, and the second feature set is a subset of the first feature set, so the obtained second feature set is compared with the original The amount of data in some first feature sets is significantly reduced, which is more conducive to improving the efficiency of data processing.

Specifically, in the embodiment of the present application, the specific process of using the random forest algorithm for training can be performed according to the following steps:

Sub-step 1: Construct n new training sets by sampling with replacement for the pre-acquired feature data in the first feature set;

Wherein, the characteristic data in different new training sets may be repeated, and the characteristic data in the same new data set may also be repeated.

Wherein, the feature data may refer to the specific value of each feature in the first feature set for a certain domain name.

Sub-step 2: construct sub-decision trees according to the new training set; wherein, each sub-decision tree corresponds to a new training set;

Among them, the random forest is composed of n (n is a positive integer, and the specific value of n is random) sub-decision trees, and each node in the decision tree is a judgment condition about specific characteristic data. For example, if one of the nodes is used as the parent node, then the node has two child nodes located on the left and right sides of the node, which are respectively assumed to be the left child node and the right child node. If the parent node is a DGA value, its left child node can be generated if it is greater than a certain value, and its right child node can be generated if it is less than a certain value; if the parent node is a terminal operating system attribute, it can be The judgment logic of /No generates its left and right child nodes.

Sub-step 3: Obtain the second feature set related to the potential threat domain name by voting on the terminal nodes of the n sub-decision trees.

Wherein, the voting refers to performing statistical analysis on the end nodes of n decision trees, and selecting the node with the most occurrence times as the voting result.

In order to intuitively display the second feature set related to the potential threat domain name, the exemplary data of the second feature set related to the potential threat domain name obtained through the above sub-steps is shown in Table 1 below.

Table 1

Step 120, obtain a set of suspected threat domain names by performing correlation calculation on the features in the second feature set.

The method for mining potentially threatening domain names provided in the embodiment of the present application obtains a second feature set based on a pre-acquired first feature set, wherein the first feature set is a feature set related to a potentially threatening domain name, and the second The feature set is a subset of the first feature set and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing correlation calculation on features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

In the embodiment of the present application, various methods may be used to perform association calculation on the features in the second feature set, for example, use Apriori algorithm, FP-Tree algorithm, etc. to perform association calculation. The following only uses the Apriori algorithm as an example for illustration.

Wherein, the specific calculation manner of the association calculation may use the Apriori algorithm to perform the specific calculation. The Apriori algorithm is an algorithm for mining data association rules. The Apriori algorithm can be used to mine the degree of association between the second feature sets, and then form a collection of suspected threat domain names.

Specifically, referring to FIG. 3 , in the embodiment of the present application, step 120 may include the following process:

Step 1201: Determine the evaluation criteria of the potential threat domain name feature set, the evaluation criteria include at least one of support and confidence;

The evaluation criteria of the potential threat domain name feature set include support degree, confidence degree and so on. Generally speaking, to select frequent K-itemsets in a data set, custom evaluation criteria are usually required. In the Apriori algorithm described in this application, at least one of support and confidence is mainly involved as an evaluation index.

Wherein, the frequent K-itemset is composed of closely related K-item data. Wherein, K is a positive integer, and K≥2.

Wherein, the support degree is the probability that several feature specific values in the second feature set simultaneously appear in a domain name.

If in a domain name, there are two characteristic values X and Y in the second feature set whose correlation needs to be analyzed, then the corresponding support value is

Support(XY)＝P(XY)＝P(X)P(Y|X)

Among them, P(XY) represents the probability of X and Y occurring at the same time; P(X) represents the probability of X occurring; P(Y|X) represents the probability of Y occurring under the condition that X occurs; Support(XY) represents the corresponding XY Support.

By analogy, if in a domain name, there are three feature specific values X, Y, and Z in the second feature set whose relevance needs to be analyzed, then their corresponding support values are

Support(XYZ)＝P(XYZ)＝P(X)P(Y|X)P(Z|XY)

Among them, P(XYZ) represents the probability of X, Y and Z occurring at the same time; P(X) represents the probability of X occurring; P(Y|X) represents the probability of Y occurring under the condition that X occurs; P(Z|XY) Indicates the probability of Z occurring under the condition that XY occurs; Support(XYZ) indicates the support corresponding to XYZ.

Wherein, the confidence degree is the meaning of conditional probability in statistics, combined with the above example, the specific Confidence(X<=Y)=P(X|Y)

Confidence(X<=YZ)＝P(X|YZ)

Among them, Confidence(X<=Y) represents the confidence of X to Y, Confidence(X<=YZ) represents the confidence of X to YZ, P(X|Y) represents the probability of X under the condition that Y occurs, P (X|YZ) represents the probability that X occurs under the condition that YZ occurs.

Step 1202: Based on the determined evaluation criteria and the Apriori algorithm, perform correlation analysis on the features in the second feature set to determine a set of suspected threat domain names.

Specifically, step 1202 may include the following sub-steps 1 to 3.

Sub-step 1: performing data connection on the data in the second feature set; pruning the candidate 1-itemsets obtained after the connection according to the preset evaluation criteria to obtain frequent 1-itemsets;

Wherein, the candidate 1-itemset is obtained by performing pairwise data connection of the initial valid feature data and removing identical items. For example, for data 0 and 1, after connecting two pairs of data, you will get itemsets 01 and 10, and itemsets 01 and 10 are essentially the same items, but the order is different, then after removing the same items, you can get the candidate Any one of itemsets 01 or 10 in an itemset of 1. As for the connection methods of other data in the data, they are connected according to this pairwise connection method.

Wherein, the pruning process refers to deleting candidate one-itemsets that are lower than a preset evaluation standard.

Sub-step 2: Connect the elements in the frequent 1-itemset to obtain the candidate 2-itemset, and prune the elements in the candidate 2-itemset that are lower than the preset evaluation standard to obtain the frequent 2-itemset;

Among them, for the elements in the frequent 1-itemset, refer to the above sub-step 1 for data connection, so as to obtain the candidate 2-itemset. For candidate 2-itemsets, pruning is also performed in accordance with the above sub-step 1 to obtain frequent 2-itemsets. What needs to be pointed out here is that for a candidate 1-itemset that has been pruned, the candidate 2-itemset related to the candidate 1-itemset does not need to be judged according to the preset evaluation criteria before pruning , but directly execute the pruning process directly.

Sub-step 3: For frequent K-itemsets, iterate according to the above process until frequent K+1 itemsets cannot be found, and determine the obtained frequent K-itemsets as a set of suspected threat domain names; where K is a positive integer, and K ≥2.

The candidate K-itemsets mentioned in this application may be a set composed of K-1 initial effective features; frequent K-itemsets refer to candidate K-itemsets that meet preset evaluation criteria.

In the operation process of the Apriori algorithm, if a K-itemset is frequent, then all subsets of the K-itemset are frequent. It is understandable that during the pruning process, if a candidate item set is lower than the preset evaluation standard, it will be deleted, and all its subsets will be pruned. By using the above process, the algorithm traversal time can be greatly shortened, and the processing efficiency of the algorithm can be further improved.

Through the Apriori algorithm, the network operation behavior in all domain name scenarios can be collected. The following is an example of a simple single-chain model. In the actual model, it is a hybrid model composed of multiple single-chains.

The following is an example analysis of relationship links in a simple dimension, and FIG. 3 is a schematic diagram of analysis of feature association relationship links in this application.

The following is a specific example for each sub-step in step 1202. As shown in Figure 3, in the domain name A, there are initial effective feature data DGA, IP attributes, the number of users visiting on the day and the survival time of the domain name in the second feature set . The DGA is marked as 0, the IP attribute is marked as 1, the number of users visiting on that day is marked as 2, and the domain name survival time is marked as 3.

Perform the above sub-step 1 on the above-mentioned label features, and perform data connection on the initial effective feature data of the features in the second feature set to obtain a connected data set.

Specifically, data set 01 is obtained after data connection between DGA (0) and IP attribute (1); data set 02 is obtained after connection between DGA (0) and the number of users visiting on the day (2); IP attribute (1) and the current day Dataset 12 is obtained after the number of users visited (2) is connected; IP attribute (1) and domain name survival time (3) are connected to obtain data set 13; number of users visited on the day (2) and domain name survival time (3) Connect to get data set 23.

Then the data sets obtained after connection are pruned according to the preset evaluation standard. Assuming that the data set 23 is lower than the preset evaluation standard, it is pruned to obtain frequent 1-itemsets: 01, 02, 03 , 12, 13.

Execute the above sub-step 2 on the above frequent 1-itemset, connect the elements in the frequent 1-itemset, and obtain the candidate 2-itemset.

Specifically, the frequent 1-itemset 01, the frequent 1-itemset 02 and the frequent 1-itemset 12 are connected to obtain the candidate 2-itemset 012; the frequent 1-itemset 01, the frequent 1-itemset 03 and the frequent 1-itemset 13 are connected Candidate 2-itemset 013 is obtained; frequent 1-itemset 02, frequent 1-itemset 03 and data set 23 are connected to obtain candidate 2-itemset 023; frequent 1-itemset 12, frequent 1-itemset 13 and candidate 1-itemset 23 are performed Candidate 2-itemset 123 is obtained after connection.

The elements in the candidate 2-item set that are lower than the preset evaluation standard are pruned to obtain frequent 2-item sets. It should be noted here that, in the above step 1, the candidate 1-itemset 23 has been pruned, then the candidate 2-itemset 023 and 123 with the candidate 1-itemset 23 as the parent node are simultaneously pruned. Then, the frequent 2-itemsets after pruning are 012 and 013.

Execute the above sub-step 3 for the above frequent 2-itemsets, for the frequent K-itemsets (K=2 in this example analysis), iterate according to the above process until no frequent K+1 can be found (K=2 in this example analysis ) itemsets, the obtained frequent K-itemsets are determined as the collection of suspected threat domain names.

Specifically, the frequent 2-itemset 012, the frequent 2-itemset 013, the candidate 2-itemset 023 and the candidate 2-itemset 123 are connected to obtain a candidate 3-itemset 0123. What needs to be explained here is that in the above step 2, the candidate 2-itemset 023 and 123 have been pruned, then the candidate 3-itemset 0123 with the candidate 2-itemset 023 and 123 as the parent node is pruned synchronously Processing, that is, the frequent 3-itemset cannot be found at this time, and the obtained frequent 2-itemset is determined as a set of suspected threat domain names.

Optionally, in one embodiment, the method for mining potentially threatening domain names provided in the embodiments of the present application may further include the following steps:

Obtain the dynamic and static characteristics of the domain name data in the monitoring data source;

Wherein, the domain name data in the monitoring data source refers to the specific domain name data under the domain name in the monitoring real-time network.

The acquired dynamic and static features of the domain name data are compared with the acquired set of suspected threat domain names to obtain a domain name detection result.

Optionally, in one embodiment, the method for mining potentially threatening domain names provided in the embodiments of the present application may further include the following steps: when the domain name detection result is a detected threatening domain name, feeding back the domain name detection result to the malicious link library.

FIG. 4 is a flow chart of a method for mining potentially threatening domain names provided by an embodiment of the present application. As shown in Figure 4, a method for mining potentially threatening domain names provided in the embodiment of the present application may include the following steps:

Step 410: Acquire operator-side data; obtain a first feature set related to potential threat domain names based on the acquired operator-side data.

Step 420: Train the pre-acquired feature data in the first feature set with a random forest algorithm to obtain a second feature set related to potential threat domain names.

For the specific data processing process of the random forest algorithm, refer to the above-mentioned step 110, which will not be repeated here.

The second feature set is obtained after the first feature set is trained by the random forest algorithm, and the second feature set is a subset of the first feature set, so the obtained second feature set is compared with the original The amount of data in some first feature sets is significantly reduced, which is more conducive to improving the efficiency of data processing.

Step 430: Determine the evaluation standard of the potential threat domain name feature set, the evaluation standard includes at least one of support degree and confidence degree.

Step 440: performing data connection on the initial effective feature data of the features in the second feature set; pruning the candidate 1-itemsets obtained after connection according to the preset evaluation criteria to obtain frequent 1-itemsets.

Step 450: Connect the elements in the frequent 1-itemset to obtain the candidate 2-itemset, and prune the elements in the candidate 2-itemset that are lower than the preset evaluation standard to obtain the frequent 2-itemset.

Step 460: for frequent K-itemsets, iterate according to the above process until no frequent K+1-itemsets can be found, and determine the obtained frequent K-itemsets as the set of suspected threat domain names.

Step 470: Obtain the dynamic and static characteristics of the domain name data in the monitoring data source.

Step 480: Compare the obtained dynamic and static features of the domain name data with the obtained set of suspected threat domain names to obtain a domain name detection result.

The method for mining potentially threatening domain names provided in the embodiment of the present application obtains a second feature set based on a pre-acquired first feature set, wherein the first feature set is a feature set related to a potentially threatening domain name, and the second The feature set is a subset of the first feature set and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing correlation calculation on features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

Fig. 5 is a structural block diagram of a system for mining potentially threatening domain names provided by an embodiment of the present application. Referring to Fig. 5, the mining system for potentially threatening domain names provided by the embodiment of the present application may include:

The obtaining module 502 is configured to obtain a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set a subset of a feature set and said second feature set includes dynamic scene features and static scene features;

The processing module 504 is configured to obtain a set of suspected threat domain names by performing correlation calculation on the features in the second feature set.

The system for mining potentially threatening domain names provided by an embodiment of the present invention obtains a second feature set based on a pre-acquired first feature set, wherein the first feature set is a feature set related to a potentially threatening domain name, and the second The feature set is a subset of the first feature set and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing correlation calculation on features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

Optionally, in one embodiment of the present application, during the process of acquiring the second feature set, the acquisition module 502 may be configured to use the random forest algorithm to train the pre-acquired feature data in the first feature set , to obtain the second feature set related to the potential threat domain name.

Optionally, in one embodiment of the present application, before acquiring the second feature set based on the pre-acquired first feature set, the acquiring module 502 is further configured to:

Obtaining data on the operator side; obtaining a first feature set related to a potentially threatening domain name based on the acquired data on the operator side;

In the embodiment of the present application, optionally, the first feature set includes: the number of domain name IPs, the number of visiting users, the number of visits, the number of URLs, the maximum number of visiting users, the average number of URLs per user, and the average number of URL users Number, average number of URL visits, standard deviation of the number of users, standard deviation of the number of visits, dispersion coefficient of the number of URL visits, dispersion of the number of visiting users, frequency of single user visiting the domain name on the day, number of visiting users, proportion of the most URLs visited by the domain name, single user At least two of the number of URLs accessed, the survival time of the domain name, the terminal operating system attribute, the access carrier attribute, the domain name return code attribute, the IP region attribute, and the URL return code attribute.

In this embodiment of the present application, optionally, the dynamic scene feature includes: at least one of a communication feature and an associated feature, wherein the communication feature includes a DGA attribute, and the associated feature includes: a domain name return code attribute, At least one of domain name IP region attributes and URL return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users , at least one of the proportion of the maximum number of URLs accessed by the domain name, the number of URLs accessed by a single user, and the survival time of the domain name. The fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute.

In this embodiment of the application, optionally, the processing module 502 is used to:

Determining evaluation criteria for a feature set of potentially threatening domain names, where the evaluation criteria include at least one of support and confidence;

Based on the determined evaluation criteria and the Apriori algorithm, an association analysis is performed on the features in the second feature set to determine a set of suspected threat domain names.

Optionally, in the process of performing association analysis on the features in the second feature set based on the determined evaluation criteria and the Apriori algorithm to determine a set of suspected threat domain names, the processing module 504 is configured to:

performing data connection on the initial effective feature data of the features in the second feature set; performing pruning on the candidate item sets obtained after the connection according to preset evaluation criteria to obtain frequent item sets;

Connect the elements in the frequent 1-itemset to obtain the candidate 2-itemset, and prune the elements in the candidate 2-itemset that are lower than the preset evaluation standard to obtain the frequent 2-itemset;

For frequent K-itemsets, iterate according to the above process until frequent K+1 itemsets cannot be found, and determine the obtained frequent K-itemsets as a set of suspected threat domain names; where K is a positive integer, and K≥2.

Optionally, the acquisition module 502 is further configured to: acquire dynamic and static characteristics of the domain name data in the monitoring data source. Correspondingly, the processing module 504 is further configured to: compare the obtained dynamic and static characteristics of the domain name data with the obtained set of suspected threat domain names, so as to obtain a domain name detection result.

The specific process of the steps performed by each module in the system for mining potentially threatening domain names provided in the embodiment of the present application may refer to the method embodiment, and details are not repeated here.

FIG. 6 is a structural block diagram of a server provided by an embodiment of the present application. Referring to Figure 6, the server of the potentially threatening domain name provided by the embodiment of the present application may include:

The acquiring module 602 is configured to acquire a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set a subset of a feature set and said second feature set includes dynamic scene features and static scene features;

The processing module 604 is configured to obtain a set of suspected threat domain names by performing correlation calculation on the features in the second feature set.

In this embodiment of the present application, the second feature set is obtained based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set A subset of a feature set, and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing correlation calculation on the features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

Optionally, in one embodiment of the present application, during the process of acquiring the second feature set, the acquisition module 602 may be configured to use the random forest algorithm to train the pre-acquired feature data in the first feature set , to obtain the second feature set related to the potential threat domain name.

Optionally, in one embodiment of the present application, before acquiring the second feature set based on the pre-acquired first feature set, the acquiring module 602 is further configured to:

Obtaining data on the operator side; obtaining a first feature set related to a potentially threatening domain name based on the acquired data on the operator side;

In the embodiment of the present application, optionally, the first feature set includes: the number of domain name IPs, the number of visiting users, the number of visits, the number of URLs, the maximum number of visiting users, the average number of URLs per user, and the average number of URL users Number, average number of URL visits, standard deviation of the number of users, standard deviation of the number of visits, dispersion coefficient of the number of URL visits, dispersion of the number of visiting users, frequency of single user visiting the domain name on the day, number of visiting users, proportion of the most URLs visited by the domain name, single user At least two of the number of URLs accessed, the survival time of the domain name, the terminal operating system attribute, the access carrier attribute, the domain name return code attribute, the IP region attribute, and the URL return code attribute.

In this embodiment of the present application, optionally, the dynamic scene feature includes: at least one of a communication feature and an associated feature, wherein the communication feature includes a DGA attribute, and the associated feature includes: a domain name return code attribute, At least one of domain name IP region attributes and URL return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users , at least one of the proportion of the maximum number of URLs accessed by the domain name, the number of URLs accessed by a single user, and the survival time of the domain name. The fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute.

In this embodiment of the application, optionally, the processing module 602 is used to:

Determining evaluation criteria for a feature set of potentially threatening domain names, where the evaluation criteria include at least one of support and confidence;

Based on the determined evaluation criteria and the Apriori algorithm, an association analysis is performed on the features in the second feature set to determine a set of suspected threat domain names.

Optionally, in the process of performing correlation analysis on the features in the second feature set based on the determined evaluation criteria and the Apriori algorithm to determine a set of suspected threat domain names, the processing module 604 is configured to:

performing data connection on the initial effective feature data of the features in the second feature set; performing pruning on the candidate item sets obtained after the connection according to preset evaluation criteria to obtain frequent item sets;

Connect the elements in the frequent 1-itemset to obtain the candidate 2-itemset, and prune the elements in the candidate 2-itemset that are lower than the preset evaluation standard to obtain the frequent 2-itemset;

For frequent K-itemsets, iterate according to the above process until frequent K+1 itemsets cannot be found, and determine the obtained frequent K-itemsets as a set of suspected threat domain names; where K is a positive integer, and K≥2.

Optionally, the acquiring module 602 is further configured to: acquire the dynamic and static characteristics of the domain name data in the monitoring data source. Correspondingly, the processing module 604 is further configured to: compare the obtained dynamic and static characteristics of the domain name data with the obtained set of suspected threat domain names, so as to obtain a domain name detection result.

For the specific process of the steps performed by each module in the server provided in the embodiment of the present application, reference may be made to the method embodiment, and details are not repeated here.

In addition, an embodiment of the present application also provides a computer-readable storage medium, where a program is stored in the computer-readable storage medium, and when the program is executed, the following process is implemented:

Obtaining a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set, and The second feature set includes dynamic scene features and static scene features;

A set of suspected threat domain names is obtained by performing correlation calculation on the features in the second feature set.

Wherein, the specific implementation process of the above steps can refer to the above description, and will not be repeated here.

The storage medium provided by the embodiment of the present application obtains the second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is all A subset of the first feature set and the second feature set includes dynamic scene features and static scene features; a set of suspected threat domain names is obtained by performing associated calculations on the features in the second feature set. In this way, by obtaining the first feature set in advance and obtaining the second feature set from it, the feature set related to the potential threat domain name can be quickly delineated in the big data, and the acquisition of the suspected threat domain name set based on the second feature set can greatly reduce data processing. In this way, the efficiency of front-end identification of threat domain names can be effectively improved, and the effective analysis of logs on the entire network can be realized.

It should be pointed out that in the embodiment of the present application, the random forest algorithm is used to train the effective features associated with the threat domain name, which effectively avoids interference caused by features unrelated to the threat domain name. Due to the introduction of two randomnesses in the random forest algorithm (random selection of new training set samples and random selection of frequent features), the random forest algorithm is not easy to fall into overfitting, and the effective features selected by voting are relatively accurate. Extract the dynamic and static features of threat domain names, and use the association rule Apriori algorithm to mine potential threat domain name feature sets in mixed scenarios, which can fully reflect the static and dynamic laws of domain names, and feature sets composed of multiple effective features can more accurately identify unknown domain names. The domain name is fitted to improve the ability to identify unknown domain names. By mining the massive data in the mobile communication network according to the processing flow of this application, the scope of potential threat domain names can be quickly and accurately delineated in the big data, thereby improving the efficiency of domain name identification.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

Embodiments of the present invention have been described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementations, and the above-mentioned specific implementations are only illustrative, rather than restrictive, and those of ordinary skill in the art will Under the enlightenment of the present invention, without departing from the gist of the present invention and the protection scope of the claims, many forms can also be made, all of which belong to the protection of the present invention.

Claims (9)

1. A method for mining potential threat domain names, characterized in that, comprising: Obtaining a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set, and The second feature set includes dynamic scene features and static scene features; Obtaining a set of suspected threat domain names by performing correlation calculation on the features in the second feature set; Wherein, the dynamic scene features include: at least one of communication features and associated features, wherein the communication features include domain name generation algorithm DGA attributes, and the associated features include: domain name return code attributes, domain name IP region attributes and URL At least one of the return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users, and the most URLs visited by domain names At least one of the proportion of times, the number of URLs accessed by a single user, and the duration of the domain name, the fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute; wherein, the DGA attribute specifically refers to the calculation domain name The degree of random dispersion of characters, that is, the DGA attribute is the ratio of the entropy value of the domain name to the proportion of vowels in the domain name, where the entropy value refers to the degree of character randomness, and the vowel value of the domain name refers to the ratio of the number of vowels in the domain name to the total length of the domain name. 2. The mining method according to claim 1, wherein said obtaining a second feature set based on the pre-acquired first feature set comprises: using random forest for feature data in the pre-acquired first feature set An algorithm is trained to obtain a second set of features associated with potentially threatening domain names. 3. mining method according to claim 1, is characterized in that, before described first feature set based on pre-acquiring, obtains the second feature set, described method also comprises: Obtain data from the operator side; Obtaining a first feature set related to the potentially threatening domain name based on the acquired data on the operator side; Wherein, the first feature set includes: the number of domain name IPs, the number of visiting users, the number of visits, the number of URLs, the maximum number of visiting users, the average number of URLs for a single user, the average number of URL users, the average number of URL visits, and the number of users Standard deviation, standard deviation of number of visits, dispersion coefficient of URL visits, dispersion of number of visiting users, frequency of single user visiting domain names on the day, number of visiting users, proportion of most URLs visited by domain names, number of URLs visited by a single user, duration of domain name survival, At least two of terminal operating system attributes, access carrier attributes, domain name return code attributes, IP region attributes, URL return code attributes, and DGA attributes. 4. The mining method according to any one of claims 1-3, wherein the obtaining of the collection of suspected threat domain names by performing association analysis on features in the second feature set comprises: Determining evaluation criteria for a feature set of potentially threatening domain names, where the evaluation criteria include at least one of support and confidence; Based on the determined evaluation criteria and the Apriori algorithm, an association analysis is performed on the features in the second feature set to determine a set of suspected threat domain names. 5. The mining method according to claim 4, characterized in that, based on the determined evaluation criteria and the Apriori algorithm, the features in the second feature set are associated and analyzed to determine that the suspected threat domain name set includes : performing data connection on the initial effective feature data of the features in the second feature set; performing pruning on the candidate item sets obtained after the connection according to preset evaluation criteria to obtain frequent item sets; Connect the elements in the frequent 1-itemset to obtain the candidate 2-itemset, and prune the elements in the candidate 2-itemset that are lower than the preset evaluation standard to obtain the frequent 2-itemset; For frequent K-itemsets, iterate according to the above process until frequent K+1 itemsets cannot be found, and determine the obtained frequent K-itemsets as a set of suspected threat domain names; where K is a positive integer, and K≥2. 6. mining method according to claim 1, is characterized in that, described method also comprises: Obtain the dynamic and static characteristics of the domain name data in the monitoring data source; The acquired dynamic and static features of the domain name data are compared with the acquired set of suspected threat domain names to obtain a domain name detection result. 7. A system for mining potentially threatening domain names, comprising: An acquisition module, configured to acquire a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set set and said second feature set includes dynamic scene features and static scene features; A processing module, configured to obtain a set of suspected threat domain names by performing correlation calculations on features in the second feature set; Wherein, the dynamic scene features include: at least one of communication features and associated features, wherein the communication features include domain name generation algorithm DGA attributes, and the associated features include: domain name return code attributes, domain name IP region attributes and URL At least one of the return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users, and the most URLs visited by domain names At least one of the proportion of times, the number of URLs accessed by a single user, and the duration of the domain name, the fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute; wherein, the DGA attribute specifically refers to the calculation domain name The degree of random dispersion of characters, that is, the DGA attribute is the ratio of the entropy value of the domain name to the proportion of vowels in the domain name, where the entropy value refers to the degree of character randomness, and the vowel value of the domain name refers to the ratio of the number of vowels in the domain name to the total length of the domain name. 8. A server, characterized in that, comprising: An acquisition module, configured to acquire a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, and the second feature set is the first feature set set and said second feature set includes dynamic scene features and static scene features; A processing module, configured to obtain a set of suspected threat domain names by performing correlation calculations on features in the second feature set; Wherein, the dynamic scene features include: at least one of communication features and associated features, wherein the communication features include domain name generation algorithm DGA attributes, and the associated features include: domain name return code attributes, domain name IP region attributes and URL At least one of the return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users, and the most URLs visited by domain names At least one of the proportion of times, the number of URLs accessed by a single user, and the duration of the domain name, the fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute; wherein, the DGA attribute specifically refers to the calculation domain name The degree of random dispersion of characters, that is, the DGA attribute is the ratio of the entropy value of the domain name to the proportion of vowels in the domain name, where the entropy value refers to the degree of character randomness, and the vowel value of the domain name refers to the ratio of the number of vowels in the domain name to the total length of the domain name. 9. A computer-readable storage medium, wherein a program is stored in the computer-readable storage medium, and when the program is executed, the following process is implemented: Obtaining a second feature set based on the pre-acquired first feature set, wherein the first feature set is a feature set related to a potential threat domain name, the second feature set is a subset of the first feature set, and The second feature set includes dynamic scene features and static scene features; Obtaining a set of suspected threat domain names by performing correlation calculation on the features in the second feature set; Wherein, the dynamic scene features include: at least one of communication features and associated features, wherein the communication features include domain name generation algorithm DGA attributes, and the associated features include: domain name return code attributes, domain name IP region attributes and URL At least one of the return code attributes; the static scene features include: at least one of behavioral features and fingerprint features, wherein the behavioral features include: the number of visiting users, the dispersion of the number of visiting users, and the most URLs visited by domain names At least one of the proportion of times, the number of URLs accessed by a single user, and the duration of the domain name, the fingerprint feature includes: at least one of the terminal operating system attribute and the access carrier attribute; wherein, the DGA attribute specifically refers to the calculation domain name The degree of random dispersion of characters, that is, the DGA attribute is the ratio of the entropy value of the domain name to the proportion of vowels in the domain name, where the entropy value refers to the degree of character randomness, and the vowel value of the domain name refers to the ratio of the number of vowels in the domain name to the total length of the domain name.