CN111723846B - Encrypted and compressed traffic identification method and device based on random characteristics - Google Patents

Abstract

The invention belongs to the technical field of network traffic data classification and discloses an encryption and compression traffic identification method based on random characteristics, which includes: collecting network data and parsing to obtain the traffic data; calculating and obtaining the random characteristics ECF characteristics of the traffic data. Vector; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fourier transform, non-overlapping matching, serialization and cumulative sum; take the ECF feature vector as input , identification is performed through a machine learning model, and the identification results include encrypted traffic and compressed traffic; the invention also discloses an encryption and compressed traffic identification device based on random characteristics. The present invention combines machine learning to construct an effective random feature ECF feature vector, which can still identify encrypted and compressed traffic with high accuracy even when partial data is obtained or the amount of data is small.

Description

Encrypted and compressed traffic identification method and device based on random characteristics

Technical field

The invention belongs to the technical field of network traffic data classification, and in particular relates to a method and device for encrypted and compressed traffic identification based on random characteristics.

Background technique

As the basic work of network management, traffic identification has always been the research focus of network managers. With the popularization and widespread use of encryption technologies such as HTTPS, SSL, and VPN, encrypted traffic identification has become the main task of traffic identification. In addition, because the compression algorithm has the advantages of being easy to implement, requiring no key interaction, strong randomness of compressed data, and smaller amount of data compared to plaintext, a large number of malicious behaviors avoid traffic supervision and behavior detection and cover up their actual communication content during the communication process. Use compressed transmission to avoid security detection. At the same time, wireless communication technology has developed rapidly in recent years, and wireless network traffic has grown extremely rapidly. With the popularization of 5G networks and the large-scale application of Internet of Things (IoT) devices, there is also a large amount of compressed and encrypted data in wireless networks. How to detect wireless communications using secure confidential communications. To sum up, no matter in network management, equipment security detection and network security detection work, effective methods are needed to identify encrypted and compressed data.

Currently, in terms of compressed data and encrypted data identification, there are mainly methods such as identification field detection, decompression exhaustion, and statistical characteristic analysis. Identity field detection distinguishes encrypted or compressed traffic by identifying special file identifiers (Malhotra P. Detection of encrypted streams for egress monitoring [J]. Dissertations & Theses-Gradworks, 2007.). Generally, the identification of compressed files is mainly concentrated in the header or tail of the file. However, when compressed data is transmitted over the network, it is usually broken down into many data packets, and there are often only one or two data packets containing file identification. If only a small number of packets of traffic can be intercepted, then this method cannot be used to determine the nature of the traffic (encryption or compression). The second type of method is to identify compressed data through exhaustive decompression methods (Conte T M, Wolfe A. Techniques for detecting encrypted data: U.S. Patent 8,799,671[P].2014-8-5.). Due to the data correlation of the compression algorithm, to decompress the compressed data, a complete compressed file must be obtained. For compressed files transmitted through the network, the detector is required to obtain all transmission data packets and splice them in the correct order. , to get the complete compressed file for detection. In the information age where network traffic is increasing day by day, due to limitations of processing speed, storage space and other factors, it is difficult to completely capture and monitor all data packets on the network. Therefore, the method of obtaining complete compressed files and detecting them is very difficult to implement in actual work. big. The third type of method is to identify the encrypted or compressed data transmitted over the network by analyzing the overall performance characteristics of the compressed or encrypted data. This detection method only needs to obtain part of the packet load in network communication, and has good practicability in actual network traffic detection. However, the recognition accuracy of this method is greatly affected by the length of the acquired payload data. When the data length is short, the recognition accuracy is generally not high.

Contents of the invention

In order to solve the problem of low accuracy of existing encryption and compressed traffic identification methods, the present invention proposes a method and device for encrypted and compressed traffic identification based on random characteristics, which improves the identification accuracy of encrypted and compressed traffic.

In order to achieve the above objects, the present invention adopts the following technical solutions:

An encrypted and compressed traffic identification method based on random characteristics, including:

Step 1: Collect network data and parse to obtain traffic data;

Step 2: Calculate and obtain the random characteristic ECF feature vector of the traffic data; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fourier transform, non-overlapping Matching, serialization and cumulative sums;

Step 3: Take the ECF feature vector as input and identify it through the machine learning model. The identification results include encrypted traffic and compressed traffic.

Further, the step 1 includes:

Step 1.1: Obtain data packets from the external network and save them as pcap files;

Step 1.2: Divide the acquired data packets into network flows according to five-tuple groups and save them as flow files;

Step 1.3: For each flow file, analyze it according to the TCP/IP protocol format, obtain the data payload part, and splice it into a flow data of variable length according to the order in which the data packets are obtained.

Further, the step 2 includes:

Step 2.1: Use bytes as the statistical unit to obtain the length Len of the traffic data;

Step 2.2: Calculate the ECF feature vector to obtain the traffic data.

Further, before step 3, it also includes:

Build a machine learning model based on the ECF feature vector; the machine learning algorithms included in the machine learning model include random forest, Xgboost and MLP.

Further, step 3 includes:

Step 3.1: Based on the length Len of the traffic data and the expected test accuracy, select the trained machine learning model for identification;

Step 3.2: Input the ECF feature vector of the traffic data into the machine learning model to obtain the identification results. The identification results include encrypted traffic and compressed traffic.

An encryption and compression traffic identification device based on random characteristics, including:

The collection and analysis module is used to collect network data and analyze the traffic data;

Feature extraction module, used to calculate and obtain the random characteristic ECF feature vector of traffic data; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fourier transform , non-overlapping matching, serialization and cumulative sum;

The traffic identification module is used to use the ECF feature vector as input to identify through the machine learning model. The identification results include encrypted traffic and compressed traffic.

Further, the collection and analysis module includes:

The data packet acquisition submodule is used to obtain data packets from the external network and save them as pcap files;

The network flow division sub-module is used to divide the acquired data packets into network flows according to five-tuple groups and save them as flow files;

The parsing sub-module is used to parse each flow file according to the TCP/IP protocol format, obtain the data payload part, and splice it into a flow data of variable length according to the order in which the data packets are obtained.

Further, the feature extraction module includes:

The traffic data length acquisition submodule is used to obtain the length Len of the traffic data in bytes as the statistical unit;

The feature extraction submodule is used to calculate the ECF feature vector of traffic data.

Furthermore, it also includes:

The traffic identification model building module is used to build a machine learning model based on the ECF feature vector; the machine learning algorithms included in the machine learning model include random forest, Xgboost and MLP.

Further, the traffic identification module includes:

The model selection submodule is used to select a trained machine learning model for identification based on the length Len of the traffic data and the expected test accuracy;

The traffic identification submodule is used to input the ECF feature vector of traffic data into the machine learning model to obtain identification results. The identification results include encrypted traffic and compressed traffic.

Compared with the prior art, the present invention has the following beneficial effects:

According to the random feature ECF proposed by the present invention, the ECF feature vector is extracted from the traffic data, and the ECF feature vector is input into the pre-trained machine learning model to identify the type of traffic, including encrypted traffic and compressed traffic. The traditional traffic identification method based on statistical features selects fewer statistical features, has a single discrimination method, and has low identification accuracy for encrypted and compressed traffic. The method provided by the present invention utilizes the learning advantages of machine learning algorithms on big data to design an effective machine learning identification model, overcomes the above shortcomings, improves the identification accuracy of encrypted and compressed traffic, and provides a basis for the refinement of network traffic in network management work. It provides technical support for chemical identification. Especially when partial data is obtained or the amount of data is small, it can still identify encrypted and compressed traffic with high accuracy.

Description of the drawings

Figure 1 is one of the flow charts of an encryption and compression traffic identification method based on randomness characteristics according to an embodiment of the present invention;

Figure 2 is a graph comparing the accuracy of the present invention and existing related algorithms;

Figure 3 shows the generalization effect of the machine learning model of the present invention between different data sets with a data length of 1KB;

Figure 4 shows the model generalization effect when the training data set of the present invention is D_1KB;

Figure 5 shows the model generalization effect when the training data set of the present invention is D_64KB;

Figure 6 shows the impact of the amount of training data on accuracy on the D_1KB data set;

Figure 7 is a second flow chart of an encryption and compression traffic identification method based on randomness characteristics according to an embodiment of the present invention;

Figure 8 is a schematic diagram of the architecture of an encryption and compression traffic identification device based on random characteristics according to an embodiment of the present invention.

Detailed ways

The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments:

As shown in Figure 1, an encrypted and compressed traffic identification method based on random characteristics includes:

Step S101: Collect network data and parse to obtain traffic data;

Step S102: Calculate and obtain the randomness feature ECF (Encrypted and Compresseddata Feature) feature vector of the traffic data; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fu Leaf transform, non-overlapping matching, serialization and cumulative sum;

Step S103: Take the ECF feature vector as input and perform identification through the machine learning model. The identification results include encrypted traffic and compressed traffic.

Further, the step S101 includes:

Step S101.1: Obtain data packets from the external network and save them as pcap files;

Step S101.2: Divide the obtained data packets into network flows according to the five-tuple (source IP address, destination IP address, source port, destination port, protocol number) and save them as flow files;

Step S101.3: Parse each flow file according to the TCP/IP protocol format, obtain the data payload part, and splice it into a flow data of variable length according to the order in which the data packets are obtained.

Further, the step S102 includes:

Step S102.1: Using bytes as the statistical unit, obtain the length Len of the traffic data;

Step S102.2: Calculate and obtain the ECF feature vector of the traffic data; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fourier transform, non-overlapping matching, Serialization and cumulative sum.

Specifically, using bytes as data statistical elements, the element set is {0x00,0x01...,0xFF}. Assume the input data is data, the data length is Len, and the number of times each byte element appears in data is F. _i , i=0...255. In data statistics, chi-square can well reflect the degree of deviation between the actual observed value of a statistical sample and the theoretically inferred value. Renyi cross entropy (α-Renyi entropy), as a generalization of Shannon entropy, Hartley entropy and minimum entropy, can Great for quantifying uncertainty and randomness in statistics. Therefore, the present invention selects the above two statistical values to analyze the randomness of compressed and encrypted data. Chi-square uses uniform distribution as the theoretical inference value, and the random parameter α of Renyi cross entropy is 0.5. Formula (1) and formula (2) respectively provide the calculation formulas of chi-square and Renyi cross-entropy in this embodiment.

The remaining features use data as a binary data string and input the corresponding detection items in the NIST SP 800-22 random number test set to obtain the detection value p_value.

The ECF feature vector is shown in Table 1. Its vector elements are single-precision floating point numbers with a dimension of 12:

Table 1 ECF feature vector elements

Further, before step S103, it also includes:

Build a machine learning model based on the ECF feature vector; the machine learning algorithms included in the machine learning model include random forest, Xgboost and MLP.

Specifically, building a machine learning model based on ECF feature vectors includes:

In order to simulate the real network traffic detection environment as much as possible, as an implementable method, 6 types of files were selected from network public channels, a basic data set was constructed, and the most widely used encryption and compression algorithms in current Internet communications were used to design To study the data set generation algorithm. Classified according to data length, 7 fixed-size research data sets were constructed, with data lengths ranging from 1KB to 64KB. In the subsequent sections, D_1KB, D_2KB, D_4KB, D_8KB, D_16KB, D_32KB and D_64KB represent the 7 constructed studies. data set.

(1) Basic data set selection

Table 2 shows the source of the basic data set. It should be noted that the original data sets of text, pictures, and videos are relatively large. This embodiment only selects part of the data from the public data set and adds it to the basic data set. Downloaded from the public network The audio files are mainly Chinese and English songs, and the mixed documents are also mainly Chinese and English documents, with a small amount of Japanese documents.

Table 2 Basic data set situation

(2) Encryption and compression algorithm selection:

Table 3 shows the selected encryption and compression algorithms. The encryption mode of the encryption algorithm is CBC mode, the WinRAR version is 5.71 (64-bit), the zip and Gzip algorithms are derived from open source code, and the three encryption algorithms all use cryptoPP-820 code. library implementation.

Table 3 selected encryption and compression algorithms

In order to realize the randomness of encryption and facilitate the verification of the correctness of encryption and decryption, the encryption key, key length and initial IV (basic key) are mainly randomized. The parameter initialization algorithm of the encryption algorithm is seeded with the file name. The initial parameters of the encryption algorithm are pseudo-randomized. Algorithm 1 gives the specific operation process:

(3) Target research data set generation algorithm:

Algorithm 2 provides a basic description of the research data set generation process. All encryption and compression algorithms used are given in Table 2. Each file of the basic data set corresponds to 6 compressed or encrypted files, and then each The encrypted or compressed files are cut into different lengths to form 7 research data sets of different lengths.

Due to the different characteristics of encryption and compression algorithms, the file size of the same file remains unchanged after encryption, but is generally smaller than the original file after compression. In order to maintain the balance of the sample distribution, make the encrypted data, compressed data, and original data sources of the constructed data set equal in total, and provide a more credible research data set for subsequent experiments, based on the minimum number of cuts of the entire data, equal amounts All processed data were randomly selected to form the research data set (D_1KB, D_2KB,...D_64KB). For example, the construction process of D_1KB is as follows: Table 4 shows the size of the basic data set after compression and encryption, and the corresponding number after cutting according to the length of 1KB. It can be seen that the minimum number of cuts is 227169 (the number of cuts after the text is compressed by RAR ), based on 227169, randomly select 227169 pieces of cut data (length 1KB) from all 36 generated compressed or encrypted files to generate a D_1KB data set, then the total amount of data in the D_1KB data set is 8178084 pieces. Through the selection of the above steps, the amount of encrypted or compressed data in the D_1KB data set (4089042 items), the amount of encrypted and compressed data corresponding to each type of original file (681507 items), and the amount of data corresponding to each encryption or compression algorithm (1363014 ) are all equal.

Table 4 Basic situation after compression and encryption of basic data set

(4) Feature vector extraction

ECF feature vector extraction was performed on each research data set.

(5) Select an appropriate classification algorithm, take the ECF feature vector as input, and output encrypted or compressed traffic (corresponding to data labels 0 and 1) to complete the training of the machine learning model. As an implementable manner, this embodiment selects two integrated learning algorithms, random forest and Xgboost, and the MLP deep learning algorithm in machine learning as the classification algorithm of the machine learning model.

Further, the step S103 includes:

Step S103.1: Based on the length Len of the traffic data and the expected test accuracy, select the trained machine learning model for identification;

Step S103.2: Input the ECF feature vector of the traffic data into the machine learning model to obtain the identification results. The identification results include encrypted traffic and compressed traffic.

In order to verify the effectiveness of the present invention, the following experiments were carried out:

In the 7 research data sets constructed and the public data set given in literature 1 (Hahn D, Apthorpe N, Feamster N. DetectingCompressed Cleartext Traffic from Consumer Internet of Things Devices[J]. arXiv preprint arXiv:1805.02722, 2018.) ( detect), the classification effect, generalization performance and computational complexity of the present invention were tested respectively. The ECF feature vector extraction is implemented under the VS2017 platform using C++. The machine learning classification algorithm is implemented under the PyCharm platform using python 2.7. The operating system of the experimental machine is win10 (x64), and the processor is Core ^TM i7-7700 CPU@3.60HZ, memory size 8G.

(a) Classification effect

According to the method of the present invention, ECF feature extraction is first performed on all data of the seven research data sets. After feature extraction, all data were matched with a 12-dimensional ECF feature vector, and then 50% of the data was randomly selected as the training set, and the remaining data was used as the test set. The three classification algorithms selected for the detection model were trained and trained 20 times respectively. test. At the same time, in order to verify the effectiveness of the detection model, the public data set (detect) provided in Document 1 was used to train and test the machine learning model of the present invention. Since the detect data set has a small data size (16,796 items), a 3-fold Using the cross-validation method, the three classification algorithms selected by the machine learning model were trained and tested 30 times respectively.

The present invention uses test accuracy as an evaluation index for model classification effect, and the test accuracy takes the average of the overall accuracy of multiple tests. The overall accuracy of each test = (TP+FN)/(TP+TN+FP+FN), where TP represents the number of encrypted data in the set classified as encrypted data, and FP represents the number of compressed data in the set classified as encrypted data. , TN represents the number of encrypted data in the set classified as compressed data, and FN represents the number of compressed data in the set classified as compressed data. Table 5 gives the main parameters and test accuracy of the detection model. It can be seen that the three classification algorithms, except for the n_estimators parameters of random forest and Xgboost, are slightly different when facing data sets less than 4KB in length. The remaining other parameters are basically the same when facing different data sets, which shows that 3 This classification algorithm has a certain stability when facing data of different lengths. It also shows that using ECF feature vectors can characterize encrypted and compressed data of different lengths. In terms of test accuracy, as the data length increases, the test accuracy also increases, which is consistent with intuitive expectations. Especially when the data reaches 8KB, the classification accuracy reaches more than 92%, which proves that the machine learning proposed by the present invention is Under the model, with only 8KB of data, the ECF feature vector can fully express the difference between encrypted and compressed data.

Table 5 Main parameters and performance of machine learning model

Figure 2 lists the method of the present invention and Document 1 and Document 2 (Casino F, Choo K K R, Patsakis C. HEDGE: Efficient Traffic Classification of Encrypted and Compressed Packets [J]. IEEE Transactions on Information Forensics and Security, 2019: 2916-2926 .) Comparison of the test accuracy of the proposed detection methods. In tests on different data sets, the method of the present invention has achieved better detection results. The method of the present invention achieves a classification accuracy of 99.51% for detection data with a length of 64KB; when the data reaches more than 8KB, a classification accuracy of more than 92% is achieved; when the data is 1KB, a classification accuracy of 71.42% is achieved, compare to literature 2 The proposed HEDGE method has improved test accuracy by 2.74%-7.22% to varying degrees on 7 data sets with data lengths from 1KB to 64KB. Especially on the public data set detect (data length is 1KB), the detection method of the present invention also performs stably. Compared with the method proposed in Document 1 (Daniel Hahn), it has improved by 7%. Compared with the HEDGE method proposed in Reference Document 2 , there was also an increase of 3.3%.

It can be seen from the experiment that when the data length is less than 2KB, improving the classification accuracy is still a difficulty in this work. For example, taking the byte data set {0x00,0x01...,0xFF} as the statistical element, the frequency of each byte element uniformly distributed in 1KB data is 1/256, and the frequency of occurrence is 4 times; evenly distributed in 64KB data The frequency of distribution is also 1/256, and the frequency of occurrence is 256 times. If in the actual data, the frequency of a certain byte increases by 1 time compared with the uniform distribution, then in the 1KB statistical data, its frequency becomes 5 times, and the frequency becomes 5/1024; in the 64KB statistical data, Its frequency becomes 257 times, and its frequency becomes 257/(64*1024). It can be seen that in the face of the same fluctuation with the frequency increased by 1 time, the change rates of frequency on data with lengths of 1KB and 64KB are 25% and 0.39% respectively. This shows that from a statistical point of view, the smaller the amount of data, the smaller the number of bytes. The statistical frequency fluctuates more and more obviously, which leads to the insufficient performance of different characteristics of encryption and compression algorithms. This is also reflected in the work of Reference 2, so it is more difficult to improve the accuracy by 3.3% on 1KB data than on 64K data by 4.8%. In real-life detection, more small data are often encountered, so the improvement of accuracy on small data sets will be more helpful to actual detection work.

In addition, the randomness feature ECF feature vector selected by this invention mainly depicts the randomness of data from a statistical perspective. In comparison, the randomness of encrypted data is better than that of compressed data, which results in a shorter data length. In this case, the recall rate of encrypted data is high and the recall rate of compressed data is low, which leads to low accuracy of encrypted data and high accuracy of compressed data in the classification results. Table 6 gives the detailed report of a single test of Xgboost on the D_1KB data set, from which the above situation can be clearly seen. Therefore, finding a more complete feature set to improve the recall rate of compressed data, thereby improving the encrypted data and overall classification accuracy, is still one of the research directions and challenges for this type of work in the future.

Table 6 Test accuracy report of Xgboost algorithm on 1KB data set

(b) Generalization performance test

(b.1) Generalization performance test between data sets of the same length

This test uses two data sets, D_1KB and detect, both with a data length of 1KB, as training sets to train two machine learning models separately, and uses these two data sets as test sets to test the two trained models. Generalization testing. The test results in Figure 3 show that using the classification model trained on the detect data set, the D_1KB data set was tested, and the highest test accuracy of 70.92% (MLP algorithm) was achieved. The classification model trained on the D_1KB data set was used to test the detect data set. , achieving the highest test accuracy of 72.98% (random forest algorithm). It can be seen that the three classification algorithms selected by the machine learning model of the present invention still achieve good test accuracy when the data lengths of the training set and the test set are the same and the data sources are different.

(b.2) Generalization performance test between data sets of different lengths

This experiment was completed on the 7 research data sets constructed in this embodiment, and mainly tested two types of generalization situations: in the first type of situation, D_1KB was used as the training data set, and the test data set was all research data sets. In the second type of case, D_64KB is taken as the training data set, and the test data set is all research data sets. Figure 4 and Figure 5 show the test results respectively. It can be seen that when the training data set is D_1KB, the trained model can also achieve better classification accuracy on the other 6 research data sets; especially when the data length reaches When the size is more than 8KB, the test accuracy can reach more than 90%. When the training data set is D_64KB, although the trained model performs better on the D_32KB data set, the test accuracy on data sets less than 4KB is lower; especially on the D_1KB data set, compared with the training data set and When the test data sets are all D_1KB, the test accuracy of the model algorithm using Xgboost is reduced by 10%, which has a great impact on data classification with a length of 1KB. The above situation shows that a model trained on a data set with a small data length can be used to detect data with a large length, but the reverse is not feasible.

(c) Time complexity test

In order to test the training time consumption of the three classification algorithms in the model, the average time consumption of 30 training times of each algorithm was taken as the evaluation index of its time complexity, and the time complexity of the three algorithms was briefly evaluated. Table 7 shows the time required for training once on the D_1KB data set when 900,000 training data are selected. It can be seen that under the same conditions of training data and hardware configuration, MLP requires the shortest training time, while Xgboost requires the longest training time (almost 20 times that of MLP).

Table 7 Time comparison of different learning models

At the same time, it was found in the experiment that when the amount of training data set is small, the selection of classification algorithm and the adjustment of algorithm parameters have a greater impact on the test accuracy. For example, on the detect data set (16796 items), the test accuracy of random forest is higher than that of MLP (see Figure 3 for test accuracy); when the training data set is large enough, the test accuracy of the three classification algorithms is almost equal, and the test accuracy is The adjustment of model parameters also becomes insensitive. Figure 6 shows the impact of different classification algorithms on the D_1KB data set and the model parameters (Table 7) unchanged, the amount of training data on the test accuracy (the average of the overall accuracy of 15 training times). When the amount of data is less than 25,000 items, the test accuracy of different classification algorithms differs greatly. At this time, the model parameters can be adjusted to make the accuracy difference of different models smaller; when the amount of data reaches 100,000 items, the three classification algorithms The test accuracy tends to be equal and stable; when there are more than 300,000 items, there is almost no difference between different algorithms. Therefore, when building a model, you need to make flexible choices based on the amount of data in the training data set. When the amount of data is large, you can choose a model (MLP) with less training time. Coarse-grained adjustment of the model parameters can achieve a better result. Good classification effect; when the amount of data is small, a model (random forest) with higher accuracy should be selected, and its parameters should be carefully optimized to achieve better classification effect.

It is worth noting that in order to quickly verify the effect of the present invention, the present invention constructed 7 research data sets (composed of encrypted traffic and compressed traffic). The above experiments were all conducted based on the machine learning model of this embodiment as a two-class model. Those skilled in the art will know that based on the above embodiments, the present invention can also construct a research data set including encrypted traffic, compressed traffic and other traffic (non-encrypted traffic, non-compressed traffic), thereby building a three-class machine learning model , as shown in Figure 7, to conduct related experiments and obtain experimental results similar to those based on the two-classification machine learning model.

Based on the difference in randomness characteristics of encrypted data and compressed data, this paper constructs an effective randomness feature ECF and designs a reasonable machine learning model for traffic identification. It implements the classification of encrypted and compressed traffic based on mainstream machine learning algorithms. . In addition, the present invention constructs a research data set with balanced distribution, and conducts experimental verification on the feature set and recognition model. The results show that the method proposed by the present invention can achieve effective distinction between encrypted and compressed data. It can be seen from the experimental detection effect and generalization effect that the machine learning model proposed by the present invention only relies on the payload data length of the data packet in the actual network traffic test, and has nothing to do with other factors such as data packet sequence and network protocol. Easier to implement cross-platform migration. Since the random characteristics of all public encryption algorithms behave similarly, this method has strong generalization performance. On other data sets, you can directly use the model trained on the current data set to achieve better classification results even without retraining. Of course, if further training is done based on new data sets, the accuracy can be improved to a certain extent.

According to the random feature ECF proposed by the present invention, the ECF feature vector is extracted from the traffic data, and the ECF feature vector is input into the pre-trained machine learning model to identify the type of traffic, including encrypted traffic and compressed traffic. The traditional traffic identification method based on statistical features selects fewer statistical features, has a single discrimination method, and has low identification accuracy for encrypted and compressed traffic. The method provided by the present invention utilizes the learning advantages of machine learning algorithms on big data to design an effective machine learning identification model, overcomes the above shortcomings, improves the identification accuracy of encrypted and compressed traffic, and provides a basis for the refinement of network traffic in network management work. It provides technical support for chemical identification. Especially when partial data is obtained or the amount of data is small, it can still identify encrypted and compressed traffic with high accuracy.

Based on the above embodiments, as shown in Figure 8, the present invention also discloses an encryption and compression traffic identification device based on random characteristics, including:

The collection and analysis module is used to collect network data and analyze the traffic data;

Feature extraction module, used to calculate and obtain the random characteristic ECF feature vector of traffic data; the ECF feature vector includes: chi-square, Renyi cross entropy, single-bit frequency, intra-block frequency, run length, maximum run length, Fourier transform , non-overlapping matching, serialization and cumulative sum;

The traffic identification module is used to use the ECF feature vector as input to identify through the machine learning model. The identification results include encrypted traffic and compressed traffic.

Further, the collection and analysis module includes:

The data packet acquisition submodule is used to obtain data packets from the external network and save them as pcap files;

The network flow division sub-module is used to divide the acquired data packets into network flows according to five-tuple groups and save them as flow files;

The parsing sub-module is used to parse each flow file according to the TCP/IP protocol format, obtain the data payload part, and splice it into a flow data of variable length according to the order in which the data packets are obtained.

Further, the feature extraction module includes:

The traffic data length acquisition submodule is used to obtain the length Len of the traffic data in bytes as the statistical unit;

The feature extraction submodule is used to calculate the ECF feature vector of traffic data.

Furthermore, it also includes:

An identification model building module is used to build a machine learning model based on the ECF feature vector; the machine learning algorithms included in the machine learning model include random forest, Xgboost and MLP.

Further, the traffic identification module includes:

The model selection submodule is used to select a trained machine learning model for identification based on the length Len of the traffic data and the expected test accuracy;

The traffic identification submodule is used to input the ECF feature vector of traffic data into the machine learning model to obtain identification results. The identification results include encrypted traffic and compressed traffic.

What is shown above is only the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications can also be made. should be regarded as the protection scope of the present invention.

Claims (4)

1. An encryption and compression traffic recognition method based on randomness features, comprising:

step 1: collecting network data and analyzing to obtain flow data;

step 2: calculating and obtaining a random characteristic ECF characteristic vector of the flow data; the ECF feature vector includes: chi-square, renyi cross entropy, single bit frequency, intra-block frequency, run, maximum run, fourier transform, non-overlapping matching, serialization, and accumulation sum;

step 3: the ECF feature vector is taken as input, recognition is carried out through a machine learning model, and the recognition result comprises encrypted flow and compressed flow;

the step 1 comprises the following steps:

step 1.1: acquiring a data packet from an external network and storing the data packet as a pcap file;

step 1.2: dividing the acquired data packet into network flows according to the five-tuple, and storing the network flows as flow files;

step 1.3: analyzing each flow file according to a TCP/IP protocol format to obtain a data load part, and splicing the data load part into flow data with an indefinite length according to the sequence of data packet acquisition;

before the step 3, the method further comprises the following steps:

building a machine learning model based on the ECF feature vector; the machine learning algorithm contained in the machine learning model comprises a random forest, xgboost and MLP;

the step 3 comprises the following steps:

step 3.1: selecting a trained machine learning model for recognition according to the length Len of the flow data and the expected test precision;

step 3.2: and inputting the ECF feature vector of the flow data into a machine learning model to obtain a recognition result, wherein the recognition result comprises encrypted flow and compressed flow.

2. The method for identifying encrypted and compressed traffic based on random characteristics according to claim 1, wherein said step 2 comprises:

step 2.1: taking bytes as a statistical unit, and acquiring the length Len of the flow data;

step 2.2: and calculating ECF characteristic vectors of the acquired flow data.

3. An encryption and compression traffic recognition device based on a randomness feature, comprising:

the acquisition analysis module is used for acquiring network data and analyzing the network data to obtain flow data;

the feature extraction module is used for calculating and obtaining a random feature ECF feature vector of the flow data; the ECF feature vector includes: chi-square, renyi cross entropy, single bit frequency, intra-block frequency, run, maximum run, fourier transform, non-overlapping matching, serialization, and accumulation sum;

the flow identification module is used for taking the ECF feature vector as input, identifying through a machine learning model, and the identification result comprises encrypted flow and compressed flow;

the acquisition and analysis module comprises:

the data packet acquisition sub-module is used for acquiring a data packet from an external network and storing the data packet as a pcap file;

the network flow dividing sub-module is used for dividing the acquired data packet into network flows according to the five-tuple and storing the network flows as flow files;

the analysis submodule is used for analyzing each flow file according to the TCP/IP protocol format to obtain a data load part, and splicing the data load part into flow data with an indefinite length according to the sequence of data packet acquisition;

the apparatus further comprises:

the flow identification model construction module is used for constructing a machine learning model based on the ECF feature vector; the machine learning algorithm contained in the machine learning model comprises a random forest, xgboost and MLP;

the flow identification module comprises:

the model selection submodule is used for selecting a trained machine learning model for recognition according to the length Len of the flow data and the expected test precision;

and the flow identification sub-module is used for inputting the ECF feature vector of the flow data into the machine learning model to obtain an identification result, wherein the identification result comprises encrypted flow and compressed flow.

4. The encryption and compression traffic recognition device based on random features of claim 3, wherein the feature extraction module comprises:

the flow data length obtaining sub-module is used for obtaining the length Len of the flow data by taking bytes as a statistical unit;

and the feature extraction sub-module is used for calculating ECF feature vectors for acquiring flow data.