CN101645806A - Network flow classifying system and network flow classifying method combining DPI and DFI - Google Patents

Abstract

The invention discloses a network traffic classification system and a classification method combining DPI and DFI, which are formed by combining two modules of a DPI business identification system and a DFI traffic identification system; wherein the DPI module further includes a flow table detection module and a traffic identification module ; The DFI module includes a sample acquisition module, a classifier training module and a classifier classification prediction module. The sample acquisition module divides the data flow that can be accurately identified by the traffic identification module in DPI into several major categories, and uses them as samples to train the classifier training module to obtain a classification model that can distinguish between major categories of network traffic, and then The traffic that cannot be identified by the traffic identification module of DPI can be classified and predicted by the classifier of DFI to achieve the purpose of classifying the traffic that cannot be identified by DPI. The present invention is more comprehensive than simply using DPI or DFI, can accurately identify services that are not encrypted at the application layer, and can also distinguish large categories of services that are encrypted at the application layer.

Description

Network Traffic Classification System and Classification Method Combining DPI and DFI

technical field

The invention relates to a network flow classification system and a classification method combining DPI and DFI, belonging to the field of network data transmission.

Background technique

With the continuous emergence of network applications, emerging services such as P2P, online games, IPTV, and WEBTV occupy most of the bandwidth of the Internet. P2P applications represented by BT and Edonkey have already occupied more than 2/3 of the entire Internet traffic. Network construction has fallen into an abnormal situation of "congestion-expansion-re-congestion", and the profitability is correspondingly reduced. Failure to realize service identification increases the operator's operating costs and reduces customer satisfaction. Therefore, how to deeply perceive network applications, provide network service control and management methods, build a harmonious network that can be operated and managed, effectively restrict P2P, guide it reasonably, and turn disadvantages to my own use, has become an urgent need for research by telecom operators. A hot topic.

Based on the above reasons, different network data flows must be identified through technical means so that they can be controlled and managed.

Currently, there are mainly the following methods for identifying network data flow services:

(1) Port-based network data flow service identification technology: This identification technology is identified through different port numbers registered in the IANA (Internet Assigned Numbers Authority) by various applications. For example, when the detected port number is 80, it is considered that the application represents a common Internet access application. However, some illegal applications on the current network hide or forge port numbers to evade detection and supervision, causing the data flow of counterfeit legal packets to erode the network. For example, the port used by the new P2P protocol changes, so the accuracy of port number identification has become lower and lower, and this method has become increasingly unsuitable for identification of existing network data flow services.

(2) DPI (Deep Packet Inspection) deep packet inspection network data flow service identification technology: When encountering some new protocols that use dynamic ports, the port-based identification technology will be powerless. In addition to analyzing the basic information below the 4th layer, DPI technology also adds application layer analysis to identify various applications and their contents. It is to identify various services by analyzing the application layer load characteristics of a series of data packets to find out the characteristic words of the application layer. This method will be very difficult to deal with when encountering application layer data encryption.

(3) DFI (Deep Flow Inspection) deep flow inspection network data flow service identification technology: When DPI identification technology encounters application layer data encryption, it is difficult to identify it by analyzing the characteristics of application layer data. DFI technology is a technology for identifying services based on flow characteristics, that is, different application types have different states reflected in session connections or data flows. The characteristic of DFI is to analyze the characteristics of the entire data flow, such as the average packet length of each flow, the time interval between the arrival of each packet, and so on. There is no need to detect the application layer data, so whether the application layer data is encrypted or not has no difference for this identification technology. The characteristics of the data flow belonging to the same type of business are generally very close. For example, the traffic characteristics of the two IM software, QQ and MSN, may be very close. Therefore, the disadvantage of this method is that it can only analyze several large network traffic. class to distinguish. Such as IM, P2P, WEB, etc.

Contents of the invention

Purpose of the invention:

The technical problem to be solved by the present invention is that, for the low accuracy rate of the port identification technology, the DPI and DFI technologies respectively have the defects that it is very difficult to identify the business of application layer data encryption, and they can only distinguish between large categories of network traffic. A network traffic classification system and classification method combining DPI and DFI are proposed.

Technical solutions:

The technical solution adopted by the present invention to solve the technical problem is: first classify the network traffic into categories, then construct a DPI network data flow business identification system, and extract the application layer features for the business without encryption in the application layer, and extract the extracted The features are put into the feature library, and then the data flow of the protocol that DPI can recognize is used as the sample of the DFI business recognition module to train DFI. After the training is completed, the DFI module is added behind the DPI business recognition system, so that the data that DPI cannot recognize The stream is then identified by the DFI module, so that the data streams that cannot be identified by the DPI are classified into categories. The specific technical scheme is as follows:

The network traffic classification system combining DPI and DFI of the present invention is formed by combining two modules of a DPI business identification system and a DFI traffic identification system;

In the described DPI service identification system, comprising:

A. The flow table detection module determines whether the current data flow is a marked type of data flow;

B. Data stream feature library, which stores the features of data stream;

C. The traffic identification module identifies different services represented by network traffic according to the characteristics in the data flow feature database;

D. The protocol processing module is used for processing specific services and processing network categories;

The DFI traffic identification system includes:

E. The sample acquisition module is used to extract the flow characteristics of the business that can be accurately identified by the DPI, and divide them into different categories as training samples for the classifier training module;

F. The classifier training module trains the samples provided by the sample acquisition module to obtain a training model;

G. The classifier classification prediction module classifies other data according to the model obtained by the classifier training module;

The present invention also provides a network traffic classification method based on the present invention, comprising the following steps:

(a) The data flow first passes through the flow table detection module in the DPI service identification system, and the flow table detection module detects whether the current data flow is in the state table maintained by the flow table detection module. When the data flow is in the state table, the flow table After the detection module directly marks the current data flow, it is sent to the protocol processing module; when the data flow is not in the state table, the flow label detection module sends the data flow to the traffic identification module, and enters (b) step;

(b) The traffic identification module checks whether the data stream contains any feature in the data stream feature library in the DPI business identification system; when the traffic identification module identifies a matching traffic feature with the data stream in the data stream feature library, Then mark the data flow corresponding to the current message as a specific data flow, update the state table maintained in the flow table detection module, and send the current data flow mark to the protocol processing module; when the traffic identification module is in the data flow feature library If no traffic characteristic matching the data flow is identified, the data flow is sent to the DFI traffic identification system, and step (c) is entered;

(c) The traffic identification module sends the identifiable data stream to the sample acquisition module in the DFI traffic identification system. After the sample acquisition module obtains the sample file of the data stream online, it sends the sample file to the classifier training module for offline training , to obtain a classification model, the classifier training module sends the classification model to the classifier classification prediction module; the classifier classification prediction module classifies the data streams that cannot be identified by the traffic identification module in (b) step according to the classification model obtained through training;

(d) The classification prediction module of the classifier will mark the classified data streams and send them to the protocol processing module, and the protocol processing module will perform specific services or processes for different categories according to the different marks of the data streams in the above steps .

In the DPI service identification system of the present invention, the data flow feature library includes application layer features of some services in each category of network traffic. For example, services belonging to the category of instant messaging include QQ and Baidu HI. The application layer characteristic of QQ is that the data packet starts with 0x02 and ends with 0x03. The application layer characteristic of Baidu HI is that the first eight bytes are 0x0000010031564d49. Services belonging to the P2P category include TTlive and Sopcast, etc. The application layer of TTlive is characterized by the fact that the payload length of the first packet of each flow is 52 bytes, the first three bytes are 0xffff01, and the last two bytes are 0xffff01. It is 0x0002, and the Sopcast application layer feature is the feature word of the first data packet with payload, which is expressed by regular expression as: ^DESCRIBE.*User-Agent: WMPlayer.

Beneficial effect

The network traffic classification system and method combining DPI and DFI of the present invention firstly performs DPI identification on network data, and the data flow that cannot be identified by DPI is then classified through DFI, which increases the accuracy of classifying network traffic.

Description of drawings

Fig. 1 is a structural block diagram of a DPI identification module;

Fig. 2 is a structural block diagram of a DFI identification module;

Fig. 3 is the block diagram of the network traffic classification method that DPI and DFI combine in the present invention;

Fig. 4 is a flow chart of the network traffic classification method combining DPI and DFI according to the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

As shown in Figure 1, in the first implementation step of the network traffic classification system combining DPI and DFI of the present invention, the network traffic identification system is connected in the network based on TCP/IP protocol, wherein there is a flow table detection module, A protocol processing module, a traffic identification module and a data flow feature library.

The data flow feature library includes various services belonging to several network traffic categories. Examples are as follows:

(1) There are QQ and Baidu HI belonging to the category of IM (Instant Messaging), etc. The application layer characteristic of QQ is that the data packet starts with 0x02 and ends with 0x03, and the application layer characteristic of Baidu HI is that the first eight bytes are 0x0000010031564d49.

(2) P2P services include TTlive and Sopcast, etc. The application layer of TTlive is characterized by the fact that the payload length of the first packet of each stream is 52 bytes, the first three bytes are 0xffff01, and the last two bytes are 0xffff01. A byte is 0x0002, and the Sopcast application layer feature is the feature word of the first data packet with a payload. The regular expression is expressed as: ^DESCRIBE.*User-Agent: WMPlayer.

The characteristics of the above-mentioned various services are stored in the data flow characteristic library.

The flow table detection module maintains a state table. The information in the table includes the quintuple of data flow (source ip address, destination ip address, source port, destination port, protocol number) and the ID of the protocol type. After the network data flow enters First compare your own quintuple with the information in the state table to see if it is in the state table, if it is in the state table, mark it with the ID of the protocol type it belongs to and send it to the protocol processing module.

For example, the format of a piece of information maintained in the status table is as follows in the second line of the table

source ip address destination ip address source port destination port agreement type Protocol ID 119.147.18.47 10.8.7.43 8000 4000 0x11 5

Among them, 119.147.18.47 is the source IP address, 10.8.7.43 is the destination IP address, 8000 is the source port, 4000 is the destination port, 0x11 is the protocol number (UDP protocol), and 5 is the protocol ID that can be defined by ourselves. The protocol ID is set to 5, then 5 represents the data flow of QQ. Once a new data flow enters the flow table detection module, first compare its own five-tuple with the first five items (five-tuple) of the information in the table, if it is found that there is its own five-tuple in the state table, then The data flow is marked with the protocol ID and then sent to the protocol processing module. If no record matching its own quintuple is found in the state table, it will enter the traffic identification module.

The traffic identification module first analyzes the application layer data of the network data flow, and compares its application layer characteristics with the characteristics in the data flow characteristic database. If the characteristic string of the application layer data matches one or more of the data flow characteristic database feature, the traffic identification module marks it as the corresponding protocol ID, and updates the traffic to the flow table detection module. If there is no feature matching its feature string in the data flow feature database, the data traffic identification module is wrong. Instead of marking it, it is fed into the DFI recognition module, which further recognizes it.

The data flow feature database stores the application layer feature words of the previously identified services. For example, the first 20 bytes of the application layer of bitspirit are always 0x13426974546f7272656e742070726f746f636f6c, and the first 5 bytes of the application layer are always 0x3c00000001 when PP downloads files. The traffic identification module judges whether the data stream can be identified and which protocol it belongs to by comparing it with the characteristics in the library.

As shown in Figure 2, it is a structural block diagram of the DFI part of the network traffic classification system combining DPI and DFI, which mainly includes a sample acquisition module, a classifier training module, and a classifier classification prediction module, and the sample acquisition module will be as shown in Figure 1. The data flow that can be accurately identified by the traffic identification module in the network is used as a sample, and it is classified into several categories of network traffic that have been divided before, and the required flow characteristics are extracted from it. For example, QQ is a data flow that the traffic identification module can accurately identify , and QQ belongs to the category of IM (instant messaging), then each QQ network data stream can be used as a sample of the category of IM. Similarly, we can also accurately identify Baidu HI, and Baidu HI also belongs to the category of IM, so each Baidu HI network data stream can also be used as a sample of the category of IM. After obtaining the sample, we calculate the flow characteristics of each sample, such as the average packet length of the flow, the average time interval of the packet, etc., and mark this sample to determine the category it belongs to. Using the same method, we can extract samples of the P2P category and several other categories of samples from TTlive and Sopcast network data streams, and we can obtain a sample file by combining all these samples together. Its file format is as follows:

Each row in the file represents a sample, and the first character in each column indicates the category to which the row sample belongs. For example, we use the ID 1 for the category P2P, and the category IM (instant messaging) Use 2 to represent the category of WEB applications as 3, then the first and third lines of this file represent P2P sample data, the second line represents IM (instant messaging) sample data, and the fourth line Indicates the sample data of the WEB application. The category ID of each line of the file is followed by the feature index and the value of the feature. For example, if we index the flow feature of the average packet length of the flow with 1, and use the average time interval between packet arrivals with 2, it means that the first line indicates that The average packet length of this sample data is 1000, and the average time interval between packet arrivals is 0.005. There must be more than two features of each stream, other features will not be listed here. The role of the sample acquisition module is to extract the flow characteristics from the data flow that can be accurately identified by the traffic identification module, and save the characteristics in the form of sample files.

The classifier training module obtains a prediction model by training the samples obtained by the sample obtaining module.

The classification prediction module of the classifier classifies the traffic that cannot be identified by the traffic identification module through the prediction model.

Figure 3 is the combination of the DPI identification module and the DFI identification module, which can be divided into two categories: online and offline, flow table detection module, protocol processing module, traffic identification module, data flow feature library, sample acquisition module, classifier classification The prediction module is online, and the classifier training module is offline. Before performing online classification, it is necessary to carry out the process of sample acquisition and classifier training to generate a classification model. At this time, the traffic identification module will directly send the data flow that can be accurately identified to the sample acquisition module.

After the sample acquisition module obtains the sample files online, it can perform offline training on the classifier to obtain the classification model. When the traffic identification module in the DPI system cannot identify it, it will pass through the classifier classification prediction module of the DFI system. The classifier classification prediction module is based on the training The obtained classification model classifies the data flow that cannot be identified by the traffic identification module.

Fig. 4 is a flow chart of the network traffic classification method combining DPI and DFI according to the present invention.

The flow chart is the processing process of network data during online classification, the premise is that the classifier has been trained and the classification model has been obtained.

First, when network traffic arrives, it first reaches the flow table detection module, and detects whether the current packet has been marked according to the header in the packet. If the type of the data flow corresponding to the current packet has been marked, the current data flow is processed in a manner corresponding to the type. If the type of the corresponding data flow of the current message is not marked, then enter the traffic identification module to identify and judge, the basis for identification of the traffic identification module is the data flow feature library in Figure 1, if the traffic identification module can identify, then update the flow table detection module, In order to enable packets belonging to the same flow to be detected during flow table detection. If the traffic identification module cannot identify it, it enters the classifier classification prediction module, and the classifier classification prediction module classifies the unidentifiable traffic according to the classification model obtained by DFI offline training. Since all network data traffic must belong to one of several major categories, all traffic that cannot be identified by the traffic identification module of the DPI is classified according to the major categories. After the classification is completed, it is sent to the protocol processing module, and the protocol processing module performs processing according to different categories. The protocol processing module here includes two processing objects, one is processing specific services, and the other is processing network categories.

Processing network traffic through the above method is more comprehensive than simply using DPI or DFI. It can accurately identify services that are not encrypted at the application layer, and can also distinguish between categories of encrypted services at the application layer.

Claims (2)

1. The network traffic classification system combining DPI and DFI is characterized in that it is composed of two modules: DPI service identification system and DFI traffic identification system; In the described DPI service identification system, comprising: A. The flow table detection module determines whether the current data flow is a marked type of data flow; B. Data stream feature library, which stores the features of data stream; C. The traffic identification module identifies different services represented by network traffic according to the characteristics in the data flow feature database; D. The protocol processing module is used for processing specific services and processing network categories; The DFI traffic identification system includes: E. The sample acquisition module is used to extract the flow characteristics of the business that can be accurately identified by the DPI, and divide them into different categories as training samples for the classifier training module; F. The classifier training module trains the samples provided by the sample acquisition module to obtain a training model; G. The classifier classification prediction module classifies other data according to the model obtained by the classifier training module. 2. A network traffic classification method based on a network traffic classification system combining DPI and DFI according to claim 1, comprising the following steps: (a) The data flow first passes through the flow table detection module in the DPI service identification system, and the flow table detection module detects whether the current data flow is in the state table maintained by the flow table detection module. When the data flow is in the state table, the flow table After the detection module directly marks the current data flow, it is sent to the protocol processing module; when the data flow is not in the state table, the flow label detection module sends the data flow to the traffic identification module, and enters (b) step; (b) The traffic identification module checks whether the data stream contains any feature in the data stream feature library in the DPI business identification system; when the traffic identification module identifies a matching traffic feature with the data stream in the data stream feature library, Then mark the data flow corresponding to the current message as a specific data flow, update the state table maintained in the flow table detection module, and send the current data flow mark to the protocol processing module; when the traffic identification module is in the data flow feature library If no traffic characteristic matching the data flow is identified, the data flow is sent to the DFI traffic identification system, and step (c) is entered; (c) The traffic identification module sends the identifiable data stream to the sample acquisition module in the DFI traffic identification system. After the sample acquisition module obtains the sample file of the data stream online, it sends the sample file to the classifier training module for offline training , to obtain a classification model, the classifier training module sends the classification model to the classifier classification prediction module; the classifier classification prediction module classifies the data streams that cannot be identified by the traffic identification module in (b) step according to the classification model obtained through training; (d) The classification prediction module of the classifier will mark the classified data streams and send them to the protocol processing module, and the protocol processing module will perform specific services or processes for different categories according to the different marks of the data streams in the above steps .

Images