CN103780610A - Network data recovery method based on protocol characteristics - Google Patents

Abstract

The invention relates to the technical field of the Internet, and specifically discloses a method for recovering network data based on protocol features. The present invention includes the following steps: determining the data message identifier and the data stream identifier; completing data recovery based on protocol features; classifying data streams; analyzing the application layer; The first message of the data flow, if it is, create a new data flow for it, otherwise, discard the data message; the data message is reassembled. Compared with the prior art, the invention has the beneficial effects of realizing real-time collection and recovery of network data, and providing strong technical support for network management, network security monitoring, and network online evidence collection.

Description

Network Data Recovery Method Based on Protocol Features

technical field

The invention relates to the technical field of the Internet, and specifically discloses a method for recovering network data based on protocol features.

Background technique

The rapid development of the Internet has brought huge changes to social production and people's lifestyles. Users can quickly exchange information and data through the Internet, but there are also huge security risks in the process. Network hackers often use the Internet to spread various malicious software, such as placing Trojan horse viruses on web pages for users to download, or attaching them to emails and sending them to users. Therefore, it is necessary to monitor the data flow transmitted in the network, analyze the status of network operation and user behavior, and help network administrators find security problems in time to ensure the normal operation of the network.

Data recovery is to filter and reassemble network data packets, and restore the original data streams carried by these data packets. The current data recovery method mainly judges the application layer protocol used by the data packet according to the commonly used ports of each application layer protocol. For example, port 80 is judged as HTTP protocol, and port 21 is judged as FTP protocol. The defect of this method is that the application layer protocol of the data packet cannot be accurately and comprehensively analyzed, because the HTTP protocol can use other ports, and port 80 may also be used by other protocols. Especially with the widespread use of port conversion technology and the continuous improvement of covert communication technology, most traffic in the network no longer uses standard ports for transmission.

Aiming at the problems existing in the existing data recovery methods, the present invention implements a data recovery method based on protocol features, which can cache, analyze and reorganize intercepted data messages in real time according to the features of the application layer protocol, thereby accurately Completely restore the data stream. This method is widely used in network management, security monitoring, network behavior analysis and other fields.

Contents of the invention

The purpose of the present invention is to reorganize the message according to the characteristics of the application layer protocol used by the message, thereby completely and accurately recovering the data flow carried by the message, and providing technical support for network management, network security monitoring, and online network forensics .

In order to realize the foregoing invention object, the technical scheme that the present invention adopts is as follows:

A network data recovery method based on protocol features, comprising steps:

1. Determine the data message identifier and data flow identifier;

Data packet identifier: The data packet identifier is defined as a quaternion DataPacketID=<SrcIP, SrcPort, DstIP, DstPort>.

Among them, SrcIP, SrcPort, DstIP, and DstPort respectively represent the source IP address, source port, destination IP address and destination port of the data packet. The data packet identifier can be obtained by parsing the network layer (IP) and transport layer (TCP or UDP) headers of the packet.

In a network, a data flow is usually divided into multiple data packets for transmission. In order to restore a certain data flow, it is necessary to obtain all the packets carrying the data flow. Only data packets with the same identifier can be used to reassemble a certain data flow, and omission or misjudgment of a certain data packet will destroy the correctness and integrity of the restored data flow.

Data flow identifier: The data flow identifier is defined as a quaternion DataFlowID=<Protocol, FlowName, Timestamp, DataPacketID>.

Among them, Protocol indicates the protocol for transmitting the data flow, FlowName indicates the name of the transmitted data flow, Timestamp indicates the interception timestamp of the first data packet of the output data flow, and DataPacketID indicates the identifier of the data packet message transmitting the output data flow. The data flow identification needs to be obtained by parsing the application layer data according to the characteristics of the application layer protocol.

The data flow identifier (DataFlowID) is used to identify the data flow transmitted between two computers through a certain application layer protocol.

2. Complete data recovery based on protocol features;

A global IP address needs to be assigned to the nodes on the intranet.

(1) Data collection. The main task of data collection is to obtain all data packets in the network. In this step, any packet capture software can be used to intercept data packets in the network;

(2) Data packet analysis. The original data captured in the network during the data acquisition process is an Ethernet data frame (the message format is shown in Figure 2). The Ethernet data frame must be analyzed at each layer protocol to obtain the required application layer message. The steps are as follows:

① Parse the Ethernet data frame. Taking the most widely used DIX Ethernet V2 protocol as an example, after parsing out, its frame header contains 6 bytes of source MAC address field, 6 bytes of destination MAC address field and 2 bytes of network protocol type Field, the check sequence at the end of the frame is a field with a length of 4 bytes;

② Analyze IP packets. The data after removing the header of the Ethernet frame is an IP packet, and the currently used IP protocol is mainly IPv4. The length of the IP header, the source IP address SrcIP of the message and the destination IP address DstIP can be obtained from the header of the IP message;

③Analysis of TCP/UDP packets. The network transport layer has two protocols, TCP and UDP, which need to be parsed according to different protocol types. Taking a TCP packet as an example, the source port, destination port, sequence number, and length of the header are obtained from the TCP header. After the analysis of the TCP packet is executed, the identifier DataPacketID=<SrcIP, SrcPort, DstIP, DstPort> of the data packet can be obtained.

3. Data flow classification. The main function of data flow classification is to classify a newly intercepted data packet. Classify the message according to the data message identifier DataPacketID, the execution steps are as follows:

(1) If the DaraPacketID belongs to the successor message of a certain existing data flow, then insert it in the message queue of the data flow, go to step 5;

(2) Otherwise, go to step 4;

4. Application layer analysis. Application layer protocol analysis is to analyze application layer messages according to the characteristics of the protocol, determine which application layer protocol they use, extract the required characteristic information from it, and assign a data flow identifier to it;

5. Message selection. Determine whether the data message is the first message to transmit a certain data flow according to the result of the application layer analysis, if so, create a new data flow for it, otherwise, discard the data message;

6. Data packet reassembly. Data message reassembly is mainly responsible for extracting the payload data in the data packet, reassembling into a complete data packet and extracting the application layer message in it, and then recovering the data flow of the message load according to the identifier of the application layer message.

Compared with prior art, the beneficial effect of the present invention is:

1. It realizes real-time collection and recovery of network data, supports both IPv4 and IPv6 networks, supports analysis and decoding of common Internet protocols (such as ICMP, ICMPv6, TCP, UDP, etc.), supports HTTP, FTP, SMTP, POP3 and other common protocols Network data flow analysis, network data recovery, real-time synchronous playback of network data flow, network data flow replay, network data retrieval and archiving;

2. Provide application programming interface API, which can easily realize data exchange and information communication with security management and security monitoring systems such as intrusion detection system, attack source tracking system, and online evidence collection system. The invention can provide strong technical support for network management, network security monitoring and network online evidence collection.

Description of drawings

Fig. 1 is a flow chart of the data recovery method of the present invention;

Fig. 2 is a schematic diagram of the data message format of the present invention;

Fig. 3 PSNDR structural diagram of the present invention;

Fig. 4 is main working flowchart of the present invention;

Fig. 5 is the message parsing flowchart schematic diagram of PSNDR of the present invention;

Fig. 6 is a flow chart of data exchange and information communication of the PSNDR of the present invention.

Fig. 7 is the data display subsystem interface schematic diagram of PSNDR of the present invention;

Fig. 8 is an example of implementation of the present invention.

Detailed ways

The above content of the invention of the present invention will be further described in detail below in conjunction with specific embodiments.

However, it should not be construed that the scope of the above-mentioned subject matter of the present invention is limited to the following examples. Without departing from the above-mentioned technical idea of the present invention, various replacements and changes made according to common technical knowledge and customary means in this field shall be included in the scope of the present invention.

Example

The network data recovery method based on the protocol feature that the present embodiment enumerates comprises the following steps:

1. Network data recovery system structure and workflow based on protocol features;

In the specific implementation process, we installed and deployed the network data recovery system PSNDR based on the protocol characteristics developed and realized based on this method on a Linux operating system CentOS6.0, as shown in Figure 3, PSNDR is mainly composed of data collection, message analysis , data flow classification, application layer analysis, message reassembly, data display, data storage, and management control modules. As shown in Figure 4, the implementation process mainly includes the following steps:

(1) After the network data recovery system based on protocol features is started, it starts to monitor network data and perform initialization work: open the data acquisition module, in order to improve work efficiency, the system uses Libpcap to intercept data messages in the network; initialize message identification character set S_DataPacketID; initialize data flow identifier set S_DataFlowID;

(2) Through the scheduling of the management control unit, the system submits the collected raw data to the message analysis module for analysis and processing;

(3) As shown in Figure 5, the message parsing module parses the received message, and resolves the Ethernet data frame, IP message, TCP/UPD message in turn, and finally obtains the identifier of the data message;

(4) The message identifier returned by the message parsing module is passed to the data flow classification module through the management control unit as a feature of message classification. If the identifier DataPacketID of message belongs to message identifier collection S_DataPacketID, then DataPacketID belongs to the successor message of certain existing data flow, then it is inserted in the message queue of this data flow, turn step (6); Otherwise, It is necessary to analyze the application layer of the message, and go to step (5);

(5) Application layer analysis. The main function of the application layer protocol analysis module is to analyze the application layer messages according to the characteristics of the protocol, determine which application layer protocol they use, and extract the required characteristic information from it, and finally return a data flow identifier DataFlowID. The application layer analysis module mainly implements the analysis process of several common application layer protocols such as HTTP, FTP, SMTP and POP3:

① HTTP protocol analysis. According to the provisions of the HTTP protocol, the client sends a GET request to the server to start the file transfer session, and the file content is sent to the client along with the server's response. This method judges whether the message belongs to the initial session process of HTTP file transmission by checking whether the data load header of the message is "GET", and then adds the data flow responded by the server to the cached flow classification list. If the data packet load in the reverse direction has a header in the form of "HTTP/*.*200", it means that these data packets are loaded with the data of the file sent by the server, and this part of the packet needs to be cached for use. Subsequent data reorganization. Go to step (6);

②FTP protocol analysis. The FTP protocol divides the control and data into two different network connections, and the communication parties negotiate the address and port number of the data connection in the session. Therefore, it is necessary to continuously monitor several sessions of the control connection to obtain sufficient information. The format of the FTP transfer file is divided into two modes: PORT and PASV, which need to be parsed separately;

Parsing of PORT mode. The PORT mode is characterized by the beginning of the "PORT" string in the payload, and its command format is "PORT h1, h2, h3, h4, p1, p2" (where h1, h2, h3, and h4 respectively correspond to the IP addresses used by the server to transmit data. The four-segment decimal number of the address, p1, p2 represent the high 8-bit and low 8-bit decimal numbers of the port where the server transmits data). The parser obtains information such as the network address and port from the PORT command, and then obtains the required data packets by monitoring the RETR or STOR command in the connection. Go to step (6);

Parsing of PASV mode. The PASV mode is characterized by the beginning of the "PASV" string in the payload. The network address and port used for data transmission exist in the 227 response message of the PASV mode, and its format is "Entering Passive Mode(h1, h2, h3, h4, p1, p2)" (the meaning of the parameter is the same as the PORT command), after which the client will send a RETR or STOR command to upload or download a file in this network connection. Go to step (6).

③ SMTP agreement. The SMTP protocol is the protocol used by the mail client to send mail to the server. According to the protocol, when the client sends an email to the server, the message carrying the email data starts with "DATA", and the text of the mail including the attachment will be Transmit in the same direction on the same connection. It only needs to reassemble after caching all the data packets in the connection. Go to step (6);

④ POP3 agreement. The POP3 protocol is a communication protocol used by the mail client to request the server to receive e-mail. When the client sends a receiving request to the server, the request message starts with "Received", and then the data message sent by the server is loaded with the relevant data of the requested email. It is only necessary to cache these data packets and then reassemble them. Go to step (6).

(6) judge whether this data message is the first message of transmitting a certain data stream according to the analysis result, if yes, then create a new data stream for it, otherwise discard this data message;

(7) Data packet reassembly. The data message reassembly module is mainly responsible for extracting the payload data in the data packet, reorganizing into a complete data packet and extracting the application layer message in it, and then recovering the data flow of the message load according to the identifier of the application layer message. Since the IP protocol provides an unreliable connectionless service, the IP packet may be lost and fail to reach the destination after being transmitted through the network, or the IP packet sent first may arrive later than the IP packet sent later. When an IP packet PakN is intercepted, the following situations may occur, which need to be handled separately:

①PakN is a repeated message. The handling of this situation is the simplest, only need to discard the duplicate packet PakN;

② When PakN is intercepted, the part of the message before its serial number has not been intercepted. In this case, PakN should temporarily cache and wait for the delayed message to be intercepted;

③PakN was intercepted in the expected order. This situation is relatively complicated to deal with. Although the new message does not need to be reassembled, it may activate the packets that arrived earlier in the cache. Therefore, it is also necessary to process the data packets intercepted in advance in the cache, and reorganize the packets whose serial numbers meet the reassembly requirements. The message is reassembled with previous messages until the last data message is obtained.

(8) Data storage and display. After the data flow of the transmission data flow ends, the management control unit will process the restored complete data flow returned by the data message reassembly module to the data storage module, and write the reorganized message data from the cache to the disk, and then the The original data flow is restored. At the same time, the management control unit invokes the data display module for front-end display, as shown in Figure 7 is a screenshot of the data display subsystem, which only needs to be input.

2. Operation example of network data recovery system PSNDR based on protocol features

After the network data recovery system PSNDR based on protocol characteristics is running, the front-end system mainly includes the protocol analysis and data recovery management subsystem (as shown in Figure 7) and the data query and display subsystem (as shown in Figure 8).

Figure 7 shows the management interface of the protocol analysis and data recovery management subsystem of PSNDR. Users only need to click a monitored object in the left pane to see very detailed and rich content in the right pane. , such as raw data, protocol decoding data of each layer, the access target of the monitored object, and the time of occurrence, etc., which reflect the behavior characteristics of the monitored object.

Figure 8 shows the main interface of PSNDR's data query and display subsystem. When users browse the web, PSNDR captures the user's network traffic through the data acquisition module, and displays it synchronously in the front-end display system after being processed by the background analysis and decoding system. The pages viewed.

Claims (10)

1. The network data recovery method based on the protocol feature, is characterized in that, comprises steps: Determining the data packet identifier and the data stream identifier; Complete data recovery based on protocol features; data flow classification; Application layer analysis; Message selection; Datagram reassembly. 2. the network data recovery method based on protocol feature according to claim 1, is characterized in that, described data packet identifier is defined as a quadruple DataPacketID=<SrcIP, SrcPort, DstIP, DstPort>, wherein SrcIP, SrcPort, DstIP, DstPort represent the source IP address of data message respectively, source port, purpose IP address and purpose port; ) head is obtained. 3. The network data recovery method based on protocol features according to claim 1, wherein the data flow identifier is defined as a four-tuple DataFlowID=<Protocol, FlowName, Timestamp, DataPacketID>, wherein Protocol represents transmission The protocol of the data stream, FlowName represents the name of the transmitted data stream, Timestamp represents the interception timestamp of the first data packet of the data stream, and DataPacketID represents the identifier of the packet message transmitting the data stream; the data stream The identifier needs to be obtained by parsing the application layer data according to the characteristics of the application layer protocol. 4. the network data recovery method based on protocol feature according to claim 1, is characterized in that, described data recovery based on protocol feature comprises: data collection and data message analysis; Described data collection can use any packet capture software to intercept the data message in the network; the data message analyzing party includes: analyzing the Ethernet data frame, analyzing the IP message, and analyzing the TCP/UDP message. 5. the network data recovery method based on protocol feature according to claim 4, is characterized in that, the analysis method of described Ethernet data frame is: take the DIX Ethernet V2 agreement that is most widely used at present as an example, after analyzing , the frame header contains a 6-byte source MAC address field, a 6-byte destination MAC address field, and a 2-byte network protocol type field, and the check sequence at the end of the frame is a 4-byte long field. 6. the network data restoration method based on protocol feature according to claim 4, is characterized in that, the analytical method of described IP message is: remove the data behind the Ethernet frame head and be IP message, the currently used IP The protocol is mainly IPv4, and the length of the IP header, the source IP address SrcIP of the message, and the destination IP address DstIP can be obtained from the header of the IP packet. 7. the network data restoration method based on protocol feature according to claim 4, is characterized in that, the analytical method of described TCP/UDP message is: take TCP message as example, obtain source port, purpose from TCP header port, serial number and length of the header; after the analysis of the TCP message is executed, the identifier DataPacketID=<SrcIP, SrcPort, DstIP, DstPort> of the data message can be obtained. 8. The network data recovery method based on protocol features according to claim 1, wherein the data stream classification method is: classify the message according to the data message identifier DataPacketID, if the DataPacketID belongs to a certain If there is a subsequent packet of the data flow, insert it into the packet queue of the data flow, and judge whether the data packet is the first packet of a certain data flow according to the analysis result of the application layer, and if so, it is It creates a new data stream, otherwise, discards the data message; if the DataPacketID is not a successor message of an existing data stream, judge which application layer protocol it uses, and extract the required characteristic information from it, for its Assign a stream identifier. 9. the network data restoration method based on protocol characteristic according to claim 1, is characterized in that, described application layer analysis is to analyze application layer message according to the characteristic of agreement, judges which application layer protocol they have used, and Extract the required characteristic information therefrom, and assign a data flow identifier to it; the choice of the message is to judge whether the data message is the first message to transmit a certain data flow according to the result of the application layer analysis, if it is , create a new data stream for it, otherwise, discard the data packet. 10. The method for recovering network data based on protocol features according to claim 1, wherein said data message reassembly is mainly responsible for extracting the load data in the data packet, reorganizing into a complete data packet and extracting the application data therein. layer message, and then recover the data flow carried by the message according to the identifier of the application layer message.

Images