CN102664789B - The processing method of a kind of large-scale data and system - Google Patents

Abstract

The present invention provides a large-scale data processing system and method, wherein the system includes: a flow collection subsystem and a flow processing subsystem; the flow collection subsystem is used to collect data flow, and mirror the collected data flow , and divide the obtained mirrored traffic into P-way sub-traffic and send it to the traffic storage cluster in the traffic processing subsystem, where P is an integer greater than 1; the traffic storage cluster is composed of M storage servers, and each storage server has Hang N disks, M is a positive integer, N is an integer greater than 1, and M×N≥P; each storage server receives the sub-traffic distributed, and uses load balancing technology to write the sub-traffic distributed to N disks attached to it. In this way, the pressure of continuous writing on the disk is reduced, and the problem of large-scale data storage is better solved.

Description

A large-scale data processing method and system

【Technical field】

The invention relates to computer network technology, in particular to a large-scale data processing method and system.

【Background technique】

With the continuous expansion of network users, the amount of data on the Internet has grown explosively, and people have a new understanding of network transmission speed, data security and reliability. User data is widely distributed in many places. For users, data storage and backup that are not well managed pose hidden dangers to business operations. The speed and quality of data transmission affect user experience. In addition, with the development of cloud services Gradually rising and popularizing, the processing needs of large-scale data storage, statistics or analysis have become urgent problems to be solved. However, the existing data processing systems and methods cannot meet the processing requirements of large-scale data due to the impact of performance. For example, if the existing data processing systems and methods are directly applied to the storage of large-scale data, it will bring unbearable Data read and write pressure.

【Content of invention】

The invention provides a large-scale data processing method and system, so as to meet the large-scale data processing requirements.

The specific technical scheme is as follows:

A large-scale data processing system, the system includes: a flow acquisition subsystem and a flow processing subsystem;

The traffic acquisition subsystem is used to collect data traffic, mirror the collected data traffic, and divide the obtained mirrored traffic into P-way sub-traffic and send it to the traffic storage cluster in the traffic processing subsystem, where P is greater than an integer of 1;

The traffic storage cluster is composed of M storage servers, and each storage server is connected with N disks, the M is a positive integer, and N is an integer greater than 1, and M×N≥P; each storage server receives and distributes traffic to sub-traffic, and use load balancing technology to write the split sub-traffic to the N disks attached to it.

According to a preferred embodiment of the present invention, the flow collection subsystem includes:

A traffic collection unit for collecting data traffic at the egress of the external network core switch and mirroring the collected data traffic, and

A split processing unit for splitting mirrored traffic into sub-traffic by using load balancing technology.

According to a preferred embodiment of the present invention, the flow collection unit is composed of an optical splitter and an optical amplifier;

The optical splitter performs optical splitting processing on the data traffic at the exit of the core switch of the external network, and the optical amplifier performs optical amplification on the data traffic after optical splitting processing to form a mirrored traffic.

According to a preferred embodiment of the present invention, the distribution processing unit is a distribution switch, which uses a trunk method to distribute mirrored traffic into P paths of sub-traffic using load balancing technology.

According to a preferred embodiment of the present invention, multiple processes run on each storage server, each process corresponds to a part of the N disks, and each process is responsible for receiving a part of the sub-traffic and the part of the received The sub-flows are written to the corresponding disk in turn according to the preset time length.

According to a preferred embodiment of the present invention, the traffic processing subsystem further includes a real-time analysis cluster;

The traffic acquisition subsystem mirrors the collected data traffic to obtain two mirrored traffics, wherein one mirrored traffic is used to perform the split processing, and the other mirrored traffic is sent to the real-time analysis cluster;

The real-time analysis cluster is used to perform statistics on the traffic information of the received mirrored traffic, and use the statistical results to generate analysis files.

According to a preferred embodiment of the present invention, the real-time analysis cluster includes: a real-time receiving module and a summary statistics module composed of server clusters;

Several servers in the real-time receiving module receive the mirrored traffic, and write the statistical traffic information into a log file;

The summary statistics module downloads the log files generated by the several servers, summarizes the traffic information in each log file to obtain and output an analysis file, wherein the download cycle length is greater than the traffic counted by the real-time receiving module Period length for information to be written to the log file.

According to a preferred embodiment of the present invention, the traffic processing subsystem further includes a non-real-time analysis cluster, which is used to analyze the sub-traffic stored in the traffic storage cluster after summarizing, and the analysis includes: mining of network attack behavior or demand data extraction.

A large-scale data processing method, the method is applied to a large-scale data processing system including a traffic collection subsystem and a traffic processing subsystem, the traffic storage cluster in the traffic processing subsystem is composed of M storage servers, each N disks are hung under the storage server, and the method includes:

The traffic acquisition subsystem collects data traffic, mirrors the collected data traffic, and divides the obtained mirrored traffic into P-way sub-traffic and sends it to the traffic storage cluster, where P is an integer greater than 1;

Each storage server receives the distributed sub-traffic, and uses load balancing technology to write the distributed sub-traffic to the N disks attached to it; wherein M is a positive integer, N is an integer greater than 1, and M×N ≥P.

According to a preferred embodiment of the present invention, the collecting data flow specifically includes: collecting data flow of an external network core switch.

According to a preferred embodiment of the present invention, the mirroring of the collected data traffic is specifically:

An optical splitter is used to split the collected data flow, and an optical amplifier is used to optically amplify the split-processed data flow to form a mirrored flow.

According to a preferred embodiment of the present invention, the splitting of the obtained mirrored traffic into P-path sub-traffic is specifically:

The trunk mode of the distribution switch is used to distribute the mirrored traffic into P-way sub-traffic using load balancing technology.

According to a preferred embodiment of the present invention, the use of load balancing technology to write the distributed sub-traffic to the N disks attached to it is specifically: running multiple processes on each storage server, and each process corresponds to the N disks For some of the disks in the disk, each process is responsible for receiving a part of the sub-traffic and writing the part of the received sub-traffic to the corresponding disk in turn according to the preset time length.

According to a preferred embodiment of the present invention, when the flow collection subsystem mirrors the collected data flow, it obtains two mirrored flows, wherein one mirrored flow is used to perform the splitting process, and the other mirrored flow is sent to a real-time analysis cluster of the traffic processing subsystem;

The real-time analysis cluster performs statistics on the traffic information of the received mirrored traffic, and uses the statistical results to generate an analysis file.

According to a preferred embodiment of the present invention, the statistics of the traffic information of the received mirrored traffic, and using the statistical results to generate an analysis file are specifically:

Several servers in the real-time analysis cluster receive the mirrored traffic, and write the statistical traffic information into a log file;

The summary statistics module in the real-time analysis cluster downloads the log files generated by the several servers, summarizes the traffic information in each log file to obtain and output the analysis file, wherein the download cycle length is longer than the statistics The cycle length for writing traffic information to the log file.

According to a preferred embodiment of the present invention, the method also includes:

The non-real-time analysis cluster summarizes the sub-traffic stored in the traffic storage cluster for analysis, and the analysis includes: mining of network attack behavior or extraction of demand data.

It can be seen from the above technical solutions that in the system and method provided by the present invention, the traffic acquisition subsystem first mirrors the collected data traffic, and then divides the obtained mirrored traffic into multiple sub-traffic flows that are sent to the traffic processing subsystem Storage cluster, traffic storage cluster is composed of several storage servers. Each storage server uses load balancing technology to write the sub-traffic received by the sub-traffic to multiple disks attached to it. In this way, the continuous The pressure of writing can better solve the problem of large-scale data storage, improve disk utilization, and effectively save server costs.

【Description of drawings】

Fig. 1 is a schematic diagram of a large-scale data processing system provided by an embodiment of the present invention;

Fig. 2 is a system example diagram provided by the embodiment of the present invention;

FIG. 3 is a flowchart of a large-scale data processing method provided by an embodiment of the present invention.

【detailed description】

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

First, the large-scale data processing system provided by the present invention will be described. As shown in FIG. 1 , the system may include: a flow collection subsystem 100 and a flow processing subsystem 200 .

The traffic collection subsystem 100 is configured to collect data traffic, and mirror the collected data traffic to the server cluster in the traffic processing subsystem 200 .

Specifically, it may include: a traffic collection unit 110 for collecting data traffic and mirroring the collected data traffic, and may further include: a distribution processing unit 120 for splitting the mirrored traffic into sub-traffic by adopting load balancing technology.

Wherein, when the flow collection unit 110 collects data flow, the collection point can be arranged at the exit of the core switch of the external network. On the one hand, this arrangement can collect all flow data without loss, and on the other hand, it can be completed with fewer collection points. The expected goal is to save costs and reduce engineering difficulty for engineering implementation. In addition, the traffic collection unit 110 may collect data traffic and perform mirroring in the following two ways:

First, the port mirroring method: by mirroring the data of one or more ports of the external network core switch to another or more ports, the collection of data traffic is realized. This method is an existing technology and will not be repeated here. detail.

Second, optical splitting mirroring method: firstly, the export data of the external network core switch is split through the optical splitter. Since the signal strength will be attenuated after optical splitting, the optically amplified traffic after optical splitting can be further carried out to ensure the traffic after optical splitting. The signal strength is sufficient to ensure the integrity and reliability of the data. Compared with the port mirroring method, the optical splitting mirroring method has the advantages of higher stability and reliability. The port mirroring method will affect the core switch itself. For online services, the failure of the core switch will have a fatal impact on the service. Therefore, the spectroscopic mirroring method is used as a preferred data flow collection method.

One path of traffic obtained after mirroring can be sent to the real-time analysis cluster in the traffic processing subsystem 200 for real-time analysis of the traffic, and the other path of traffic can be sent to the distribution processing unit 120 for further processing. The distribution processing unit 120 may be implemented by using a distribution switch. The trunk mode can be used during distribution. The distribution switch uses load balancing technology to distribute the received mirrored traffic into multiple sub-traffic and sends it to the server cluster in the traffic processing subsystem 200 to perform the same processing on the multi-path sub-traffic. Here the main It is to store and process multi-path sub-traffic separately. Taking the processing of 10G data traffic as an example, a 10G port of the switch is used as the ingress port to access the 10G data traffic, and there are 8 Gigabit ports as a trunk at the same time, so the 8 Gigabit ports will be scheduled in round-robin The (round-robin) method evenly distributes the traffic of the ingress port on the 8 Gigabit ports, realizing the first load balancing of high-speed traffic.

Fig. 2 is a schematic diagram of one embodiment of the traffic collection subsystem 100, that is, the splitting device splits the egress traffic of the external network core switch, and the optical amplifier performs optical amplification on the traffic after the splitting process, and then realizes the splitting of the traffic through the splitting switch shunt. One stream of traffic obtained after optical splitting can be sent to the real-time analysis cluster in the traffic processing subsystem 200, and the other stream of traffic obtained after optical splitting can be processed by the splitting switch to obtain each sub-traffic that can be sent to the traffic storage cluster in the traffic processing subsystem 200 , for subsequent non-real-time analysis.

The real-time analysis cluster 210 and the traffic storage cluster 220 in the traffic processing subsystem 200 will be described in detail below.

The real-time analysis cluster 210 performs statistics on the flow information of the received flow, and uses the statistical results to generate an analysis file. Specifically, the real-time analysis cluster 210 may specifically include a real-time receiving module and a summary statistics module (not shown in FIG. 1 ).

The real-time receiving module may be composed of a server cluster, and each server in the server cluster runs the same packet capture and statistical program, and writes the statistical results into a log (log) file. Taking the 10G server as an example, each 10G server supports 2 10G network cards and can handle 20G data traffic at the same time. The packet capture program can complete efficient packet collection from the 10G network card, and the statistics program uses the destination ip as the unit for traffic Separate statistics of information, the statistical content may include but not limited to: tcp flow value, udp flow value, icmp flow value, etc., the unit is usually bps; tcp packet rate, udp packet rate, icmp packet rate, etc., the unit is usually pps; The number of visits per second of non-service ports; the number of get requests per second of http, the length of get data packets; the number of data packets per second of http main status code responses, etc. The statistical results can then be written to the log file in binary format.

The summary statistics module downloads the log files generated by the server cluster of the real-time receiving module, and the download cycle length is usually longer than the cycle length of the real-time receiving module writing the statistical traffic information into the log file. Then, the flow information in each log file is summarized to obtain an analysis file, and the analysis file is output. For example, the traffic information of the same destination IP in each log file can be summarized.

The traffic storage cluster 220 is a cluster composed of M storage servers, where M is a positive integer, and its main function is to write the received traffic to disk in an efficient and reliable manner for storage. Since a large number of data packets are received, and the traffic processed by the actual online application is usually tens or even hundreds of G/s, it is necessary to store large-scale traffic on a slow disk at a relatively low cost. In the present invention, N disks are attached to each storage server, N is a positive integer greater than 1, and M×N≥P, and P is the number of sub-flows obtained after the traffic processing subsystem 200 splits. The storage server receives the distributed traffic, and writes the received traffic to each disk using a load balancing technology. Specifically, the traffic may be written to each disk in turn according to a preset time length. Each storage server can run multiple processes, and each process corresponds to a part of the disk. Each process is responsible for receiving a part of the sub-traffic and writing the part of the sub-traffic in turn according to the preset time length. into the corresponding disk.

To give an example, assume that the traffic storage subsystem includes two storage servers, each of which carries a 4-port Gigabit network card and mounts 8 disks, each 1T. The offload processing unit 120 obtains 8 sub-flows after offloading, and each storage server runs 4 independent processes at the same time, respectively receiving traffic from 4 Gigabit network cards, that is, responsible for receiving 4 sub-flows, and each process corresponds to 2 disks. When each process writes traffic to the disk, the load balancing strategy is adopted again, that is, the second load balancing can be written to two disks in turn in minutes, and the traffic in the first minute is written to the first disk. , the traffic of the second minute is written to the second disk, the traffic of the third minute is written to the first disk, the traffic of the fourth minute is written to the second disk, and so on. This load balancing strategy makes full use of the independence of each process and disk, reduces the pressure of continuous disk writing, better solves the problem of large-scale data storage, improves disk utilization, and effectively saves server costs.

In addition, the traffic processing subsystem 200 may further include a non-real-time analysis cluster 230, which is used for summarizing and analyzing the traffic stored in the traffic storage cluster 220, including but not limited to: mining of network attack behavior or collection of demand data extraction etc.

When mining network attack behavior, the traffic during the attack period can be extracted, and attack behavior analysis can be performed based on the characteristics of the extracted traffic. For example, common network attacks mainly include bandwidth attacks at the network level, synflood and ack flood attacks at the tcp layer, and distributed request attacks at the application layer. Various network attacks will affect the stable operation of the product. Based on the stored historical data, that is, the traffic stored in the traffic storage cluster 220, we can deeply analyze the attack characteristics and provide services for the defense of the product line and the forensics of attack behaviors. For bandwidth-based attacks at the network level, common ones are udp flood and icmp flood. We extract the traffic during the attack period, and then count the size of various types of traffic during this period to determine the attack type and attack scale. For tcp layer protocol stack resource exhaustion attacks, the attack type and attack scale are judged by extracting the traffic during the attack period and counting the packet rates of various types of tcp flags during the period. For distributed request attacks at the application layer, the attack type is determined by extracting data packets during the attack period and counting the fields of the HTTP request header during this period, including host, url, cookie, User-Agent, or referer, and further judging Attacked product lines and related pages, and at the same time summarize the request characteristics of the http header to provide identification marks for the banning strategy.

The current business needs for past access records include tracking problems and offline testing of products, and the extraction of demand data is precisely to meet this demand. The specific implementation method is based on the traffic stored by the traffic storage cluster 220, and the non-real-time analysis cluster 230 extracts the data packets corresponding to the destination IP from the traffic stored by the traffic storage cluster 220 according to the purpose IP of the product line and sends them as a packet capture (pcap) file The format is stored and used to provide the data package to the business demander later.

The large-scale data processing method realized based on the above-mentioned processing system can be shown in Figure 3, mainly including the following steps:

Step 301: the traffic collection subsystem collects data traffic, mirrors the collected data traffic, and executes step 302 with one of the mirrored traffic obtained; sends the other mirrored traffic obtained to the real-time analysis cluster in the traffic processing subsystem, Execute step 305.

When collecting data flow, the collection point can be arranged at the exit of the core switch of the external network, that is, to collect the data flow of the core switch of the external network.

The manner of mirroring the collected data traffic may specifically include the following two types:

First, the port mirroring method: by mirroring the data of one or more ports of the external network core switch to another or more ports, the collection of data traffic is realized. This method is an existing technology and will not be repeated here. detail.

Second, optical splitting mirroring method: firstly, the export data of the external network core switch is split through the optical splitter. Since the signal strength will be attenuated after optical splitting, the optically amplified traffic after optical splitting can be further carried out to ensure the traffic after optical splitting. The signal strength is sufficient to ensure the integrity and reliability of the data. Compared with the port mirroring method, the optical splitting mirroring method has the advantages of higher stability and reliability. The port mirroring method will affect the core switch itself. For online services, the failure of the core switch will have a fatal impact on the service. Therefore, the spectroscopic mirroring method is used as a preferred data flow collection method.

Step 302: Split the mirrored traffic into P-path sub-traffic and send it to the traffic storage cluster in the traffic processing subsystem. P is an integer greater than 1.

The distribution processing performed in this step may be implemented by a distribution switch, which uses a trunk method to distribute the mirrored traffic into P-path sub-traffic by using a load balancing technology.

Step 303: M storage servers in the flow storage cluster respectively receive the distributed sub-traffic, and use load balancing technology to write the distributed sub-traffic to the N disks attached to it; wherein M is a positive integer, and N is An integer greater than 1, and M×N≥P.

The load balancing method adopted in this step can write to the disk in turn according to the preset time length as a unit. Each storage server can run multiple processes, and each process corresponds to a part of the disk. Each process is responsible for receiving part of the sub-traffic and writing the part of the traffic to the corresponding part of the traffic in turn according to the preset time length. disk. This load balancing strategy makes full use of the independence of each process and disk, reduces the pressure of continuous disk writing, better solves the problem of large-scale data storage, improves disk utilization, and effectively saves server costs.

Step 304: The non-real-time analysis cluster in the traffic processing subsystem summarizes the sub-traffic stored in the traffic storage cluster and then conducts analysis. The analysis performed includes but is not limited to: mining of network attack behavior or extraction of demand data.

When mining network attack behavior, the traffic during the attack period can be extracted, and attack behavior analysis can be performed based on the characteristics of the extracted traffic. For example, common network attacks mainly include bandwidth attacks at the network level, synflood and ack flood attacks at the tcp layer, and distributed request attacks at the application layer. Various network attacks will affect the stable operation of the product. We can analyze the attack characteristics in depth based on the stored historical data, that is, the traffic stored in the traffic storage cluster, and provide services for the defense of the product line and the forensics of attack behaviors. For bandwidth-based attacks at the network level, common ones are udpflood and icmp flood. We extract the traffic during the attack period, and then count the various types of traffic during this period to determine the attack type and attack scale. For tcp layer protocol stack resource exhaustion attacks, the attack type and attack scale are judged by extracting the traffic during the attack period and counting the packet rates of various types of tcp flags during the period. For distributed request attacks at the application layer, the attack type is determined by extracting data packets during the attack period and counting the fields of the HTTP request header during this period, including host, url, cookie, User-Agent, or referer, and further judging Attacked product lines and related pages, and at the same time summarize the request characteristics of the http header to provide identification marks for the banning strategy.

The current business needs for past access records include tracking problems and offline testing of products, and the extraction of demand data is precisely to meet this demand. The specific implementation method is based on the traffic stored in the traffic storage cluster. The non-real-time analysis cluster extracts the data packets corresponding to the destination IP from the traffic stored in the traffic storage cluster according to the destination IP of the product line and stores them in a format such as a pcap file for subsequent use. Provide the data package to the business demand side.

Step 305: The real-time analysis cluster performs statistics on the traffic information of the received mirrored traffic, and uses the statistical results to generate an analysis file.

In this step, the mirror traffic received by several servers in the cluster is analyzed in real time, and the statistical traffic information is written into a log file. Then the summary statistics module in the real-time analysis cluster downloads the log files generated by the above-mentioned several servers, summarizes the flow information in each log file to obtain and output the analysis file, and the cycle length of the summary statistics module downloading log files is longer than the above-mentioned several The cycle length for the server to write the statistical traffic information into the log file.

The above-mentioned several servers run the same packet capture and statistics program. The packet capture program can complete the efficient packet collection from the 10 Gigabit network card. The statistics program uses the destination IP as the unit to perform statistics on the flow information. The statistics can include but not limited to: tcp flow value, udp flow value, icmp flow value, etc., the unit is usually bps; tcp packet rate, udp packet rate, icmp packet rate, etc., the unit is usually pps; non-service port access times per second; http get per second The number of requests, the length of the get data packet; the number of data packets responded by the main status code of http per second and other information. The statistical results can then be written to the log file in binary format.

The above-mentioned system and method provided by the present invention realize the storage requirements of large-scale data through traffic mirroring, storage server clusters, and traffic storage load balancing of attached disks, and further realize real-time analysis of large-scale mirrored traffic through real-time analysis clusters The non-real-time analysis requirements for large-scale data are realized through the non-real-time analysis cluster to aggregate and analyze the data stored in the storage server cluster. It has been verified that the present invention can well process data streams with a bandwidth exceeding 100G, and the data has integrity and stability, and has obvious advantages in terms of network equipment cost.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the present invention. within the scope of protection.

Claims (14)

1. A large-scale data processing system, characterized in that the system comprises: a flow collection subsystem and a flow processing subsystem; The flow collection subsystem is used to collect the data flow at the exit of the external network core switch, perform optical splitting processing on the collected data flow, use an optical amplifier to optically amplify the data flow after the splitting processing to form a mirror image flow, and convert the obtained The mirrored traffic distribution is sent to the traffic storage cluster in the traffic processing subsystem as P-way sub-traffic, and P is an integer greater than 1; The traffic storage cluster is composed of M storage servers, and each storage server is connected with N disks, the M is a positive integer, and N is an integer greater than 1, and M×N≥P; each storage server receives and distributes traffic to sub-traffic, and use load balancing technology to write the split sub-traffic to the N disks attached to it. 2. The system according to claim 1, wherein the flow collection subsystem comprises: A traffic collection unit for collecting data traffic at the egress of the external network core switch and mirroring the collected data traffic, and A split processing unit for splitting mirrored traffic into sub-traffic by using load balancing technology. 3. The system according to claim 2, wherein the flow collection unit is composed of an optical splitter and an optical amplifier; The optical splitter performs optical splitting processing on the data traffic at the exit of the core switch of the external network, and the optical amplifier performs optical amplification on the data traffic after optical splitting processing to form a mirrored traffic. 4. The system according to claim 2, wherein the distribution processing unit is a distribution switch, which uses a trunk method to distribute mirrored traffic into P-way sub-traffic using load balancing technology. 5. The system according to claim 1, wherein a plurality of processes are run on each storage server, and each process corresponds to a part of the N disks respectively, and each process is responsible for receiving a part of sub-traffic and The received part of the sub-traffic is sequentially written to the corresponding disk according to the unit of the preset time length. 6. The system according to claim 1, wherein the traffic processing subsystem further comprises a real-time analysis cluster; The traffic acquisition subsystem mirrors the collected data traffic to obtain two mirrored traffics, wherein one mirrored traffic is used to perform the split processing, and the other mirrored traffic is sent to the real-time analysis cluster; The real-time analysis cluster is used to perform statistics on the traffic information of the received mirrored traffic, and use the statistical results to generate analysis files. 7. The system according to claim 6, wherein the real-time analysis cluster comprises: a real-time receiving module and a summary statistics module composed of server clusters; Several servers in the real-time receiving module receive the mirrored traffic, and write the statistical traffic information into a log file; The summary statistics module downloads the log files generated by the several servers, summarizes the traffic information in each log file to obtain and output an analysis file, wherein the download cycle length is greater than the traffic counted by the real-time receiving module Period length for information to be written to the log file. 8. The system according to claim 1, wherein the traffic processing subsystem further includes a non-real-time analysis cluster for analyzing sub-traffic stored in the traffic storage cluster, and the analysis includes: network Attack behavior mining or demand data extraction. 9. A large-scale data processing method, characterized in that the method is applied to a large-scale data processing system comprising a flow acquisition subsystem and a flow processing subsystem, and the flow storage cluster in the flow processing subsystem consists of M units Composed of storage servers, N disks are hung under each storage server, and the method includes: The traffic collection subsystem collects the data traffic at the outlet of the external network core switch, performs optical splitting processing on the collected data traffic, uses an optical amplifier to optically amplify the split-processed data traffic to form mirror traffic, and shunts the obtained mirror traffic Sending sub-traffic of P paths to the traffic storage cluster, where P is an integer greater than 1; Each storage server receives the distributed sub-traffic, and uses load balancing technology to write the distributed sub-traffic to the N disks attached to it; wherein M is a positive integer, N is an integer greater than 1, and M×N ≥P. 10. The method according to claim 9, wherein said splitting the obtained mirrored traffic into P-way sub-traffic is specifically: The trunk mode of the distribution switch is used to distribute the mirrored traffic into P-way sub-traffic using load balancing technology. 11. The method according to claim 9, characterized in that, the use of load balancing technology to write the distributed sub-traffic to the N disks attached to it is specifically: running multiple processes on each storage server, each The processes respectively correspond to some of the N disks, and each process is responsible for receiving a part of the sub-traffic and writing the part of the received sub-traffic to the corresponding disk in turn according to the preset time length. 12. The method according to claim 9, characterized in that, when the traffic collection subsystem mirrors the collected data traffic, it obtains two paths of mirrored traffic, wherein one path of mirrored traffic is used to perform the splitting process, Another path of mirrored traffic is sent to the real-time analysis cluster of the traffic processing subsystem; The real-time analysis cluster performs statistics on the traffic information of the received mirrored traffic, and uses the statistical results to generate an analysis file. 13. The method according to claim 12, characterized in that, performing statistics on the traffic information of the received mirrored traffic, and using the statistical results to generate an analysis file is specifically: Several servers in the real-time analysis cluster receive the mirrored traffic, and write the statistical traffic information into a log file; The summary statistics module in the real-time analysis cluster downloads the log files generated by the several servers, summarizes the traffic information in each log file to obtain and output the analysis file, wherein the download cycle length is longer than the statistics The cycle length for writing traffic information to the log file. 14. The method of claim 9, further comprising: The non-real-time analysis cluster summarizes the sub-traffic stored in the traffic storage cluster for analysis, and the analysis includes: mining of network attack behavior or extraction of demand data.