CN105138661B - A kind of network security daily record k-means cluster analysis systems and method based on Hadoop - Google Patents

Abstract

A Hadoop-based network security log k-means cluster analysis system and method, including a log data acquisition subsystem, a log data mixing mechanism storage management subsystem, and a log data analysis subsystem; in the data storage layer, Hadoop and traditional data are used The hybrid storage mechanism of warehouse collaboration stores log data and provides an interface for Hive operations at the data access layer. The data storage layer and computing layer receive instructions from the Hive engine, and through HDFS, cooperate with MapReduce to realize efficient query and analysis of data; When data is mined and analyzed, MapReduce is used to implement the k-means algorithm for cluster mining analysis; Hadoop and the traditional data warehouse collaboration architecture are used to make up for the shortcomings of traditional data warehouses in massive data processing and storage, and also make the original The traditional data warehouse makes the best use of everything; the MapReduce-based k-means algorithm is used for cluster analysis, and the security level assessment and early warning of log data can be performed in a timely manner.

Description

A Hadoop-based network security log k-means clustering analysis system and method

technical field

The invention belongs to the technical field of computer information processing, and in particular relates to a Hadoop-based network security log k-means cluster analysis system and method.

Background technique

With the explosion of data and the sharp increase in the amount of information, the existing traditional data warehouses of enterprises have been unable to cope with the growth rate of data. Traditional data warehouses are usually built with high-performance all-in-one machines, which are costly and poor in scalability, and traditional data warehouses are only good at processing structured data. This feature affects the mining of intrinsic value of traditional data warehouses when faced with massive heterogeneous data. This is the biggest difference between Hadoop and traditional data processing methods. We need to make reasonable use of the existing traditional data warehouse of the enterprise, and at the same time integrate the existing traditional data warehouse and big data platform to establish a unified data analysis and data processing architecture, so that through the collaboration between Hadoop and traditional data warehouse Monitoring and statistical analysis of network logs.

Hadoop is an open source distributed computing platform managed by the Apache organization and a software framework capable of distributed processing of large amounts of data. Hadoop, with the Hadoop distributed file system HDFS and MapReduce as the core, provides users with a distributed infrastructure that is transparent to the underlying details of the system. The advantages of HDFS, such as high fault tolerance, high scalability, high scalability, high availability, and high throughput, allow users to deploy Hadoop on low-cost hardware to form a distributed system; the MapReduce distributed programming model allows users to Develop parallel applications without the low-level details of distributed systems.

HDFS is the basis of data storage management in distributed computing, and is developed based on the requirements of streaming data mode to access and process very large files. Its characteristics provide storage for massive data without fear of failure, and bring a lot of convenience to the application processing of very large data sets. HDFS is a master/slave (Mater/Slave) architecture. There are two types of nodes in its architecture, one is NameNode, also called "metadata node"; the other is DataNode, also called "data node". The two types of nodes are the execution nodes responsible for the specific tasks of Master and Worker respectively. But due to the nature of distributed storage, HDFS cluster has a NameNode and multiple DataNodes. Metadata nodes are used to manage the namespace of the file system; data nodes are where data is actually stored in the file system.

The MapReduce parallel computing framework is a parallel program execution system. It provides a parallel processing model and process including two stages of Map and Reduce, processes data in the form of key-value pair data input, and can automatically complete data division and scheduling management. During program execution, the MapReduce parallel computing framework will be responsible for scheduling and allocating computing resources, dividing and inputting and outputting data, scheduling program execution, monitoring program execution status, and being responsible for the synchronization of each computing node during program execution and the collection of intermediate results .

Sqoop is a tool for fast batch data exchange between relational database and Hadoop platform. It can import data from a relational database into Hadoop's HDFS and Hive in batches, and can also import data from the Hadoop platform into a relational database in reverse.

Hive is a data warehouse built on Hadoop for managing structured/semi-structured data stored in HDFS. It allows directly using the SQL-like HiveQL query language as a programming interface to write data query analysis programs, and provides data extraction and conversion, storage management, and query analysis functions required by data warehouses, while HiveQL statements are converted into corresponding MapReduce in the underlying implementation program to be executed.

Contents of the invention

In order to overcome the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a Hadoop-based network security log k-means clustering analysis system and method, on the basis of rationally utilizing the traditional data warehouse that has been built, the big data The platform is integrated to establish a unified data storage and data processing architecture, which overcomes the shortcomings of traditional data warehouses, which are poor in scalability, only good at processing structured data, and unable to mine the intrinsic value of massive heterogeneous data.

In order to achieve the above object, the technical solution adopted in the present invention is: a Hadoop-based network security log k-means cluster analysis system, including a log data acquisition subsystem, a log data mixing mechanism storage management subsystem, and a log data analysis subsystem. system;

The log data acquisition subsystem collects the network security log data of all devices;

The log data mixing mechanism storage management subsystem manages and stores all log data;

The log data analysis subsystem is to quickly query and analyze all log data, and to mine and analyze the potential value of the log data.

The log data acquisition subsystem configures a Syslogd centralized log server in a Linux environment, collects recording equipment and system log data in a Syslog manner, and centrally manages the log data.

The log data mixing mechanism storage management subsystem integrates Hadoop platform and traditional data warehouse, including HDFS distributed file system module, Hadoop platform and traditional data warehouse cooperation module.

The log data query and analysis subsystem mainly uses the tool hive to perform simple statistical query and analysis on the data, writes the HiveQL query statement according to the requirements, and completes the lexical analysis, syntax analysis, compilation, Optimization and query plan generation. The generated query plan is stored in HDFS and then executed by MapReduce. For the analysis of potential information in log data, its intrinsic value can be tapped by writing a MapReduce program that implements the corresponding algorithm.

The basic file access process of the HDFS distributed file system module is:

1) The application sends the file name to the NameNode through the client program of HDFS;

2) After NomeNode receives the file name, it retrieves the data block corresponding to the file name in the HDFS directory, and then finds the DataNode address for saving the data block according to the data block information, and sends these addresses back to the client;

3) After receiving the addresses of these DataNodes, the client performs data transmission operations in parallel with these DataNodes, and at the same time submits the relevant logs of the operation results to the NameNode.

The Hadoop platform and the traditional data warehouse collaboration module configure the MySQL database as the metabase of Hive for storing the Schema table structure information of Hive, etc., and realize the transmission of data between the traditional data warehouse and the Hadoop platform through the Sqoop tool.

Using Hadoop cluster as a platform, integrate traditional data warehouse and big data platform, use MySQL database as Hive's metabase to store Hive's Schema table structure information, and use Sqoop tool to realize data exchange between traditional data warehouse and big data platform Transmission; including data source layer, data storage layer, calculation layer, data analysis layer and result display layer;

The data source layer collects the log data in all devices by configuring the Syslogd centralized log server, and then imports the log data from the traditional data warehouse to the data storage layer through the Sqoop tool;

The data storage layer adopts the hybrid storage architecture of Hadoop and traditional data warehouse collaboration, imports data into HDFS through Sqoop data transfer tool, processes metadata, and imports the processed data into corresponding tables in Hive;

In the result display layer, the user sends a request to the data analysis layer;

The data analysis layer converts the request sent by the user into a corresponding HiveQL statement, and completes the execution operation under the drive of the Hive Driver;

The calculation layer receives instructions from the Hive engine, and through the HDFS of the data storage layer, cooperates with MapReduce to realize data processing and analysis, and finally returns the result to the result display layer.

A Hadoop-based network security log k-means cluster analysis method, characterized in that, comprising the following steps:

1) Log data preprocessing, converting the Syslog_incoming_mes file of the text content of the log description information into a text vector file;

2) Based on the implementation of the k-means algorithm of MapReduce, run the k-means clustering algorithm on the text vector.

The log data preprocessing includes the following steps:

1) Remove function words, remove meaningless words in the log description text;

2) To mark the part of speech, the system uses the english-left3words-distsim.tagger tokenizer to mark the words in each log description;

3) Extract useful words. After word segmentation, the system extracts nouns (NN, NNS, NNP, NNPS), verbs (VB, VBP, VBN, VBD) and adjectives (JJ, JJR, JJS). These words have practical Meaning, can accurately express the log information;

4) Obtain the frequent dictionary, and count the frequency of all records. The high-frequency words have a representative role in the description field. The words exceeding the threshold are used as keyword elements, and the frequent dictionary is selected. Using the frequent dictionary can effectively express log information;

5) Generate a text vector file, compare the log description field with the frequent dictionary to obtain a keyspace consisting of a series of 0, 1, and multiple keyspace sets form a text vector file.

The realization of the described k-means algorithm based on MapReduce comprises the following steps:

1) Scan all the points in the original data set, and randomly select k points as the initial cluster centers;

2) Each Map node reads the local data set, uses the k-means algorithm to generate a clustering set, and finally uses several clustering sets to generate a new global clustering center in the Reduce stage, and repeats this process until the end condition is met;

3) According to the final cluster centers, all data elements are divided and clustered.

The advantage of technical solution of the present invention is mainly reflected in:

1) Provide Hadoop support on the basis of existing traditional data warehouses, establish a unified data storage and data processing architecture, make up for the shortcomings of traditional data warehouses in massive data processing and storage, and also make the original Traditional data warehouses use everything.

2) With more and more data, more cluster resources are needed to process the data, and Hadoop is an easy-to-expand system. It only needs to configure a new node to easily expand the cluster and improve its performance. Calculate ability.

3) For heterogeneous data in massive network logs, first use MapReduce to process metadata, then import the processed data into the corresponding table in Hive, and then write hiveQL statements to query and analyze the data simply according to requirements, and use MapReduce Implement the k-means algorithm to mine and analyze the data. The efficiency of query analysis is improved, and the potential value of data is also tapped.

Description of drawings

Fig. 1 is a functional block diagram of the system structure of the present invention.

FIG. 2 is an architecture diagram of a network log analysis system based on collaboration between Hadoop and traditional data warehouses of the present invention.

Fig. 3 is the research framework of the log data k-means clustering algorithm of the present invention.

Detailed ways

The technical solution of the present invention will be described in detail below in conjunction with the embodiments and the accompanying drawings, but is not limited thereto.

Referring to Fig. 1, a kind of Hadoop-based network security log k-means clustering analysis system includes a log data acquisition subsystem 11, a log data mixing mechanism storage management subsystem 12, and a log data analysis subsystem 13;

The log data acquisition subsystem 11 is to collect the network security log data of all devices;

Log data mixing mechanism storage management subsystem 12 is to manage and store all log data;

The log data analysis subsystem 13 performs fast query analysis processing on all log data, and conducts mining analysis on the potential value of the log data.

The operation process steps of each module of the system are as follows:

Step 1: Log data acquisition: Configure the Syslogd centralized log server, use UDP as the transmission protocol, and send the log management configuration of all security devices to the log server installed with the Syslog software system through the destination port. The Syslog log server automatically receives the log data and sends it. write to the log file;

Step 2: Use Sqoop to import the log information table syslog_incoming in MySQL to HDFS, using the command: sqoop import --connect jdbc:mysql://219.245.31.39:3306/syslog --usernamesqoop --password sqoop --table syslog_incoming -m1

Sqoop imports a table from MySQL through a MapReduce job. This job extracts a row of records from the table, and then writes the records to HDFS. The namenode in the cluster is responsible for storing the data and telling the client the storage location information. After the location information, the client starts to write data. When writing data, the data is divided into blocks and stored in multiple copies, which are placed on different datanode nodes. The client first writes the data to the first node, and on the first node While receiving data, it pushes the received data to the second node, the second node to the third node, and so on;

Step 3: Write a MapReduce program to extract useful information from the original log data imported into HDFS;

Step 4: Use Sqoop to generate a hive table according to the table in the extracted relational data source, use the command to directly generate the definition of the corresponding hive table, and then load the data stored in HDFS:

sqoop create-hive-table --connect jdbc:mysql://219.245.31.39:3306/syslog --table syslog_incoming --fields-terminated-by ','

Start hive and load data:

load data inpath “syslog_incoming” into table syslog_incoming;

Step 5: According to business requirements, write corresponding hiveQL statements or MapReduce programs to perform statistical analysis on the log data; the specific steps of the statistical analysis are: partition the data table according to business requirements, and the partition is PARTITIONED when creating the table BY clause defined. Therefore, according to the requirements, the table records are defined to be composed of grades and time (year, quarter, month) partitions. The following example defines the table records to be composed of grade partitions:

hive >create table syslog_incoming_priority (facility varchar, datadata, host varchar)

>partitioned by (priority varchar)

>row format delimited

>fields terminated by '\t'

>stored as textfile;

After defining the table structure, load the data into the partition table:

hive >insert into table syslog_incoming_priority

>partitioned (priority)

>select facility, data, host, priority

>from syslog_incoming;

At the file system level, partitions are just nested subdirectories under the table directory. At this time, there are multiple hierarchical partitions in the table directory structure, and data files are stored in the underlying directory. Finally, according to the requirements, write the query statement hiveQL, and then the cluster will Query statements are converted into MapReduce tasks for execution.

Step 6: Import the query analysis results into MySQL through Sqoop, and the front-end display interface shows the user through charts.

Referring to Fig. 3, a Hadoop-based network security log k-means clustering analysis method includes the following steps:

1) Log data preprocessing, converting the Syslog_incoming_mes file of the text content of the log description information into a text vector file;

2) Based on the implementation of the k-means algorithm of MapReduce, run the k-means clustering algorithm on the text vector.

The log data preprocessing includes the following steps:

1) Remove function words, remove meaningless words in the log description text;

2) To mark the part of speech, the system uses the english-left3words-distsim.tagger tokenizer to mark the words in each log description;

3) Extract useful words. After word segmentation, the system extracts nouns (NN, NNS, NNP, NNPS), verbs (VB, VBP, VBN, VBD) and adjectives (JJ, JJR, JJS). These words have practical meanings and can accurately express log information;

4) Obtain a frequent dictionary, count the frequency of all records, high-frequency words have a representative role in the description field, and words exceeding the threshold are used as keyword elements. Select frequent dictionaries, and use frequent dictionaries to effectively express log information;

5) Generate a text vector file, compare the log description field with the frequent dictionary to obtain a keyspace consisting of a series of 0, 1, and multiple keyspace sets form a text vector file.

The realization of the described k-means algorithm based on MapReduce comprises the following steps:

To select the initial cluster center, the data structure of the cluster is given first. This class stores the basic information of a cluster such as the cluster id, the center coordinates and the number of points belonging to the cluster. Its type is defined as follows:

public class Cluster implements writable{

private int clusterID; //cluster id

private long numOfPoints; //The number of points belonging to the cluster

private Instance center; //cluster center point information

}

Then randomly select k points as the initial cluster centers, and the extraction process: initialize the cluster center set to be empty, and then scan the entire data set. If the size of the current cluster center set is less than k, add the scanned point to the cluster center, otherwise replace a point in the cluster center set with a probability of 1/(1+k). Through this step, we write the generated cluster center information into the Cluster-0 directory, and the files in this directory are added to the distributed shared cache of MapReduce as global shared data as the global shared information in the next iteration.

Iteratively computes cluster centers. This stage needs to perform multiple iterations. Before starting, each map node first needs to read in the cluster information generated in the previous round of iterations in the setup() method, including the following steps:

1) Read the initial cluster center: read the initial cluster center data stored in the shared cache shared by all nodes;

2) Implementation of the map method: The map method needs to find the nearest cluster center for each incoming data point, and use the id in the cluster as the key, and emit the data point as a value, indicating that the data point belongs to the id. the cluster;

3) Combiner implementation: In order to reduce network data transmission overhead, we use combiner on the map side to merge the results generated by the map side, which not only reduces the data transmission overhead from the map to the reduce side, but also reduces the calculation overhead on the reduce side , the type of the key and value output by the Combiner must be the same as the type of the key and value output by the map. In the reduce program, we calculate the temporary centers of these points based on the information of all points belonging to the same cluster. Here, it is realized by a simple method of averaging, that is, adding and dividing all points in the cluster by the points contained in the cluster at this time. the number of all points;

4) Implementation of Reducer: The Reduce stage is almost the same as that of the Combiner, which further merges and outputs the output of the Combiner.

The processing of the step of calculating the cluster center is repeated until the obtained cluster center does not change any more.

Divide the data according to the final cluster centers. After obtaining the final cluster center, scan all data sets according to the obtained cluster center, and divide each data point into the nearest cluster center.

Example:

First, build a Hadoop distributed cluster environment, including 5 PCs. One master server and the remaining four are slave servers. Configure Hadoop on each machine, and then install and configure Sqoop, hive, and MySQL on the namenode. In this embodiment, the log records of all security devices in Shaanxi Li'an Electric Supermarket are used, and the file size is 16G. The log is regularly updated every day according to the requirements, and the query results are counted in the update business.

This method can realize fast statistical query through hive. Its advantages are: low learning cost, simple MapReduce statistics can be quickly realized through SQL-like statements, and no special MapReduce application needs to be developed, which is very suitable for statistical analysis of data warehouses. Using partitions can speed up the query speed of data shards and improve query efficiency. Realize the k-means algorithm through MapReduce, and evaluate the security level of the output results of the k-means clustering algorithm, and make timely prompts for those with a large proportion of alarms and danger levels in the same IP, so that the potential value of log data can be obtained. dig.

Claims (4)

1. A Hadoop-based network security log k-means cluster analysis system, characterized in that, includes log data acquisition subsystem (11), log data mixing mechanism storage management subsystem (12), log data analysis subsystem (13); The log data acquisition subsystem (11) is to collect the network security log data of all devices; The log data mixing mechanism storage management subsystem (12) manages and stores all log data; The log data analysis subsystem (13) performs fast query analysis processing on all log data, and mines and analyzes the potential value of the log data; The log data acquisition subsystem (11) configures a Syslogd centralized log server in a Linux environment, adopts Syslog mode to collect recording equipment and system log data, and centrally manages the log data; The basic file access process of the HDFS distributed file system module is: 1) The application sends the file name to the NameNode through the client program of HDFS; 2) After NomeNode receives the file name, it retrieves the data block corresponding to the file name in the HDFS directory, and then finds the DataNode address for saving the data block according to the data block information, and sends these addresses back to the client; 3) After receiving the addresses of these DataNodes, the client performs data transmission operations in parallel with these DataNodes, and at the same time submits the relevant logs of the operation results to the NameNode; Described Hadoop platform and traditional data warehouse cooperation module, configure MySQL database as the metabase of Hive, be used to store the Schema table structure information of Hive, realize the transmission of data between traditional data warehouse and Hadoop platform by Sqoop tool; The log data analysis subsystem (13) mainly uses the tool hive to perform simple statistical query and analysis on the data, writes HiveQL query statements according to requirements, and completes the lexical analysis, syntax analysis, compilation, Optimization and query plan generation. The generated query plan is stored in HDFS and then executed by MapReduce. For the analysis of potential information in log data, its intrinsic value can be tapped by writing a MapReduce program that implements the corresponding algorithm. 2. A Hadoop-based network security log k-means cluster analysis system according to claim 1, characterized in that the log data mixing mechanism storage management subsystem (12) integrates Hadoop platform and traditional data warehouse , including HDFS distributed file system module, Hadoop platform and traditional data warehouse collaboration module. 3. A network security log k-means cluster analysis system based on Hadoop, characterized in that, using Hadoop cluster as platform, integrating traditional data warehouse and big data platform, using MySQL database as the metabase of Hive for storing Hive Schema table structure information, use Sqoop tool to realize data transmission between traditional data warehouse and big data platform; including data source layer (21), data storage layer (22), computing layer (23), data analysis layer ( 24) and result display layer (25); Described data source layer (21), collects the log data in all equipments by disposing Syslogd centralized log server, then by Sqoop tool, log data is imported into data storage layer from traditional data warehouse; The data storage layer (22) adopts a hybrid storage architecture in which Hadoop cooperates with traditional data warehouses, imports data into HDFS through Sqoop data transfer tools, processes metadata, and imports the processed data into corresponding tables in Hive inside; In the result display layer (25), the user sends a request to the data analysis layer; The data analysis layer (24) converts the request sent by the user into a corresponding HiveQL statement, and completes the execution operation under the drive of the Hive Driver; The computing layer (23) receives instructions from the Hive engine, and cooperates with MapReduce to implement data processing and analysis through the HDFS of the data storage layer, and finally returns the results to the result display layer. 4. Utilize the method for systematic clustering analysis described in claim 1, it is characterized in that, comprise the following steps: 1) Log data preprocessing, converting the Syslog_incoming_mes file of the text content of the log description information into a text vector file; 2) Implementation of the k-means algorithm based on MapReduce, running the k-means clustering algorithm on the text vector; The log data preprocessing includes the following steps: 1) Remove function words, remove meaningless words in the log description text; 2) To mark the part of speech, the system uses the english-left3words-distsim.tagger tokenizer to mark the words in each log description; 3) Extract useful words. After word segmentation, the system extracts nouns NN, NNS, NNP, NNPS, verbs VB, VBP, VBN, VBD and adjectives JJ, JJR, JJS. These words have actual meanings and can be accurately expressed log information; 4) Obtain the frequent dictionary, and count the frequency of all records. The high-frequency words have a representative role in the description field. The words exceeding the threshold are used as keyword elements, and the frequent dictionary is selected. Using the frequent dictionary can effectively express log information; 5) Generate a text vector file, compare the log description field with the frequent dictionary to obtain a keyspace consisting of a series of 0, 1, and multiple keyspace sets form a text vector file; The realization of the described k-means algorithm based on MapReduce comprises the following steps: 1) Scan all the points in the original data set, and randomly select k points as the initial cluster centers; 2) Each Map node reads the local data set, uses the k-means algorithm to generate a clustering set, and finally uses several clustering sets to generate a new global clustering center in the Reduce stage, and repeats this process until the end condition is met; 3) According to the final cluster centers, all data elements are divided and clustered.