CN117591477A - A log aggregation query method for massive data - Google Patents

Abstract

The invention discloses a log aggregation query method of mass data, which comprises the following steps: and (3) log acquisition: collecting different types of log data from a server; processing the data fields in the log by adopting dynamic and static combined fields to obtain processed data; analyzing the processed data by adopting a cold-hot separation model to obtain cold and hot data, and shunting the cold and hot data to warehouse; the method comprises the steps of firstly carrying out log inquiry through a hot database, and then carrying out log inquiry through a cold database. The invention adopts dynamic and static combined field processing and cold and hot separation models, and then shunts the log data, so that the user directly inquires from the hot data during inquiry, thereby greatly reducing the inquiry range, reducing the magnitude and improving the overall inquiry speed.

Description

A log aggregation query method for massive data

Technical field

The invention relates to the field of log processing, and in particular to a log aggregation query method for massive data.

Background technique

With the development of the information age, a large amount of log data will be monitored in real time, and the analyzed data will be statistically analyzed to discover and solve production problems through different dimensions. How to efficiently and quickly conduct analysis based on the time range, be able to flexibly switch dimensions for analysis, and obtain corresponding data very quickly, these issues have gradually become a point of concern in the market. The more common logs currently on the market:

1. Security log: records application operations, errors, exceptions, performance indicators and other information, used to track and troubleshoot application problems.

2. Application log: records application operations, errors, exceptions, performance indicators and other information to track and troubleshoot application problems.

3. System log: records operating system events, errors, warnings and other information, including login information, service start and stop, resource utilization, etc.

4. Database log: records database operations, queries, transactions and other information, helping to track and debug database-related issues.

5. Server log: records performance indicators, errors, events and other information of server hardware and operating system, including CPU, memory, disk utilization, network traffic, etc.

6. User log: records user’s operating activities, login records, access rights and other information in the system.

7. Debug logs: Logs recorded by developers during the development and debugging process, used to track code execution paths, view variable values, etc.

8. Business logs: logs that record specific business operations, such as transaction logs, order logs, log queries, etc., used to track business processes and subsequent analysis.

Log aggregation, as a management strategy to improve efficiency and reduce noise, has become an important practice in modern IT operations. The following is some background status of log aggregation:

1. The number of log aggregation is increasing: As the scale and complexity of IT systems continue to increase, the number of logs generated by monitoring systems is also increasing. A large number of duplicate, redundant and irrelevant logs have put great pressure on operation and maintenance personnel, so log aggregation has become a necessary means.

2. Need accurate contextual information: It is often difficult to obtain the full picture and contextual information of the problem through only a single log. Log aggregation can integrate related logs to provide more accurate contextual information and help the operation and maintenance team better understand and analyze problems.

3. Improve fault handling efficiency: Log aggregation can help the operation and maintenance team locate and handle faults more quickly and accurately. By aggregating and filtering logs, operation and maintenance personnel can solve problems in a more targeted manner and avoid getting lost in massive logs.

All in all, log aggregation is receiving widespread application and attention as a strategy to deal with log flooding and improve operation and maintenance efficiency.

At present, the technology used in log aggregation is mainly offline processing. The data is calculated offline through mapReduce at fixed minutes, hours, and days. The aggregated data is stored in a table, and the fields in the table contain all fields.

The disadvantages of offline processing are as follows:

1. The delayed response time of offline processing is long, and the data must be stored in the database first and then queried from the database, which increases the pressure on the database and lowers the processing performance.

2. The original data fields are redundant to the aggregation fields, conditional query, too many fields, and the query acquisition speed is slow.

3. The log aggregation range changes and the data response is slow. For example: If the user wants to change the aggregation time range from 5 minutes to 1 hour, he must re-run a task to calculate the data from a certain point in time to the present. This often takes a long time to process, recovery is slow, and cannot be done in a timely manner. Respond to user needs. Either maintain a separate hour table, or maintain a day table, but this processing takes up server resources.

Contents of the invention

In order to solve the above problems, the present invention proposes a log aggregation query method for massive data, which greatly reduces the magnitude of log processing, speeds up query speed, and shortens response time.

The method includes the following steps:

S1. Log collection: collect different types of log data from the server;

S2. Log processing: The data fields in the log are processed using a combination of dynamic and static fields to obtain processed data;

S3. Log storage: After analyzing the processed data using the hot and cold separation model, cold and hot data are obtained, and the cold and hot data are divided into the database;

S4. Log query: First perform log query through the hot database, and then perform log query through the cold database.

The beneficial effects provided by the present invention are: due to the adoption of dynamic and static combined field processing and hot and cold separation models, the log data is shunted. When querying, the user directly queries from the hot data, which greatly reduces the scope of the query, reduces the magnitude, and improves the overall query speed.

Description of drawings

Figure 1 is a schematic flow diagram of the method of the present invention;

Figure 2 is a flow chart of real-time data processing according to the present invention;

Figure 3 is a schematic diagram of the user query process;

Figure 4 is a schematic diagram of the data dimensionality reduction process of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

Please refer to Figure 1, which is a schematic flow chart of the method of the present invention;

The present invention provides a log aggregation query method for massive data, including:

S1. Log collection: collect different types of log data from the server;

It should be noted that different types of logs include: security logs, application logs, system logs, database logs, server logs, user logs, debugging logs, business logs, etc.

S2. Log processing: The data fields in the log are processed using a combination of dynamic and static fields to obtain processed data;

It should be noted that the combination of dynamic and static fields specifically refers to dividing data fields into fixed fields and dynamic column fields.

The fixed fields are specifically fields that can be aggregated. Specifically, fixed fields are usually the most basic attributes of logs. These attributes are changeable aggregate data. For example, fixed fields are: latest occurrence time, latest occurrence time, number of aggregated entries, highest level, whether to block, etc. Fixed fields are used for calculation and statistics of data on the results page.

As an example, for example, there is the following batch of log data:

1. IP: 172.10.12.3 Traffic: 3512MB Level: Medium Generation time: 2023-09-09 08:08:00 Number of attacks on external networks: 1 Blocking: Yes;

2. IP: 172.10.12.3 Traffic: 1024MB Level: Low Generation time: 2023-11-10 20:08:00 Number of attacks on external networks: 2 Blocking: Yes;

3. IP: 172.10.12.3 Traffic: 7789MB Level: Medium Generation time: 2023-12-10 16:08:00 Number of attacks on external networks: 3 Blocking: No;

Then the fixed field can be defined as:

Minimum flow, maximum flow, highest level, earliest occurrence time, latest occurrence time, total number of external network attacks, total number of blocked items, and number of unblocked items;

It should be noted that fixed fields can be customized according to the properties of the field. According to the definition of the fixed field above, the final display results of the fixed field are: 1024MB, 7789MB, medium, 2023-09-09 08:08:00, 2023-12-10 16:08:00, 6, 2, 1.

The dynamic column fields are user-defined feature fields.

It should be noted that the dynamic column fields are the fields that users need to pay attention to.

For example, the user-defined dynamic column fields are: source address, source port, destination address, destination port, and geographical location.

As an example, the actual characteristic values of several logs of the above-mentioned customized dynamic column fields are:

172.16.1.11_8080_172.16.12.3_8081_Wuhan;

172.16.1.11_8080_172.16.12.52_8082_Yellowstone;

222.16.1.12_8097_172.16.12.52_8082_Hangzhou.

S3. Log storage: After analyzing the processed data using the hot and cold separation model, cold and hot data are obtained, and the cold and hot data are divided into the database;

It should be noted that in step S3, the hot and cold separation model specifically refers to:

The processed data is represented according to dynamic column fields;

Convert dynamic column fields into corresponding unincode values;

The cluster analysis algorithm is used to compare the new batch of log data to be processed in a certain data category. If the unincode value of the new batch of log data to be processed is different from the unincode value of a certain data category, If the difference is less than or equal to the preset value, the new batch of log data to be processed will be labeled as hot data, and the log data will be hot data; otherwise, the data will be cold data.

As an embodiment, for the dynamic column field in the above embodiment, convert it into an unincode value as follows:

Field source address_field source port_field destination address_field destination port_field geographical location:

20320_22909_65111_19990_123331;

20320_22909_65292_19220_256431;

52789_22898_65292_19220_751621;

The above unincode values are analyzed and compared through the clustering algorithm. First, the matching rules are set. The matching rules adopt the method of exact matching + fuzzy fields.

Exact match: The source address and source port are required to be exactly the same;

Fuzzy fields: The destination address, destination port, and geographical location are required to be fuzzy and the same.

Regarding fuzzy similarity, for example, the destination ports are fuzzy and the same means that the destination port contains 80; and the geographical location is fuzzy and the same means that the geographical locations are both Hubei.

Regarding the cluster analysis algorithm, the Kmeans clustering algorithm is used in this invention:

1. For the first random log, k pieces of data are randomly selected as the initial centroid k (features)

2. Calculate the distance from other log data to the k centroids (in this application, the above difference is within the preset value);

If a certain log data is closer to the nth centroid, then the point is classified as n, and it is labeled as n. The next batch of logs is entered again, the data is calculated, and the average value is calculated as the classification n.

3. Data with the same classification n will be quickly compared. Data that meets the same characteristics (the aforementioned matching rules) will be classified into the same aggregated data. Characteristics that do not match will generate a new aggregated data.

The purpose of the above approach is to divide massive data into multiple classification intervals, and then compare the data in the same category interval. For example, for geographical location comparison, you can refer to the cluster analysis algorithm to divide cities into different categories.

Take the data of the above dynamic column field as an example:

172.16.11.1_8080_Wuhan and 172.16.11_8080_Xiaogan are similar data;

The aggregation feature (matching rule) is 172.16.11.1_8080_Hubei, which is hot data;

The next batch of data comes in and collides with the same aggregated feature 172.16.11.1_8080_hubei, and the updated data is hot data.

If the aggregation feature data has not been collided, it is cold data; cold data will have no change in the data that conforms to the aggregation feature, the maximum value and the minimum value will not change, and the time is also determined.

The above is about the distinction between cold and hot data. Hot and cold data are split into different databases, cold databases and hot databases.

As another embodiment, in this application, the cold and hot data also change, hot data can be changed into cold data, and cold data can also be changed into hot data.

Among them, the specific strategy for changing hot data into cold data is:

Set a scheduled task to scan hot data. If the latest log generation time is greater than the current t days, the hot data of the corresponding feature will be updated to cold data.

As an example, t can be set to 30 days, half a year, or 1 year, etc.; the frequency of the scheduled task can be once a day, or once a few days, etc.;

The specific strategy for turning cold data into hot data is:

When a certain log data to be processed is compared in a certain data classification and fails to match in the hot data but is found in the cold data, the log data to be processed is updated to hot data.

S4. Log query: First perform log query through the hot database, and then perform log query through the cold database.

As an embodiment, the present invention adopts split inflow into the database, with cold data exclusively entering the cold database and hot data exclusively entering the hot database. At the same time, the cold and hot databases are updated frequently according to the set strategy.

When performing log query, collision is performed through the hot database first, which can greatly reduce query time and improve query efficiency.

Please refer to Figure 2, which is a flow chart of real-time data processing according to the present invention;

After collecting and processing log data from the server, the processed data first still enters the original database, and at the same time another branch is introduced for cluster analysis and processing to obtain cold and hot databases.

Please refer to Figure 3, which is a schematic diagram of the user query process.

When users query logs, they first specify the end of the time range, and then perform data collision from the hot database to obtain fields that meet the conditions.

Finally, according to the fields that meet the conditions, the more detailed content of the log is retrieved from the original database.

Please refer to Figure 4, which is a schematic diagram of the data dimensionality reduction process of the present invention.

The original data is 1 billion pieces. After processing the fixed fields and dynamic column fields, detailed data is obtained. Analyze the detailed data using the clustering algorithm of associated eigenvalues to obtain hot data and cold data.

In terms of technical implementation, this invention relies on: Spark+Kafka+ElasticSearch+HDFS+Redis cluster+big data cluster;

Database used: ES\openTsdb;

Implementation method: Build servers and collection servers, build a big data cluster environment, simulate sending security logs and program logs, observe data aggregation, and use Kafka-Manager and java jvm monitoring tools to monitor log storage. In terms of performance, it can process 20,000 pieces of aggregated data per second on average, and it takes nearly 5 seconds to query tens of millions of data.

Finally, the beneficial effects of the present invention are: due to the adoption of dynamic and static combined field processing and hot and cold separation models, the log data is shunted. When querying, the user directly queries from the hot data, greatly narrowing the query scope, reducing the magnitude, and improving Overall query speed.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (7)

1. A log aggregation query method for massive data, which is characterized by: including: S1. Log collection: collect different types of log data from the server; S2. Log processing: The data fields in the log are processed using a combination of dynamic and static fields to obtain processed data; S3. Log storage: After analyzing the processed data using the hot and cold separation model, cold and hot data are obtained, and the cold and hot data are divided into the database; S4. Log query: First perform log query through the hot database, and then perform log query through the cold database. 2. A log aggregation query method for massive data as claimed in claim 1, characterized in that: the combined dynamic and static fields specifically refer to dividing the data fields into fixed fields and dynamic column fields. 3. A log aggregation query method for massive data according to claim 2, characterized in that: the fixed fields are specifically fields that can be aggregated. 4. A log aggregation query method for massive data according to claim 2, characterized in that: the dynamic column field is a user-defined feature field. 5. A log aggregation query method for massive data as claimed in claim 4, characterized in that: in step S3, the hot and cold separation model specifically refers to: The processed data is represented according to dynamic column fields; Convert dynamic column fields into corresponding unincode values; The cluster analysis algorithm is used to compare the new batch of log data to be processed in a certain data category. If the unincode value of the new batch of log data to be processed is different from the unincode value of a certain data category, If the difference is less than or equal to the preset value, the new batch of log data to be processed will be labeled as hot data, and the log data will be hot data; otherwise, the data will be cold data. 6. A log aggregation query method for massive data as claimed in claim 5, characterized by: setting a scheduled task, scanning hot data, and updating the log to cold data if the latest generation time is greater than the current t days. 7. A log aggregation query method for massive data as claimed in claim 6, characterized in that: when a certain log data to be processed is compared in a certain data classification, it fails to match in the hot data. But if it is found in cold data, the log data to be processed is updated to hot data.