WO2022047659A1 - Multi-source heterogeneous log analysis method - Google Patents

Abstract

A multi-source heterogeneous log analysis method, belonging to the field of log data processing, to solve the problem of log analysis, comprising step 1: determining the size of a time window according to a response time required by an information system; step 2: using an SGSE algorithm to process log data within each time window into a sample which can be called by an ECC log analysis algorithm; step 3: training and using an ECC log analysis model to analyze whether it is normal within the time window; and step 4: presenting a log analysis result. The present invention has the effect of performing abnormality analysis on logs.

Description

Multi-source heterogeneous log analysis method technical field

The invention belongs to the field of log data processing, and relates to a multi-source heterogeneous log analysis method and system.

Background technique

With the development of Internet technology, the number of logs generated by each device in the information system is also increasing day by day. The analysis of logs generated by different devices and with different data characteristics is an important part of the operation and maintenance work. By analyzing the multi-source and heterogeneous log data by automated means, you can know whether the operating status of the information system is abnormal or normal in time, so as to ensure the safe and stable operation of the information system, thereby reducing the operation and maintenance cost of the enterprise.

In the current technical methods of multi-source heterogeneous log analysis, methods such as single analysis re-aggregation and association analysis are used. In the method of single analysis and aggregation, the log generated by a single device in the information system is first analyzed, and the running status of each device is analyzed, and then the existence of the entire information system is judged according to the pre-set rules according to the status of each device. abnormal situation. However, this method does not combine and analyze the logs in different devices, but judges the states of different devices separately and then analyzes them, so that the relationship between the logs of different devices cannot be mined. In the method of association analysis, characteristic events are first generated according to the content of each field in the log, and events generated by different devices in a time window are clustered and then compared for similarity, and similar events are eliminated. Then the similar events of different devices are combined, and finally statistical reports of various events are generated. However, in this method, for the purpose of generating statistical reports of various events, the relationship between various events cannot be deeply excavated and presented to the user directly, and only the clustering algorithm cannot accurately classify each event.

In the current technical methods of multi-source heterogeneous log analysis, methods such as single analysis and re-aggregation ^[1] and association analysis ^[2] are used. In the method of single analysis and aggregation, the log generated by a single device in the information system is first analyzed, the running status of each device is analyzed, and then whether the entire information system exists or not is determined according to the state of each device according to the rules set in advance. abnormal situation. In the method of correlation analysis, characteristic events are first generated according to the content of each field in the log, and events generated by different devices in a time window are compared for similarity, and similar events are eliminated. Then the similar events of different devices are combined, and finally statistical reports of various events are generated.

Based on the single analysis and aggregation method, the logs in different devices are not combined and analyzed, but the status of different devices is judged separately and then analyzed, and the relationship between the logs of different devices cannot be mined. In the method of association analysis, for the purpose of generating statistical reports of various events, it is impossible to deeply mine the relationship between various events and present them directly to the user.

SUMMARY OF THE INVENTION

In order to solve the problem of log analysis, the present invention proposes the following technical solution: a multi-source heterogeneous log analysis method, comprising the following steps:

Step 1: Determine the size of the time window according to the response time required by the information system;

Step 2: Use the SGSE algorithm to process the log data in each time window into a sample that can be called by the ECC log analysis algorithm;

Step 3: Train and use the ECC log analysis model to analyze whether the time window is normal;

Step 4: Present the log analysis results.

Further, the steps of model training are as follows:

Step 1: Count the number of logs of multi-source heterogeneous log data in normal and abnormal time windows to generate a log number status subsequence, and count the number of each log type generated on each device within the time window to generate a user behavior status subsequence , count the number of occurrences of types in some important fields of each device in the time window to generate field status subsequences;

Step 2: Generate (n+m+j) samples from n features in the log quantity state subsequence, m features in the user behavior state subsequence, and j features in the field state subsequence under each time window data set;

Step 3: Take a certain feature in the log quantity status subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the sample data sets in other normal and abnormal time windows in pairs. The difference value, the calculation expression is:

v1=1-v2 (3)

f(tableα)=v1*M′ _tableα +v2*M″ _tableα +bias (4)

tableα represents the sample corresponding to a feature in the log quantity state subsequence as a label,

tableα' represents the sample corresponding to a certain feature in the log quantity status subsequence as a label in the normal time window,

"tableα" represents the sample corresponding to a certain feature in the log quantity status subsequence under the abnormal time window as the label,

M _tableα ′ is the difference between a sample corresponding to a certain feature in the log quantity state subsequence as a label and a sample corresponding to a certain feature in the log quantity state subsequence under a normal time window as a label. mean squared error,

M _tableα ″ is the difference between the sample corresponding to a certain feature in the log quantity state subsequence as a label and the sample corresponding to a certain feature in the log quantity state subsequence under the abnormal time window as a label mean squared error,

bias is paranoia,

v1 and v2 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively, and v1=v2 during training;

f(tableα) is the difference value calculated by a feature in the log quantity state subsequence in the trained time window as a label;

Step 4: Calculate the difference value between each normal time window and other normal and abnormal time windows through equations (1)-(4) and save it as a set U ₁ , and use equations (1)-(4) and The difference value is calculated in the normal and abnormal time windows and saved as P ₁ , and the confidence interval σ(α) of the log quantity state subsequence under the normal time window is obtained as

σ(α)=[min(U1),max(U1)]∩[min(P1),max(P1)]

Step 5: Calculate the sample data set with a certain feature in the user behavior state subsequence in each normal and abnormal time window as the label according to the ECC expression and the sample data sets in other normal and abnormal time windows. The difference value, the calculation expression is:

v3=1-v4 (7)

f(tableβ)=v3*M′ _tableβ +v4*M″ _tableβ +bias (8)

tableβ represents the sample corresponding to a feature in the user behavior state subsequence as a label,

tableβ' represents the sample corresponding to a certain feature in the subsequence of user behavior status under the normal time window as a label,

"tableβ" represents the sample corresponding to a certain feature in the subsequence of user behavior status under the abnormal time window as a label,

M _tableβ ′ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under a normal time window as a label. mean squared error,

M _tableβ ″ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under an abnormal time window as a label mean squared error,

bias is paranoia,

v3 and v4 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v3=v4,

f(tableβ) is the difference value calculated by a feature in the user behavior state subsequence in the trained time window as a label;

Step 6: Calculate the difference between each normal time window and the normal and abnormal time windows through equations (5)-(8) and save it as a set U ₂ , and compare the abnormal time window with the normal and abnormal time windows through equations (5)-(8). , the difference value is calculated in the abnormal time window and saved as P ₂ , and the confidence interval σ(β) of the field state subsequence under the normal time window is obtained as

σ(β)=[min(U2),max(U2)]∩[min(P2),max(P2)]

Step 7: Use a certain feature in the field state subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the difference value in pairs with the sample sets in other normal and abnormal time windows. , the calculation expression is:

v5=1-v6 (11)

f(tableγ)=v5*M′ _tableγ +v6*M″ _tableγ +bias (12)

tableγ represents the sample corresponding to a feature in the field state subsequence as a label,

tableγ′ represents the sample corresponding to a feature in the field state subsequence in the normal time window as a label,

"tableγ" represents the sample corresponding to a feature in the field state subsequence under the abnormal time window as a label,

M _tableγ ′ is the mean square between a sample corresponding to a feature in the field state subsequence as a label and a sample corresponding to a feature in the field state subsequence as a label in a normal time window error,

M _tableγ ″ is the mean square between the sample corresponding to a certain feature in the field state subsequence as the label and the sample corresponding to a certain feature in the field state subsequence under the abnormal time window as the label error,

bias is paranoia,

v5 and v6 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v5=v6,

f(tableγ) is the difference value calculated by a feature in the field state subsequence in the trained time window as a label;

Step 8: Calculate the difference between each normal time window and the normal and abnormal time windows through formulas (9)-(12) and save it as a set U ₃ , and use formulas (9)-(12) to calculate the difference between the abnormal time window and the normal time window. , the difference value is calculated in the abnormal time window and saved as P ₃ , and the confidence interval σ(γ) of the log quantity state subsequence under the normal time window is obtained as

σ(γ)=[min(U3),max(U3)]∩[min(P3),max(P3)].

Further, the method for log analysis includes the following steps:

Step 1: when a detected time window is analyzed, randomly select a feature of each subsequence to form three labels and form three samples to represent the tested time window;

Step 2: Set the initial values v1=v2, v3=v4, v5=v6, and calculate the difference values f(tableα) and f(tableβ) of the three samples under the detection time window through formulas (1)-(12). ), whether f(tableγ) is within the corresponding confidence interval σ(α), σ(β), σ(γ);

Step 3: If the three samples are all within the confidence interval, then constrain v1, v2, v3, v4, v5, v6, and the constraint formula is

v1=a1*v1(0≤a＜1)

v3=a2*v3(0≤a2＜1)

v5=a3*v5(0≤a3＜1)

v1, v3, and v5 are the variation coefficients of the mean square error of the normal time window, respectively. If v1, v3, and v5 are reduced, the influence of the normal time window on the difference value is reduced, and the influence of the abnormal time window on the difference value is increased. According to the new v1 , v2, v3, v4, v5, v6 and the three samples re-pass formulas (1)-(12) to calculate whether the difference value of the three samples under the tested time window is within the corresponding confidence interval σ(α) , σ(β), σ(γ), and determine the number of repeated constraints according to the requirements of the false negative rate;

Step 4: If the difference value of the three samples under the tested time window is still within the confidence interval after the repeated constraint is over, it is considered that the information system is normal in the tested time window, otherwise it is considered abnormal.

Beneficial effects: The present invention proposes the SGSE algorithm to process multi-source heterogeneous logs, and process them into multi-dimensional samples that can represent the state of the information system for analysis by the algorithm. The invention proposes an ECC algorithm for multi-source heterogeneous logs that can analyze multi-dimensional samples, and analyzes the information system operating state of the time window according to the multi-source heterogeneous logs under the time window.

Description of drawings

Figure 1: Multi-source heterogeneous log analysis flowchart.

Figure 2: Sample generation graph for the window state under test.

detailed description

The present invention proposes an SGSE (State Generation Sequential Extraction) algorithm for processing the multi-source heterogeneous log data in the information system. A new ECC (Error Coefficient Constraint, Error Coefficient Constraint) algorithm is used to determine the operating state of the information system. Referring to FIG. 1, the multi-source heterogeneous log analysis method of the present invention includes the following steps:

Step 1: Determine the size of the time window according to the response time required by the information system.

Step 2: Use the SGSE algorithm to process the logs in each time window, and process the log data in each time window into samples.

Step 3: Train and use the ECC log analysis model to analyze the time window to be analyzed.

Step 4: Present the log analysis results.

In a preferred solution, the steps of training the ECC log analysis model are as follows:

Step 1: Count the number of logs of multi-source heterogeneous log data in normal and abnormal time windows to generate a log number status subsequence, and count the number of each log type generated on each device within the time window to generate a user behavior status subsequence , count the number of occurrences of types in some important fields of each device in the time window to generate field status subsequences;

Step 2: Generate (n+m+j) samples from n features in the log quantity state subsequence, m features in the user behavior state subsequence, and j features in the field state subsequence under each time window data set;

Step 3: Take a certain feature in the log quantity status subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the sample data sets in other normal and abnormal time windows in pairs. The difference value, the calculation expression is:

v1=1-v2 (3)

f(tableα)=v1*M′ _tableα +v2*M″ _tableα +bias (4)

tableα represents the sample corresponding to a feature in the log quantity state subsequence as a label,

tableα' represents the sample corresponding to a certain feature in the log quantity status subsequence as a label in the normal time window,

"tableα" represents the sample corresponding to a certain feature in the log quantity status subsequence under the abnormal time window as the label,

M _tableα ′ is the difference between a sample corresponding to a certain feature in the log quantity state subsequence as a label and a sample corresponding to a certain feature in the log quantity state subsequence under a normal time window as a label. mean squared error,

M _tableα ″ is the difference between the sample corresponding to a certain feature in the log quantity state subsequence as a label and the sample corresponding to a certain feature in the log quantity state subsequence under the abnormal time window as a label mean squared error,

bias is paranoia,

v1 and v2 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively, and v1=v2 during training;

f(tableα) is the difference value calculated by a feature in the log quantity state subsequence in the trained time window as a label;

Step 4: Calculate the difference value between each normal time window and other normal and abnormal time windows through equations (1)-(4) and save it as a set U ₁ , and use equations (1)-(4) and The difference value is calculated in the normal and abnormal time windows and saved as P ₁ , and the confidence interval σ(α) of the log quantity state subsequence under the normal time window is obtained as

σ(α)=[min(U1),max(U1)]∩[min(P1),max(P1)]

Step 5: Calculate the sample data set with a certain feature in the user behavior state subsequence in each normal and abnormal time window as the label according to the ECC expression and the sample data sets in other normal and abnormal time windows. The difference value, the calculation expression is:

v3=1-v4 (7)

f(tableβ)=v3*M′ _tableβ +v4*M″ _abkeβ +bias (8)

tableβ represents the sample corresponding to a feature in the user behavior state subsequence as a label,

tableβ' represents the sample corresponding to a certain feature in the subsequence of user behavior status under the normal time window as a label,

"tableβ" represents the sample corresponding to a certain feature in the subsequence of user behavior status under the abnormal time window as a label,

M _tableβ ′ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under a normal time window as a label. mean squared error,

M _tableβ ″ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under an abnormal time window as a label mean squared error,

bias is paranoia,

v3 and v4 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v3=v4,

f(tableβ) is the difference value calculated by a feature in the user behavior state subsequence in the trained time window as a label;

Step 6: Calculate the difference between each normal time window and the normal and abnormal time windows through equations (5)-(8) and save it as a set U ₂ , and compare the abnormal time window with the normal and abnormal time windows through equations (5)-(8). , the difference value is calculated in the abnormal time window and saved as P ₂ , and the confidence interval σ(β) of the field state subsequence under the normal time window is obtained as

σ(β)=[min(U2),max(U2)]∩[min(P2),max(P2)]

Step 7: Use a certain feature in the field state subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the difference value in pairs with the sample sets in other normal and abnormal time windows. , the calculation expression is:

v5=1-v6 (11)

f(tableγ)=v5*M′ _tableγ +v6*M″ _tableγ +bias (12)

tableγ represents the sample corresponding to a feature in the field state subsequence as a label,

tableγ′ represents the sample corresponding to a feature in the field state subsequence in the normal time window as a label,

"tableγ" represents the sample corresponding to a feature in the field state subsequence under the abnormal time window as a label,

M _tableγ ′ is the mean square between a sample corresponding to a feature in the field state subsequence as a label and a sample corresponding to a feature in the field state subsequence as a label in a normal time window error,

M _tableγ ″ is the mean square between the sample corresponding to a certain feature in the field state subsequence as the label and the sample corresponding to a certain feature in the field state subsequence under the abnormal time window as the label error,

bias is paranoia,

v5 and v6 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v5=v6,

f(tableγ) is the difference value calculated by a feature in the field state subsequence in the trained time window as a label;

Step 8: Calculate the difference between each normal time window and the normal and abnormal time windows through formulas (9)-(12) and save it as a set U ₃ , and use formulas (9)-(12) to calculate the difference between the abnormal time window and the normal time window. , the difference value is calculated in the abnormal time window and saved as P ₃ , and the confidence interval σ(γ) of the log quantity state subsequence under the normal time window is obtained as

σ(γ)=[min(U3),max(U3)]∩[min(P3),max(P3)].

In a preferred solution, the method for log analysis includes the following steps:

Step 1: when a detected time window is analyzed, randomly select a feature of each subsequence to form three labels and form three samples to represent the tested time window;

Step 2: Set the initial values v1=v2, v3=v4, v5=v6, and calculate the difference values f(tableα) and f(tableβ) of the three samples under the detection time window through formulas (1)-(12). ), whether f(tableγ) is within the corresponding confidence interval σ(α), σ(β), σ(γ);

Step 3: If the three samples are all within the confidence interval, then constrain v1, v2, v3, v4, v5, v6, and the constraint formula is

v1=a1*v1(0≤a＜1)

v3=a2*v3(0≤a2＜1)

v5=a3*v5(0≤a3＜1)

v1, v3, and v5 are the variation coefficients of the mean square error of the normal time window, respectively. If v1, v3, and v5 are reduced, the influence of the normal time window on the difference value is reduced, and the influence of the abnormal time window on the difference value is increased. According to the new v1 , v2, v3, v4, v5, v6 and the three samples re-pass formulas (1)-(12) to calculate whether the difference value of the three samples under the tested time window is within the corresponding confidence interval σ(α) , σ(β), σ(γ), and determine the number of repeated constraints according to the requirements of the false negative rate;

Step 4: If the difference value of the three samples under the tested time window is still within the confidence interval after the repeated constraint is over, it is considered that the information system is normal in the tested time window, otherwise it is considered abnormal.

The present invention also provides a log anomaly detection system, including:

The time window division module is used to determine the size of the time window according to the information system's requirements for response time.

The SGSE data processing module is used to process log data, and process it into sample data that can be called by the ECC log analysis model according to the time window.

The ECC model training module is used to train the ECC log analysis model.

The ECC log analysis module is used to judge whether the tested time window is normal according to the time window under test analyzed by the ECC log analysis model, and analyze whether the status of the information system under the time window is normal according to the logs of each device in the information system. .

In one solution, the time window dividing module determines the size of the time window as short as possible according to the response time required by the user to the information system.

In one solution, the SGSE data processing module is divided into SG state generation sub-module and SE sequential extraction sub-module.

The SG state generation sub-module determines each device in the information system, such as WAF, load balancing, firewall and the like. Count the number of logs of each device to generate a subsequence of log number status, which includes features such as the number of WAF device logs and the number of load balancing device logs.

Table 1: Log Count Status Subsequence

time window/device type Number of WAF logs The number of load balancing logs Number of firewall logs ... Number of Nginx logs time window N α ₁ α ₂ α ₃ ... α _n

Count the number of each log type generated on each device in the time window to obtain the user behavior status sub-sequence. The type is determined by combining the types of each field whose entropy value is not 0. For example, there is an action field in the WAF device for Record the behavior of the WAF device for access. There are two types of fields: alert and block; there are four types in the http_method field that records the http status. There are four types: 200, 404, 500, and 501. From these two fields, the WAF device can generate 2 *4 different kinds of logs. The subsequence includes multiple features such as the number of types of WAF logs and the number of types of firewall logs.

Table 2: User behavior state subsequences

Count the number of occurrences of types in some important fields of each device in the time window, and generate field status subsequences. The field status subsequence includes the number of alert types in the action field in the WAF, and the ratio of the number of TCP protocols in the firewall protocol field.

Table 3: Field Status Subsequence

The SE sequential extraction sub-module, in the same time window, sequentially extracts a feature in a subsequence as a label, and combines all the features of the other two subsequences into a sample, as shown in the following table:

Table 4: The number of WAF logs in the log number status subsequence is the tag feature merge table

Table 5: User behavior status sub-sequence WAF log category 1 is the tag feature merge table

Table 6: The field status subsequence WAF action field alert number is the tag feature merge table

By extracting submodules sequentially, any feature in any subsequence can be matched with all features in the other two subsequences. The generated sample features are associated with each other. When analyzing a time window, one feature of each subsequence is randomly selected to form three labels to represent the tested time window.

In one solution, the ECC model training module is divided into a small number of normal and abnormal event window determination submodules and an ECC model training submodule.

The small number of normal and abnormal event windows determine the sub-module. Only using the clustering algorithm cannot accurately classify, and the problem of inaccurate classification during analysis model training is magnified and affects the analysis results. In the existing historical data of the information system, a small number of time windows can be accurately determined as normal or abnormal time windows according to professional knowledge.

For the ECC model training submodule, the steps for model training are as follows:

Step 1: Process the multi-source heterogeneous log data in the normal and abnormal time windows through the SG status generation sub-module, and obtain the log quantity statistics of the log data to generate the log quantity status subsequence, and convert the The number of each log type is counted to generate a sub-sequence of user behavior status, and the number of occurrences of types in some important fields of each device in the time window is counted to generate a sub-sequence of field status.

Step 2: Extract the n features in the log quantity state subsequence, m features in the user behavior state subsequence, and j features in the field state subsequence in the SE order to generate (n+ m+j) sample datasets.

Step 2: Generate (n+m+j) samples from n features in the log quantity state subsequence, m features in the user behavior state subsequence, and j features in the field state subsequence under each time window data set;

Step 3: Take a certain feature in the log quantity status subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the sample data sets in other normal and abnormal time windows in pairs. The difference value, the calculation expression is:

v1=1-v2 (3)

f(tableα)=v1*M′ _tableα +v2*M″ _tableα +bias (4)

tableα represents the sample corresponding to a feature in the log quantity state subsequence as a label,

tableα' represents the sample corresponding to a certain feature in the log quantity status subsequence as a label in the normal time window,

"tableα" represents the sample corresponding to a certain feature in the log quantity status subsequence as a label under the abnormal time window,

M _tableα ′ is the difference between a sample corresponding to a certain feature in the log quantity state subsequence as a label and a sample corresponding to a certain feature in the log quantity state subsequence under a normal time window as a label. mean squared error,

M _tableα ″ is the difference between the sample corresponding to a certain feature in the log quantity state subsequence as a label and the sample corresponding to a certain feature in the log quantity state subsequence under the abnormal time window as a label mean squared error,

bias is paranoia,

v1 and v2 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively, and v1=v2 during training;

f(tableα) is the difference value calculated by a feature in the log quantity state subsequence in the trained time window as a label;

Step 4: Calculate the difference value between each normal time window and other normal and abnormal time windows through equations (1)-(4) and save it as a set U ₁ , and use equations (1)-(4) and The difference value is calculated in the normal and abnormal time windows and saved as P ₁ , and the confidence interval σ(α) of the log quantity state subsequence under the normal time window is obtained as

σ(α)=[min(U1),max(U1)]∩[min(P1),max(P1)]

Step 5: Calculate the sample data set with a certain feature in the user behavior state subsequence in each normal and abnormal time window as the label according to the ECC expression and the sample data sets in other normal and abnormal time windows. The difference value, the calculation expression is:

v3=1-v4 (7)

f(tableβ)=v3*M′ _tableβ +v4*M″ _tableβ +bias (8)

tableβ represents the sample corresponding to a feature in the user behavior state subsequence as a label,

tableβ' represents the sample corresponding to a certain feature in the subsequence of user behavior status under the normal time window as a label,

"tableβ" represents the sample corresponding to a certain feature in the subsequence of user behavior status under the abnormal time window as a label,

M _tableβ ′ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under a normal time window as a label. mean squared error,

M _tableβ ″ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under an abnormal time window as a label mean squared error,

bias is paranoia,

v3 and v4 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v3=v4,

f(tableβ) is the difference value calculated by a feature in the user behavior state subsequence in the trained time window as a label;

Step 6: Calculate the difference between each normal time window and the normal and abnormal time windows through equations (5)-(8) and save it as a set U ₂ , and compare the abnormal time window with the normal and abnormal time windows through equations (5)-(8). , the difference value is calculated in the abnormal time window and saved as P ₂ , and the confidence interval σ(β) of the field state subsequence under the normal time window is obtained as

σ(β)=[min(U2),max(U2)]∩[min(P2),max(P2)]

Step 7: Use a certain feature in the field state subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the difference value in pairs with the sample sets in other normal and abnormal time windows. , the calculation expression is:

v5=1-v6 (11)

f(tableγ)=v5*M′ _tableγ +v6*M″ _tableγ +bias (12)

tableγ represents the sample corresponding to a feature in the field state subsequence as a label,

tableγ′ represents the sample corresponding to a feature in the field state subsequence in the normal time window as a label,

"tableγ" represents the sample corresponding to a feature in the field state subsequence under the abnormal time window as a label,

M _tableγ ′ is the mean square between a sample corresponding to a feature in the field state subsequence as a label and a sample corresponding to a feature in the field state subsequence as a label in a normal time window error,

M _tableγ ″ is the mean square between the sample corresponding to a certain feature in the field state subsequence as the label and the sample corresponding to a certain feature in the field state subsequence under the abnormal time window as the label error,

bias is paranoia,

v5 and v6 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v5=v6,

f(tableγ) is the difference value calculated by a feature in the field state subsequence in the trained time window as a label;

Step 8: Calculate the difference between each normal time window and the normal and abnormal time windows through formulas (9)-(12) and save it as a set U ₃ , and use formulas (9)-(12) to calculate the difference between the abnormal time window and the normal time window. , the difference value is calculated in the abnormal time window and saved as P ₃ , and the confidence interval σ(γ) of the log quantity state subsequence under the normal time window is obtained as

σ(γ)=[min(U3),max(U3)]∩[min(P3),max(P3)].

In one solution, the ECC log analysis module has the following analysis steps:

Step 1: For the tested window, when analyzing a time window, randomly select one feature of each subsequence as three labels and form three samples to represent the tested time window, as shown in Figure 2.

Step 2: Set the initial values v1=v2, v3=v4, v5=v6, and calculate the values of Whether the difference values f(tableα), f(tableβ), and f(tableγ) of the three samples are within the corresponding confidence intervals σ(α), σ(β), and σ(γ).

Step 3: If the three samples are all within the confidence interval, the detection accuracy is improved by constraining v1, v2, v3, v4, v5, and v6. The constraint formula is:

v1=a1*v1(0≤a＜1)

v3=a2*v3(0≤a2＜1)

v5=a3*v5(0≤a3＜2)

v1, v3, and v5 are the changes of the normal time window, respectively. If v1, v3, and v5 are reduced, the influence of the normal time window on the difference value is reduced, and the influence of the abnormal event window on the difference value is increased. The obtained new v1, v2, v3, v4, v5, v6, and the three samples are brought back into the formulas (1)-(12) described in the ECC model training sub-module to check whether the difference value is within the confidence interval, and according to the prediction The set miss rate requires to determine the number of repeated constraints. If the miss rate is low, the number of constraints is small. If the false negative rate requirement is low, the number of constraints will be high.

Step 4: If the difference value of the time window is still within the confidence interval after the repeated constraint is over, it is considered that the information system is normal in the tested time window, otherwise it is considered abnormal.

Based on the single analysis and aggregation method, the logs in different devices are not combined and analyzed, but the status of different devices is judged separately and then analyzed, and the relationship between the logs of different devices cannot be mined. The SGSE algorithm proposed in the present invention can effectively process logs between different devices and aggregate the logs into samples that can reflect the state of the information system in this time window, and comprehensively judge the overall state of the information system instead of analyzing a single device to get the results post-polymerization.

In the method of association analysis, for the purpose of generating statistical reports of various events, it is impossible to deeply mine the relationship between various events and present them directly to the user. The in-depth analysis of multi-source heterogeneous logs through the SGSE-ECC algorithm not only generates statistical reports of various events through clustering, but also deeply mines the relationship between the logs to visually present the status of the information system to the user.

In general, the present invention uses the SGSE-ECC algorithm model to perform data processing, sample generation, and status analysis on multi-source heterogeneous logs, and automatically analyze logs, thereby reducing operation and maintenance costs.

The SGSE algorithm proposed by the present invention processes the log data on each device, aggregates and generates samples that can reflect the state of the time window, and proposes an ECC algorithm to analyze the multi-dimensional samples, and adjust the detection accuracy and false negative rate by constrained variation coefficients.

Claims (3)

A method for analyzing multi-source heterogeneous logs, comprising the following steps: Step 1: Determine the size of the time window according to the response time required by the information system; Step 2: Use the SGSE algorithm to process the log data in each time window into a sample that can be called by the ECC log analysis algorithm; Step 3: Train and use the ECC log analysis model to analyze whether the time window is normal; Step 4: Present the log analysis results. The multi-source heterogeneous log analysis method according to claim 1, wherein the step of model training is as follows: Step 1: Count the number of logs of multi-source heterogeneous log data in normal and abnormal time windows to generate a log number status subsequence, and count the number of each log type generated on each device within the time window to generate a user behavior status subsequence , count the number of occurrences of types in some important fields of each device in the time window to generate field status subsequences; Step 2: Generate (n+m+j) samples from n features in the log quantity state subsequence, m features in the user behavior state subsequence, and j features in the field state subsequence under each time window data set; Step 3: Take a certain feature in the log quantity status subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the sample data sets in other normal and abnormal time windows in pairs. The difference value, the calculation expression is:

v1=1-v2 (3) f(tableα)=v1*M′ _tableα +v2*M″ _tableα +bias (4) tableα represents the sample corresponding to a feature in the log quantity state subsequence as a label, tableα' represents the sample corresponding to a certain feature in the log quantity status subsequence as a label in the normal time window, "tableα" represents the sample corresponding to a certain feature in the log quantity status subsequence under the abnormal time window as the label, M _tableα ′ is the difference between a sample corresponding to a certain feature in the log quantity state subsequence as a label and a sample corresponding to a certain feature in the log quantity state subsequence under a normal time window as a label. mean squared error, M _tableα ″ is the difference between the sample corresponding to a certain feature in the log quantity state subsequence as a label and the sample corresponding to a certain feature in the log quantity state subsequence under the abnormal time window as a label mean squared error, bias is paranoia, v1 and v2 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively, and v1=v2 during training; f(tableα) is the difference value calculated by a feature in the log quantity state subsequence in the trained time window as a label; Step 4: Calculate the difference value between each normal time window and other normal and abnormal time windows through equations (1)-(4) and save it as a set U ₁ , and use equations (1)-(4) and The difference value is calculated in the normal and abnormal time windows and saved as P ₁ , and the confidence interval σ(α) of the log quantity state subsequence under the normal time window is obtained as σ(α)=[min(U1),max(U1)]∩[min(P1),max(P1)] Step 5: Calculate the sample data set with a certain feature in the user behavior state subsequence in each normal and abnormal time window as the label according to the ECC expression and the sample data sets in other normal and abnormal time windows. The difference value, the calculation expression is:

v3=1-v4 (7) f(tableβ)=v3*M′ _tableβ +v4*M″ _tableβ +bias (8) tableβ represents the sample corresponding to a feature in the user behavior state subsequence as a label, tableβ' represents the sample corresponding to a certain feature in the subsequence of user behavior status under the normal time window as a label, "tableβ" represents the sample corresponding to a certain feature in the subsequence of user behavior status under the abnormal time window as a label, M _tableβ ′ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under a normal time window as a label. mean squared error, M _tableβ ″ is the difference between a sample corresponding to a certain feature in the user behavior state subsequence as a label and a sample corresponding to a certain feature in the user behavior state subsequence under an abnormal time window as a label mean squared error, bias is paranoia, v3 and v4 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v3=v4, f(tableβ) is the difference value calculated by a feature in the user behavior state subsequence in the trained time window as a label; Step 6: Calculate the difference between each normal time window and the normal and abnormal time windows through equations (5)-(8) and save it as a set U ₂ , and compare the abnormal time window with the normal and abnormal time windows through equations (5)-(8). , the difference value is calculated in the abnormal time window and saved as P ₂ , and the confidence interval σ(β) of the field state subsequence under the normal time window is obtained as σ(β)=[min(U2),max(U2)]∩[min(P2),max(P2)] Step 7: Use a certain feature in the field state subsequence in each normal and abnormal time window as the label of the sample data set according to the ECC expression and calculate the difference value in pairs with the sample sets in other normal and abnormal time windows. , the calculation expression is:

v5=1-v6 (11) f(tableγ)=v5*M′ _tableγ +v6*M″ _tableγ +bias (12) tableγ represents the sample corresponding to a feature in the field state subsequence as a label, tableγ′ represents the sample corresponding to a feature in the field state subsequence in the normal time window as a label, "tableγ" represents the sample corresponding to a feature in the field state subsequence under the abnormal time window as a label, M _tableγ ′ is the mean square between a sample corresponding to a feature in the field state subsequence as a label and a sample corresponding to a feature in the field state subsequence as a label in a normal time window error, M _tableγ ″ is the mean square between the sample corresponding to a certain feature in the field state subsequence as the label and the sample corresponding to a certain feature in the field state subsequence under the abnormal time window as the label error, bias is paranoia, v5 and v6 are the variation coefficients of the mean square error of the normal time window and the abnormal time window, respectively. During training, v5=v6, f(tableγ) is the difference value calculated by a feature in the field state subsequence in the trained time window as a label; Step 8: Calculate the difference between each normal time window and the normal and abnormal time windows through formulas (9)-(12) and save it as a set U ₃ , and use formulas (9)-(12) to calculate the difference between the abnormal time window and the normal time window. , the difference value is calculated in the abnormal time window and saved as P ₃ , and the confidence interval σ(γ) of the log quantity state subsequence under the normal time window is obtained as σ(γ)=[min(U3),max(U3)]∩[min(P3),max(P3)]. The method for analyzing multi-source heterogeneous logs according to claim 2, wherein the method for analyzing logs comprises the following steps: Step 1: when a detected time window is analyzed, randomly select a feature of each subsequence to form three labels and form three samples to represent the tested time window; Step 2: Set the initial values v1=v2, v3=v4, v5=v6, and calculate the difference values f(tableα) and f(tableβ) of the three samples under the detection time window through formulas (1)-(12). ), whether f(tableγ) is within the corresponding confidence interval σ(α), σ(β), σ(γ); Step 3: If the three samples are all within the confidence interval, then constrain v1, v2, v3, v4, v5, v6, and the constraint formula is v1=a1*v1(0≤a＜1) v3=a2*v3(0≤a2＜1) v5=a3*v5(0≤a3＜1) v1, v3, and v5 are the variation coefficients of the mean square error of the normal time window, respectively. If v1, v3, and v5 are reduced, the influence of the normal time window on the difference value is reduced, and the influence of the abnormal time window on the difference value is increased. According to the new v1 , v2, v3, v4, v5, v6 and the three samples re-pass formulas (1)-(12) to calculate whether the difference value of the three samples under the tested time window is within the corresponding confidence interval σ(α) , σ(β), σ(γ), and determine the number of repeated constraints according to the requirements of the false negative rate; Step 4: If the difference value of the three samples under the tested time window is still within the confidence interval after the repeated constraint is over, it is considered that the information system is normal in the tested time window, otherwise it is considered abnormal.

Images