CN110262949A - Intelligent device log processing system and method - Google Patents

Abstract

The invention discloses a log processing system and method for smart devices, and relates to a log processing system for smart devices in the Internet of Things, including a log acquirer: used to obtain the original log of the smart device according to the parameters of the smart device; a log parser: used to receive the Raw logs, parsing unstructured or semi-structured raw logs into structured logs, and generating log sequences; security rule base: containing log analysis rules; log analysis engine: used to receive the structured logs and logs sequence, and perform rule matching on the structured log according to the log analysis rules of the security rule base, and perform hierarchical clustering on the log sequence; log analysis reporter: used to analyze the log according to the results of the log analysis engine and the The security rule library generates a report from the analysis results; the user interface module: used to interact with the user, receive the parameters of the smart device, and output the report to the user. The log acquisition of the invention is efficient, simple and accurate.

Description

Intelligent device log processing system and method

technical field

The invention relates to the technical field of a log processing system for an intelligent device of the Internet of Things, in particular to a log processing system for an intelligent device and a processing method thereof.

Background technique

At present, there are many audit methods for logs, including audit methods based on rule base, data mining, immune system and neural network. The audit detection accuracy based on the rule base is high, but it is not suitable for changing network attacks. Auditing based on data mining can handle large-scale data files, but the accuracy rate needs to be tuned. Auditing based on evolutionary algorithms can effectively optimize auditing through multiple dimensions, but it cannot detect simultaneous attacks. Auditing based on the immune system can detect attacks with memory and pathogens, but cannot judge attacks with race conditions and identity masquerade. Among them, the audit method based on the rule base and the data mining algorithm that has attracted attention due to the upsurge of big data are commonly used and have achieved certain results. Neural network algorithms, genetic algorithms and evolutionary algorithms related to biology are still to be further studied. For data mining, the analysis of logs is mainly divided into supervised learning and unsupervised learning, mainly depending on whether the logs are marked. Supervised learning mainly uses classification algorithms, while unsupervised learning mainly uses clustering algorithms. Through cluster analysis, the collected original logs are divided into two categories: normal logs and suspicious logs, and then the association rules of the two types of logs are mined separately. Get normal operation rules and abnormal operation rules.

At present, the mainstream log analysis tools are relatively mature, but their limitations are that when collecting logs, it is necessary to deploy collection software or install log collection tools on the collection objects. For smart devices, it is complicated and difficult to deploy software on them. big. Some smart devices do not support common protocols for log collection, making it difficult to obtain logs from smart devices.

When auditing the obtained logs, the current mainstream method is still rule-based matching. This method is efficient, simple and intuitive, but with the increase of attack methods, for attack methods that are not in the security rule base , the log audit effect is not good, and for large-scale log data, it is impossible to dig out information other than the rules.

Contents of the invention

In view of this, the purpose of the present invention is to propose a smart device log processing system and its processing method, which are used to solve the problems in the background technology that the log acquisition of smart devices is difficult and the log audit effect is not good. It has the advantages of simple log acquisition and excellent audit effect specialty.

The present invention provides an intelligent device log processing system, comprising:

Log acquirer: used to obtain the original log of the smart device according to the parameters of the smart device;

Log parser: used to receive the original log, parse the unstructured or semi-structured original log into a structured log, and generate a log sequence;

Security rule base: contains log analysis rules;

Log analysis engine: used to receive the structured log and log sequence, and perform rule matching on the structured log according to the log analysis rules of the security rule base, and perform hierarchical clustering on the log sequence;

Log analysis reporter: used to generate a report on the analysis results according to the results of the log analysis engine and the security rule base;

User interface module: used to interact with the user, receive the parameters of the smart device, and output the report to the user.

Optionally, the log acquirer identifies the brand and model of the smart device through the interface provided by the smart device, determines the protocol and communication method adopted by the smart device, and reports the IP address corresponding to the smart device according to the protocol and interaction mode adopted by the smart device. Send a request, and obtain the original log generated by the running process of the smart device by parsing the response returned by the smart device.

Optionally, the log parser includes:

Log cleaner: for removing redundant information in the original log, and parsing the original log into the structured log;

An event extractor: used to extract the operation events of the parsed structured log records;

A log combiner: used to combine a single structured log according to the context according to the occurrence time of the structured log and the operation event, and generate a corresponding log sequence.

Optionally, the log analysis engine includes:

Rule matcher: for matching the corresponding log records from the structured logs according to the rules configured by the user and the log analysis rules in the security rule base, and summarizing and sorting the log records;

Clustering Analyzer: used to perform hierarchical clustering on the log sequence generated by the log parser, find large clusters and isolated points according to the clustering results, analyze the operating rules of the equipment from the large clusters, and find abnormalities from the isolated points Behaviors to obtain information beyond the rules in structured logs;

Log analysis interface: for interacting with the log analysis reporter according to the results of the rule matcher and the cluster analyzer.

Based on the same invention, the present invention also provides a smart device log processing method, including:

Obtain unstructured or semi-structured original logs of smart devices according to the received smart device parameters;

Parsing the unstructured or semi-structured original log into a structured log to generate a log sequence;

performing rule matching on the structured log according to the log analysis rules of the security rule base;

performing hierarchical clustering on the log sequence;

generating a report according to the rule matching result and the hierarchical clustering result of the log sequence; and outputting the report.

Optionally, the parsing of the original log includes:

removing redundant information in the original log, and parsing the original log into the structured log;

extracting the operation events of the parsed structured log records;

According to the occurrence time of the structured log and the operation event, a single structured log is combined according to the context to generate a corresponding log sequence.

Optionally, the rule matching and hierarchical clustering include:

matching the corresponding log records from the structured logs according to the rules configured by the user and the log analysis rules in the security rule base;

Perform hierarchical clustering on the log sequence, find large clusters and isolated points according to the clustering results, analyze the operating rules of the equipment from the large clusters, find abnormal behaviors from the isolated points, and obtain information contained outside the rules in the structured log;

Interacting with the log analysis reporter based on the results of the rule matcher and the cluster analyzer.

Optionally, the matching of corresponding log records from the structured logs includes: using a rule matcher, using user-defined rules and rules in the security rule base to match the structured logs, counting the number of hits, and recording hit logs .

Optionally, performing hierarchical clustering on log sequences includes:

Preprocessing the log sequence, converting the log sequence into vector data;

Use the improved cosine similarity algorithm to generate a distance matrix using vector data, input the hierarchical clustering algorithm, and generate a clustering tree;

Calculate the relevant evaluation index parameters for different cluster numbers, and select the cluster number corresponding to the best parameter as the cluster number;

Traverse the logs in each cluster number, and count the number and type of logs in each cluster number.

Inductive analysis and association analysis are performed on the results of rule matching and hierarchical clustering, and new security rules are generated and compensated into the security rule base.

Based on the same invention, the present invention also provides a confidential security inspection method for a log processing system of an intelligent device, including:

Submit the parameters of the smart device to be tested;

Enter the log processing system of the smart device under test and obtain the retained log audit report;

Detection log audit report;

Based on the detection results, trace the source of illegal operations through risk positioning, risk investigation and traceability.

The present invention has the following advantages by the foregoing:

This system uses the interface provided by the smart device to connect with the log acquirer. First, it identifies the brand and model of the smart device, generates a corresponding instance, sends a request to the specified IP according to the protocol and interaction mode of different devices, and automatically obtains the device operation log by analyzing the response. There is no need to restore the storage medium of the smart device or install a log collection device on the smart device to obtain the log, and the log acquisition is efficient, simple and accurate.

In addition to using the security rule base for log processing, this system also introduces a clustering method, which can mine content outside the rules, such as the operation rules of smart devices, user habits, and abnormal events corresponding to isolated points, and can dynamically compensate abnormal events to The security rule base improves the audit effect of logs.

Description of drawings

Fig. 1 is the structural block diagram of the intelligent device log processing system described in the embodiment of the present invention;

2 is a flow chart of a method for processing a log of a smart device according to an embodiment of the present invention;

Fig. 3 is a flow chart of log analysis engine processing according to an embodiment of the present invention;

Fig. 4 is a flow chart of a method for performing a security check by the smart device log processing system according to an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

The main objective of the present invention is to develop a set of log security processing system for smart devices in the Internet of Things. The system can automatically obtain device operation logs according to the smart device model input by the user, and find the user's interest in the log according to the security rules configured by the user. At the same time, clustering methods are used to analyze logs to reflect equipment operation rules, and isolated points are used to locate abnormalities and report to users. The first step for smart devices is to consider supporting mainstream digital copier brands and their sub-models in the market.

The smart device log processing system described in the embodiment of the present invention includes:

Log acquirer: used to obtain the original log of the smart device according to the parameters of the smart device;

Log parser: used to receive the original log, parse the unstructured or semi-structured original log into a structured log, and generate a log sequence;

Security rule base: contains log analysis rules;

A log analysis engine: used to receive the log sequence, perform rule matching on the structured log according to the log analysis rules of the security rule base, and perform hierarchical clustering on the log sequence;

Log analysis reporter: used to generate a report on the analysis results according to the results of the log analysis engine and the security rule base;

User interface module: used to interact with the user, receive the parameters of the smart device, and output the report to the user.

This system uses the interface provided by the smart device to connect with the log acquirer. First, it identifies the brand and model of the smart device, generates a corresponding instance, sends a request to the specified IP according to the protocol and interaction mode of different devices, and automatically obtains the device operation log by analyzing the response. There is no need to restore the storage medium of the smart device or install a log collection device on the smart device to obtain the log, and the log acquisition is efficient, simple and accurate.

In addition to using the security rule base for log auditing, this system also introduces a clustering method, which can mine content outside the rules, such as the operation rules of smart devices, user habits, and abnormal events corresponding to isolated points, and can dynamically compensate abnormal events to The security rule base improves the audit effect of logs.

As shown in Figure 1, in some embodiments, the smart device log processing system includes:

Log acquirer: used to obtain the original log of the smart device according to the parameters of the smart device.

In an embodiment of the present invention, the above-mentioned smart device parameters may include a smart device model, a device IP, and user-defined rules.

In an embodiment of the present invention, the above-mentioned log acquirer can identify the brand and model of the smart device through the interface provided by the smart device, determine the protocol and communication method adopted by the smart device, and thereby generate an instance corresponding to the smart device; according to the smart device The adopted protocol and interaction method send a request to the corresponding IP of the smart device, and automatically obtain the original log generated by the running process of the smart device by analyzing the response returned by the smart device.

Log parser: for receiving the original log, processing and parsing the original log, thereby parsing the unstructured or semi-structured original log into a structured log, and generating a log sequence.

Log analysis engine: used to analyze the content generated by the log parser, specifically including rule matching of structured logs according to the log analysis rules in the security rule base, and hierarchical clustering of log sequences.

Security rule base: used to provide log analysis rules for the log analysis engine.

In the embodiment of the present invention, the above security rule base may further provide corresponding security risk support for the log analysis engine. The specific performance is that there are two kinds of content stored in the security rule base: one is the rules configured by users. For example, for secret-related units, the operation of U disk for secret-related equipment is a sensitive operation and should be pre-configured in the security rule base; two. It is the log operation behavior judged by the auditors as abnormal or unsafe after being processed by the system, and should be added to the security rule base in the form of rules.

In the embodiment of the present invention, the above-mentioned security rule base can also be used to provide risk improvement strategy reports, specifically, there will be a field in the rule to record the security level of the rule, such as high risk, medium, common, etc., this field is set by the user or Auditors specify that when a report is generated, the security level of sensitive events is reflected in the report, thereby assessing the risk.

Log analysis reporter: used to generate a report on the analysis results according to the results of the log analysis engine and the security rule base.

In an embodiment of the present invention, the above analysis results may include user interest rules, device operation rules, discovered security risks, and the like.

User interface module: used to interact with the user, on the one hand, receive the parameters of the smart device, and on the other hand, output the report to the user.

This system uses the interface provided by the smart device to connect with the log acquirer. First, it identifies the brand and model of the smart device, generates a corresponding instance, sends a request to the specified IP according to the protocol and interaction mode of different devices, and automatically obtains the device operation log by analyzing the response. There is no need to restore the storage medium of the smart device or install a log collection device on the smart device to obtain the log, and the log acquisition is efficient, simple and accurate.

In addition to using user-defined rules and security rule bases for log auditing, this system also introduces a clustering method, which can mine content outside the rules, such as the operating rules of smart devices, user habits, and abnormal events corresponding to isolated points, and can Abnormal events are dynamically compensated to the security rule base.

In some embodiments, the log parser may include:

Log cleaner: for removing redundant information in the original log, and parsing the original log into the structured log;

Event extractor: used to extract the operation events of the parsed structured log records. Specifically, the operation event refers to that in some smart devices, a user's operation behavior will be decomposed into a series of operation events, such as when the user utilizes When the digital copier performs a copy operation, the copy behavior corresponds to three operation events of login, copy, and logout in the digital copier;

Log combiner: used to combine a single structured log according to the context and generate a corresponding log sequence according to the occurrence time and operation event of the structured log.

In some embodiments, the log analysis engine includes:

Rule matcher: used to match the corresponding log records from the structured logs according to the rules configured by the user and the log analysis rules in the security rule base, and summarize and organize the log records. Record and highlight the hit content;

Clustering Analyzer: used to perform hierarchical clustering on the log sequence generated by the log parser, find large clusters and isolated points according to the clustering results, analyze the operating rules of the equipment from the large clusters, and find abnormalities from the isolated points Behavior, so as to obtain the information contained in the structured log outside the rules;

Log analysis interface: for interacting with the log analysis reporter according to the results of the rule matcher and the cluster analyzer.

Based on the same invention, the present invention also provides a processing method for smart device logs. As shown in Figure 2, the method includes the following steps:

S11: Obtain unstructured or semi-structured raw logs of the smart device according to the received parameters of the smart device;

S12: Parse the unstructured or semi-structured original log into a structured log to generate a log sequence;

S13: Perform rule matching on the structured log according to the log analysis rules of the security rule base;

S14: Perform hierarchical clustering on the log sequence;

S15: Generate a report according to the rule matching result and the hierarchical clustering result of the log sequence; and output the report.

In some embodiments, the acquisition of the unstructured or semi-structured original log of the smart device may specifically include: identifying the brand and model of the smart device; determining the protocol and communication method adopted by the smart device; Send a request to the IP corresponding to the smart device in an interactive way; and obtain the original log generated by the running process of the smart device by analyzing the response returned by the smart device.

In some embodiments, the parsing of the original log may also include the following steps:

S21: Remove redundant information in the original log, and parse the original log into the structured log;

S22: extracting the operation event recorded in the structured log after parsing;

S23: According to the occurrence time of the structured log and the operation event, combine the single structured log according to the context to generate a corresponding log sequence.

In addition to generating structured logs, the log parser will also analyze the structured logs according to the context to form a log combination, so as to ensure that the clustering object is not a single log but a log combination during the clustering process, thereby improving the clustering effect.

Specifically, the rule matching and hierarchical clustering may include:

matching the corresponding log records from the structured logs according to the rules configured by the user and the log analysis rules in the security rule base;

Perform hierarchical clustering on the log sequence generated by the log parser, find large clusters and isolated points according to the clustering results, analyze the operating rules of the equipment from the large clusters, find abnormal behaviors from the isolated points, and obtain the structured logs. Information implied outside the rules;

Interacting with the log analysis reporter based on the results of the rule matcher and the cluster analyzer.

As shown in FIG. 3 , in some embodiments, the above-mentioned rule matching of structured logs and hierarchical clustering analysis of log sequences may specifically include the following steps:

a) Determine whether the log to be analyzed is a structured log or a log sequence. If it is a structured log, use the user-defined rules and the rules in the security rule base to match the structured log through the rule matcher, count the number of hits, and record the hit log , go to step f, if it is a log sequence, go to step b;

b) For the log sequence, it is preprocessed to convert the log sequence into vector data;

c) Using vector data to generate a distance matrix, the calculation method is an improved cosine similarity algorithm, and input a hierarchical clustering algorithm to generate a clustering tree. The calculation method of the distance between logs does not use the Hamming distance commonly used in log clustering. Instead, the vectorized log data is calculated using the improved cosine similarity to generate a distance matrix;

d) Calculate the relevant evaluation index parameters for different cluster numbers, and select the cluster number corresponding to the best parameter as the cluster number;

e) traverse the logs in each cluster number, and count the number and type of logs in each cluster number;

f) Conduct inductive analysis and correlation analysis on the results of step a and step e, generate new safety rules and compensate them into the safety rule base.

In addition to using user-defined rules and security rule bases for log auditing, this system also introduces a clustering method, which can mine content outside the rules, such as the operating rules of smart devices, user habits, and abnormal events corresponding to isolated points, and can Abnormal events are dynamically compensated to the security rule base, which improves the audit effect of logs.

Based on the same invention, the present invention also provides a confidential security inspection method for a log processing system of an intelligent device, as shown in FIG. 4 , comprising the following steps:

S41: Submit parameters of the smart device to be tested;

S42: Enter the log processing system of the smart device under test, and obtain the retained log audit report;

S43: detection log audit report;

S44: Based on the detection results, trace the source of illegal operations through risk positioning, risk investigation, and traceability.

When the confidentiality unit conducts security inspections on its secret-related smart devices, it can use this system to obtain and detect the logs retained by the smart devices, find risky operations, and trace the source of illegal operations, which improves the practicability of the system.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present invention, the above embodiments or Combinations between technical features in different embodiments are also possible, steps may be carried out in any order, and there are many other variations of the different aspects of the invention as described above, which are not presented in detail for the sake of brevity.

Claims (10)

1. smart machine log processing system characterized by comprising

Log acquisition device: for the parameter according to smart machine, the original log of smart machine is obtained；

Log resolver: for receiving the original log, the unstructured or semi-structured original log is parsed into Structuring log, and generate logged sequence；

Security rule base: include log analysis rule；

Log analysis engine: for receiving the structuring log and logged sequence, and according to the log of the security rule base Analysis rule carries out rule match to the structuring log, and carries out hierarchical clustering to logged sequence；

Log analysis reporter: for according to the log analysis engine result and the security rule base, result will be analyzed Generate report；

Subscriber Interface Module SIM: for interacting with user, parameter described in smart machine is received, and by the report output to user.

2. system according to claim 1, which is characterized in that the interface that the log acquisition device is provided by smart machine The brand and model of identification intelligent equipment, determine agreement and communication mode used by smart machine, according to smart machine institute The agreement and interactive mode of use send request to the corresponding IP of smart machine, and are obtained by the response that parsing smart machine returns The original log for taking smart machine operational process to generate.

3. system according to claim 1, which is characterized in that the log resolver includes:

Log washer: for removing the redundancy in the original log, the original log is resolved into the knot Structure log；

Event extraction device: for extracting the action event of the structuring log recording after parsing；

Log combiner: time and the action event for being occurred according to the structuring log, by structure described in single Change log to be contextually combined, generates corresponding logged sequence.

4. system according to claim 1, which is characterized in that the log analysis engine includes:

Rule match device: for according to the log analysis rule in the rule of user configuration and the security rule base, from described Corresponding log recording is matched in structuring log, and log recording is summarized and arranged；

Clustering device: the logged sequence for generating to the log resolver carries out hierarchical clustering, is tied according to cluster Fruit discovery clusters greatly and isolated point, obtains structuring by Outlier detection abnormal behaviour by big clustering analysis equipment moving law The outer information contained of rule in log；

Log analysis interface: for according to the result of the rule match device and the clustering device and the log analysis report Device is accused to interact.

5. a kind of smart machine log processing method characterized by comprising

The parameter of smart machine based on the received obtains the unstructured or semi-structured original log of smart machine；

Described unstructured or semi-structured original log is parsed into structuring log, generates logged sequence；

Rule match is carried out to the structuring log according to the log analysis rule of security rule base；

Hierarchical clustering is carried out to the logged sequence；

Report is generated according to the rule match result of the logged sequence and hierarchical clustering result；And the output report.

6. method according to claim 5, which is characterized in that the original log, which parses, includes:

By the redundancy removal in the original log, the original log is resolved into the structuring log；

The action event of the structuring log recording after extracting parsing；

The time occurred according to the structuring log and the action event, contextually by structuring log described in single It is combined, generates corresponding logged sequence.

7. according to the method described in claim 5, it is characterized in that, the rule match and hierarchical clustering include:

According to the log analysis rule in the rule of user configuration and the security rule base, matched from the structuring log Corresponding log recording out；

Hierarchical clustering is carried out to the logged sequence, is clustered and isolated point greatly according to cluster result discovery, is set by big clustering analysis Standby moving law obtains the outer information contained of rule in structuring log by Outlier detection abnormal behaviour；

It is interacted according to the result of the rule match device and the clustering device and the log analysis reporter.

8. the method according to the description of claim 7 is characterized in that described match corresponding log from the structuring log Record includes: to be carried out using rule in user's custom rule and security rule base to structuring log by rule match device Matching counts hit quantity, record hit log.

9. according to the method described in claim 5, it is characterized in that, described include: to logged sequence progress hierarchical clustering

The logged sequence is pre-processed, logged sequence is converted into vector data；

Distance matrix is generated using vector data using improved cosine similarity algorithm, inputs hierarchical clustering algorithm, is generated poly- Class tree；

Relevant evaluation index parameter is calculated for the difference quantity that clusters, chooses the corresponding quantity that clusters of optimal parameter as clustering number Mesh；

The log that traversal each clusters in number counts log number amount and type in the number that respectively clusters.

The result of rule match and hierarchical clustering is summarized and analyzed and association analysis, generates new safety regulation and compensation enters The security rule base.

10. a kind of secret and safe inspection method of smart machine log processing system characterized by comprising

Submit smart machine parameter to be measured；

Into smart machine log processing system to be measured, and obtain the log audit report of retention；

Detect log audit report；

Based on testing result, checks and trace to the source and trace to the source violation operation by risk positioning, risk.