JP2014035749A - Log generation rule creation device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a log occurrence rule (rule) useful for a user without directly knowing the contents of a log message generated by a device to be viewed.
The present invention relates to a log template information storage storing at least a log message (hereinafter referred to as “log”) of a monitoring target device and an important part (hereinafter referred to as “template”) in the log message. Means for extracting a template corresponding to a log generated in the past, clustering templates having high co-occurrence for each monitoring target device, generating a group of the monitoring target device and a template, Stores in the group storage means as an event, reads each event in the group storage means, counts the number of occurrences for the monitored device and template of the event, and stores the occurrences with a high occurrence number in the rule storage means To do.
[Selection] Figure 2

Description

  The present invention relates to a log generation rule creation device and method, and more particularly, to a rule creation device and method for extracting information useful to a user from log information output by various machines.

  More specifically, the present invention relates to a log generation rule creation device and method for extracting necessary information (rule) without directly knowing in advance a generation rule of a log format generated by a monitoring target device.

  Today, the central reason for cost reduction is centralized monitoring and management of equipment and software with different roles from different manufacturers. On the other hand, each of these various devices and software has a mechanism for outputting a log with a unique format, and is used when monitoring and managing the device. With the development of information equipment, the log information has become complicated and large-scale, and an efficient monitoring method is required.

  Under such circumstances, there is an analysis infrastructure for simplifying log analysis (see Non-Patent Document 1, for example). However, in order to use this, prior knowledge about the meaning of each log and the contents of the log message is required, which makes it unsuitable for application to huge and complex logs.

  Also, a failure information management method is known in which a log number is assigned to a previously generated error log, retained for a certain period, and notified every time the same error log number occurs again.

  On the other hand, there is a technology that targets a syslog generated by a network device such as a router and provides a template grasp method when a certain format such as vendor, message type, error code, and detailed message content is given in advance (for example, Non-Patent Document 2). This technology proposes a method for displaying the digest information of syslog using the positional relationship of routers.

  Under these circumstances, there is a method of extracting a general log template using a priori information and not only for syslog.

Splunk http://www.splunk.com/ T. Qiu, Z. Ge, D. Pei, J. Wang, J, Xu, "What Happened in my Network? Mining Network Events from Router Syslogs", In IMC, 2010.

  However, the technique of Non-Patent Document 1 described above has a message part indicating an actually occurred event in the log generated by the machine, and a value unique to the error code such as an error code, an ID number, a place where the error occurred, and a time. In many cases, the error log is not uniquely determined even if the same event occurs. For this reason, logs indicating the occurrence of the same event are managed by different numbers, which is not inefficient and useful.

  Further, the technology of Non-Patent Document 2 requires at least prior knowledge at the stage of collecting information. For example, it is necessary to know from which vendor the log is collected and which part of the message indicates the error code. Manual input and preparation are required. Moreover, since the digest display function of syslog is a technique specialized for syslog, there is a problem that it cannot be applied to other monitoring log information.

  In addition, with the method of extracting a general log template that uses only prior information and is not the only syslog, the meaning of the log is difficult with the template alone, and a technology that automatically grasps the relationship between multiple logs that occur. Is required.

  The present invention has been made in view of the above points, and the user can create a log occurrence rule (rule) useful for the user without directly knowing the contents of the log message generated by the viewing target device. An object of the present invention is to provide an apparatus and method for creating a log generation rule that can be extracted.

In order to solve the above-mentioned problem, the present invention (Claim 1) is a log generation rule creation device for generating a log type generation rule (rule) of a monitored device,
Reference is made to log template information storage means storing at least a log message of the monitoring target device (hereinafter referred to as “log”) and an important part (hereinafter referred to as “template”) in the log message. A template extraction means for extracting a template corresponding to the log that has occurred;
Clustering a template having high co-occurrence for each monitoring target device, generating a group of the monitoring target device and the template, and storing the group in a group storage unit as an event;
A rule grasping unit that reads each event of the group storage unit, counts the number of occurrences for the monitored device and template of the event, and stores the occurrence number in the rule storage unit as an occurrence rule; .

The present invention (Claim 2) provides the rule grasping means,
Among the template IDs included in the same event of the group storage means, those associated with the same monitored device ID are extracted, and the template set and the monitored device ID are used as template rule candidates. The process of storing in the storage means is repeated for all events, the template rule candidates are compared with each other for all template rule candidates stored in the template rule candidate storage means, and the similarity to the set of template IDs is greater than a predetermined threshold value. If there are more than a predetermined number of devices with different monitoring target device IDs, a template rule extracting means that collectively sets them as template rules;
Among the monitoring target device IDs included in the same event of the group storage unit, those associated with the same template ID are extracted, and the template of the template ID and the monitoring target device ID are monitored as monitoring target device rule candidates. The process of storing in the target device rule candidate is repeated for all events, and for all the target device rule candidates stored in the target device rule candidate storage unit, the target device rule candidates are compared with each other, And a monitoring target device rule extracting unit that collectively sets a monitoring target device rule if the similarity between the pair is higher than a predetermined threshold and there are a predetermined number or more of different template IDs.

In the present invention (Claim 3), the log template information storage means includes a monitoring information label,
Means for replacing the monitoring target device ID with the monitoring information label if the monitoring target device ID included in the list of monitoring target device IDs of the template rule has a predetermined ratio or more of the monitoring target device IDs. Also have.

In the present invention (Claim 4), the log template information storage means includes a template label,
If the template ID included in the list of template IDs of the monitoring target device rule has a predetermined ratio or more, a means for replacing the template ID with the template label is further included.

  As described above, according to the present invention, it is possible to extract a log generation rule useful for a user without directly knowing the content of the log message generated by the monitoring target device.

1 is a system configuration diagram according to an embodiment of the present invention. It is a block diagram of the rule preparation apparatus in one embodiment of this invention. It is a flowchart of the process of the rule preparation apparatus in one embodiment of this invention. 1 is a schematic diagram (offline use example) of a system according to a first embodiment of this invention. It is the schematic (example of online utilization) of the system of the 2nd Example of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a system configuration according to an embodiment of the present invention.

  The system shown in the figure collects logs from a plurality of monitoring target device groups 3 and monitoring target device groups 3, reads the log collection device 10 and log DB 20 to output to the log DB 20, generates a template from the log message, A log generation device 30 that generates a log generation rule (rule) by reading the information in the log editing device 30 and the log template label information DB 40 stored in the log template label information DB 40 by assigning a label, and storing in the rule DB 200 It is composed of a rule creation device 100. In FIG. 1, the log collection device 10 and the log editing device 30 are described separately, but the present invention is not limited to this example and may be a single device.

  The log collection device 10 and the log editing device 30 described above can be realized by existing technology. The log collection device 10 collects a log from the monitoring target device 3, and the log editing device 30 that stores the log in the log DB 20, for example, acquires a log message from the log DB 20, and for each word included in the log message A score is calculated based on a set of appearance frequencies for the appearance position of the word, the word, and the number of words included in the log message, and a template is generated by performing clustering based on the score, It is possible to use a storage method. Also, it is assumed that the service type, log priority / importance, etc. are labeled according to the usage status. The label is given by the user as needed, and may be omitted.

  The log of the log DB 20 includes information indicating the monitoring target device 3 that generated the log, and the monitoring target device can be identified.

  The rule creation device 100 refers to the log / template / label information DB 40 to determine which template the log information generated in the past matches, and determines the template for each specific time width, service provision type, and monitored device 3. Clusters that have high co-occurrence, and stores monitored devices and template groups as events. The number of occurrences is counted for the obtained event templates and monitored devices, and the number of occurrences is output to the rule DB 200 as a log occurrence rule.

  Below, the rule creation apparatus 100 is demonstrated.

  FIG. 2 shows the configuration of the rule creation device according to one embodiment of the present invention.

  The rule creation apparatus 100 shown in the figure includes a template ID extraction unit 110, a log division unit 120, a group generation unit 130, a rule grasping unit 140, a rule merging unit 150, a group storage unit 160, a template rule candidate storage unit 170, and a monitoring target. The device rule candidate storage unit 180 is configured.

  The rule grasping unit 140 includes a template rule grasping unit 141 and a monitoring target device rule grasping unit 142.

  The group storage unit 160, the template rule candidate storage unit 170, and the monitoring target device rule candidate storage unit 180 are storage media such as a memory and a hard disk provided in the rule creation device 100.

  FIG. 3 is a flowchart of processing of the rule creation device according to one embodiment of the present invention.

  As a premise shown below, each monitored device 3 is assigned a unique monitored device ID. Similarly, a template ID is assigned to each template, and stored in the log template label information DB 40 together with the log. Shall.

  Step 101) The template ID extraction unit 110 reads data from the log / template / label information DB 40, which log template matches the log message that occurred in the past, along with the time of occurrence, and the ID unique to the monitoring target device. And the matching log template ID is extracted.

Step 102) The log dividing unit 120 divides the read log into M meaningful delimiters S _m (m = 1, 1) such as a specific time width (one day unit, etc.), a service providing type, a monitored device group type, 2, ..., M).

Step 103) The group generation unit 130 performs clustering by comparing the high co-occurrence of the monitoring target device and the template in each series S _m divided by the log dividing unit 120, and performs high co-occurrence monitoring. A group of target devices and templates is created and stored in the group storage unit 160. Various methods are conceivable for clustering. As one of the realization methods, non-negative matrix partitioning (for example, Non-Patent Document 3: DD Lee and HS Seung, “Algorithms for Non-negative Matrix Factorization,” In Proc. Of NIPS , 2000) will be described.

For each log series S _m in the following manner, to create a template occurrence matrix V _m.

1) The {monitoring target device unique ID, log template ID} appearing in each series S _m is represented as q _i (i = 1, 2,..., Q _m ), and a total of Q _m {monitoring target device unique IDs , Log template ID} appears. The index i of q _i is made to correspond to the row number of the template occurrence matrix V _m .

2) Divide the log sequence from the start to the end with an appropriate time width (1 sec., Etc.), and let each time window be t _j (j = 1, 2,..., T _m ). T _m is the number of time windows in S _m . It is assumed that the time window index j corresponds to the column number of the template occurrence matrix V _m .

3) If {monitored device unique ID, log template ID} q _i occurs in each time window t _j , the number of occurrences is recorded in the (i, j) component of the matrix V _m .

  4) Subsequently, the feature of the template occurrence matrix is grasped using non-negative matrix decomposition (Non-patent Document 3). Since the result of non-negative matrix decomposition changes depending on the initial value, a result that minimizes the cost function of non-negative decomposition is adopted after being executed several times.

5) The number of features K obtained by non-negative matrix decomposition needs to be given in advance, but is set appropriately according to the row size Q _m of the matrix V _m (eg, if Q _m = 250, K = 100).

6) The output of non-negative matrix factorization and Q _m × K matrix W _m and K × matrix H _m.

7) W each column vector w _m of _{m, k (k = 1,2,} ..., with respect to K), component [w _{m, k]} the _i, sorted in descending order of value, went raised adding to the order Q _i corresponding to components that sometimes exceed a certain threshold (such as 0.9) is taken as one group. This group is called an event and is represented by _{em, k} .

8) From the definition of q _i , each event em _{, k} is a set of {target device unique ID, log template ID}, which appears in the log sequence S _{m and} has a high probability of co-occurring {monitoring It can be considered as a set of target device unique ID, log template ID}. Among the series S _m , it is possible to arbitrarily select related ones. For example, the same service type and a continuous time series can be cited, but it is possible to select according to the use situation.

Step 104) The rule grasping unit 140 obtains a sequence S _m from the group storage unit 160, the selected sequence S _m event group output from e _m, with respect to _k, by doing the following rules To grasp.

All target event groups are set as {e _l : l = 1,..., E}, and a set of {monitor target device unique ID and log template ID} included in the event e _l is {q _li : i = 1 _,. , Q _l }.

  Depending on the type of rule to be grasped, the element of {monitoring target device unique ID, log template ID} is selected, and the following two kinds of rules are grasped.

Step 104 (a)) Understanding template rules:
The template rule grasping unit 141 of the rule grasping unit 140 searches for the template IDs of q _i included in the same event that are associated with the same unique ID of the monitoring target device, and sets these templates and the monitoring target. The device unique ID is stored in the template rule candidate storage unit 170. For example, if an event e _i has a configuration of {(1,10), (1,11), (1,12), (2,20)}, (1, [10, 11, 12]) Is stored in the template rule candidate storage unit 170. This is called a template rule candidate.

  Steps 105 to 108) This is repeated for all the events, and the following operations are performed for all the obtained template rule candidates, and the determined rules are output. The rule merging unit 150 compares the rules, and if there are C or more items that have a higher degree of similarity with a set of template IDs than a certain threshold and have different unique IDs of monitored devices, these are combined as a template rule. Output to the rule DB 200.

  Various methods are conceivable for comparing the similarity of a set of template IDs, for example, a Jaccard coefficient. In this case, when a set of two rule candidate templates is given by A and B, it can be calculated by 1 (A∩B) / 1 (A∪B). However, 1 (A) represents the number of IDs included in A.

  Step 107) The obtained template rate rule can be abstracted by using the monitoring information label in the abstraction unit 151 of the rule merging unit 150. Percentage of monitored device IDs in the monitored device ID list that have a common label when the template rule is expressed as {[monitored device ID list] ⇒ [template ID list]} If so, replace it with the label and replace it with the form {monitored device label⇒ [template ID list]}. If there is no label, the monitored device ID list itself can be omitted as in {* ⇒ [template ID list]}.

Step 104 (b)) Hereinafter, similarly, a method of grasping the monitoring target device rule will be described. The monitoring target device rule grasping unit 142 acquires the series S _m from the group storage unit 160 and searches for the monitoring target device IDs of q _i included in the same event that are associated with the same template ID. The template and the monitoring target device ID string are stored in the monitoring target device rule candidate storage unit 180. For example, if a certain event e ₁ is configured as {(1, 10), (2, 10)}, it is stored in the monitoring target device rule candidate storage unit 180 as {[1, 2], 10}. This is called a monitoring target device rule candidate.

  Steps 105 to 108) This is repeated for all the events, and the rule merging unit 150 compares the rules for all the obtained rule candidates, and the similarity with the set of monitored device IDs is higher than a certain threshold, and the template ID is If there are C or more different items, these are collectively output to the rule DB 200 as a monitoring target device rule.

  Step 107) Further, the obtained monitoring target device rule can be abstracted using the template label in the abstraction unit 151 in the rule merging unit 150 (Step 107). When a monitored device rule is expressed in the form {[monitored device ID list] ⇒ [template ID list]}, there are more than a certain percentage of template IDs in the template ID list that have a common label. For example, replace this with the label {[monitored device ID list] ⇒ template ID label}.

  The template rule represents a group of templates that often occur within the same time window, together with the attributes of the monitoring target device.

  The monitoring target device rule represents a group of monitoring target devices that frequently occur at the same time within the same time window for a certain type of template label.

  In the above description, the monitoring target device rule and the template rule are described, but it is also possible to create a composite rule using these rules. The template rule and the monitoring target device rule can be combined with each other by comparing the co-occurrence in all events again. In the compounding unit 152 of the rule merging unit 150, the number of simultaneous occurrences occurring in each event is calculated, and if it is C times or more, it is output as a compound rule. For example, when C = 2, if different rules r1 and r2 appear in two events, {r1, r2} is a composite rule.

  The main components of this embodiment are the log template label information DB 40 for storing log templates generated in the past and the time series of monitoring information using the template to grasp the rules. The analysis engine 100 and the rule DB 200 for performing the above, and the user interface 50 for outputting this to the user or for giving the label or the like in advance by the user. These can be implemented in the form shown in FIG. 4 or FIG. The rule grasping engine 100 corresponds to the rule creating device 100 described above.

  In FIG. 4, it is assumed that the logs collected by the log collection device 10 are stored in the log template label information DB 40 in advance. The grasped log occurrence rules can be displayed on the user terminals 1 and 2 as a digest of logs that occurred in the past.

  FIG. 5 shows a mechanism for performing clustering on a log that occurs from time to time and updating the rules sequentially. In this case, the DB (for example, the template rule candidate storage unit 170 and the monitoring target device rule candidate storage unit 180 in FIG. 2) that stores rule candidates in the previous stage to be adopted as rules, and the management engine that manages them as needed are described above. The rule grasping engine 200 corresponding to the rule creating apparatus 100 is required.

  Here, it is assumed that the log collection device 10 is prepared in advance, and the device group 3 to be monitored also has a mechanism for generating a log.

[First embodiment]
The input log can be any device, but for example, the router syslog for network monitoring can be considered. In this case, the offline form of FIG. 4 will be described. Logs generated by network devices (monitored devices) are centrally collected and stored in the log DB, and templates and monitoring device information are assigned in advance, and labels are also stored.

  The user designates the time series to be analyzed, the monitoring device group, the service type, the position, and the like from the user terminals 1 and 2. In response to this, the analysis engine (rule creation device) 100 performs clustering and rule grasping based on the simultaneous occurrence frequency of log information. These steps are all performed off-line. The output rules are provided to the user terminals 1 and 2, respectively.

[Second embodiment]
In the first embodiment, the log information is accumulated in the log template label information DB 40 in advance, and the rule grasping is applied offline. However, it can be used while updating the rule online. . Hereinafter, this usage example will be described with reference to FIG.

  Each log information sent from the log collection device 10 is grasped to which template it belongs, and at the same time, it is labeled with monitoring device information. The log information that has arrived is matched with a rule that has been grasped in the past, and it is grasped by which rule there is a log generated. On the other hand, when a new (monitoring device ID, template ID) pair that is not included in the rule DB 200 has arrived, the rule candidate DB (the template rule candidate storage unit 170 in FIG. Corresponding to candidate storage unit 180). The rule candidate DB is updated as needed, and when a new rule is completed, a new rule is added to the rule DB 200.

  Also in this case, as in the first embodiment, the user is sequentially notified through the user interface 50 which rule causes the sequentially arriving log.

  Note that the operation of each component of the rule creation device shown in FIG. 2 can be constructed as a program and installed in a computer used as the rule creation device for execution, or distributed through a network.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications are possible within the scope of the claims.

1, 2 User terminal 3 Monitored device 10 Log collection device 20 Log DB
30 Log editing device 40 Log template label information DB
50 User interface 60 Template shaping / display unit 100 Rule creation device 110 Template ID extraction unit 120 Log division unit 130 Group generation unit 140 Rule grasping unit 141 Template rule grasping unit 142 Monitored device rule grasping unit 150 Rule merging unit 151 Abstraction unit 152 Composite Unit 160 Group Storage Unit 170 Template Rule Candidate Storage Unit 180 Monitoring Target Device Rule Candidate Storage Unit 200 Rule DB

Claims (8)

A log generation rule creation device for generating a log type generation rule (rule) for a monitored device,
Reference is made to log template information storage means storing at least a log message of the monitoring target device (hereinafter referred to as “log”) and an important part (hereinafter referred to as “template”) in the log message. A template extraction means for extracting a template corresponding to the log that has occurred;
Clustering a template having high co-occurrence for each monitoring target device, generating a group of the monitoring target device and the template, and storing the group in a group storage unit as an event;
Reading each event of the group storage means, counting the number of occurrences for the monitored device and template of the event, rule grasping means for storing in the rule storage means as the occurrence rule,
A log generation rule creation device characterized by comprising: The rule grasping means is:
Among the template IDs included in the same event of the group storage means, those associated with the same monitored device ID are extracted, and the template set and the monitored device ID are used as template rule candidates. The process of storing in the storage means is repeated for all events, the template rule candidates are compared with each other for all template rule candidates stored in the template rule candidate storage means, and the similarity to the set of template IDs is greater than a predetermined threshold value. If there are more than a predetermined number of devices with different monitoring target device IDs, a template rule extracting means that collectively sets them as template rules;
Among the monitoring target device IDs included in the same event of the group storage unit, those associated with the same template ID are extracted, and the template of the template ID and the monitoring target device ID are monitored as monitoring target device rule candidates. The process of storing in the target device rule candidate is repeated for all events, and for all the target device rule candidates stored in the target device rule candidate storage unit, the target device rule candidates are compared with each other, And a monitoring target device rule extracting unit that collectively sets a monitoring target device rule when a plurality of items having a template similarity higher than a predetermined threshold and different template IDs are present. The log generation rule creation device described. The log template information storage means includes a monitoring information label,
Means for replacing the monitoring target device ID with the monitoring information label if the monitoring target device ID included in the list of monitoring target device IDs of the template rule has a predetermined ratio or more of the monitoring target device IDs. The log generation rule creation device according to claim 1 or 2, further comprising: The log template information storage means includes a template label,
The apparatus further comprises means for replacing the template ID with the template label when the template IDs included in the list of template IDs of the monitoring target device rule have a common template label or more in a predetermined ratio or more. 2. The log generation rule creation device according to 2. A log generation rule creation method for generating a log type generation rule (rule) for a monitored device,
In an apparatus having template extraction means, group generation means, rule grasping means,
Log template information storage means in which the template extraction means stores at least a log message of the monitored device (hereinafter referred to as “log”) and an important part (hereinafter referred to as “template”) in the log message. And a template extraction step for extracting a template corresponding to a log that occurred in the past,
The group generation means clusters the templates having high co-occurrence for each monitoring target device, generates a group of the monitoring target device and the template, and stores the group in the group storage means as an event Steps,
The rule grasping unit reads each event of the group storage unit, counts the number of occurrences for the monitored device and template of the event, and stores a rule having a large number of occurrences in the rule storage unit as an occurrence rule Grasp step,
A log generation rule creation method characterized by: In the rule grasping step,
Among the template IDs included in the same event of the group storage means, those associated with the same monitored device ID are extracted, and the template set and the monitored device ID are used as template rule candidates. The process of storing in the storage means is repeated for all events, the template rule candidates are compared with each other for all template rule candidates stored in the template rule candidate storage means, and the similarity to the set of template IDs is greater than a predetermined threshold value. If there is a predetermined number or more of different monitoring target device IDs, a template rule extraction step that collectively sets them as template rules;
Among the monitoring target device IDs included in the same event of the group storage unit, those associated with the same template ID are extracted, and the template of the template ID and the monitoring target device ID are monitored as monitoring target device rule candidates. The process of storing in the target device rule candidate is repeated for all events, and for all the target device rule candidates stored in the target device rule candidate storage unit, the target device rule candidates are compared with each other, If there is a predetermined number or more of the similarities with the set higher than a predetermined threshold and different template IDs, a monitoring target device rule extracting step that collectively sets them as monitoring target device rules;
The method for creating a log generation rule according to claim 5. In the rule grasping step,
When the log template information storage means includes a monitoring information label,
A step of replacing the monitoring target device ID with the monitoring information label if there is a predetermined ratio or more of the monitoring target device IDs included in the list of monitoring target device IDs of the template rule;
When the log template information storage means includes a template label,
A step of replacing the template ID with the template label if there is a predetermined ratio or more of the template IDs included in the list of template IDs of the monitoring target device rule;
Further performing either or both of the steps,
The log generation rule creation method according to claim 5 or 6. In the rule grasping step,
In the template rule and the monitoring target device rule, the number of times of simultaneous co-occurrence in all events is calculated again, and if it is equal to or more than a predetermined number, the template rule and the monitoring target device rule are combined to form a composite rule. The log generation rule creation method according to claim 5 or 6, further comprising a composite rule generation step of outputting.

Images