KR102244782B1 - Method for automatical parser matching based on nomalization rates to be used for accurately analyzing unstructured logs having arbitrary structures and device using the same - Google Patents

Abstract

A parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure is disclosed. That is, (a) when the parser matching device obtains a random analysis log list including at least one unstructured text log obtained from at least one service, the random analysis log list is referred to, and the unstructured text Obtaining policy type information corresponding to category information of each log; (b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And (d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the K-th normalization rate information. Disclosed is a method comprising:

Description

METHOD FOR AUTOMATICAL PARSER MATCHING BASED ON NOMALIZATION RATES TO BE USED FOR ACCURATELY ANALYZING UNSTRUCTURED LOGS HAVING ARBITRARY STRUCTURES AND DEVICE USING THE SAME}

The present invention relates to a normalization rate-based parser matching method and apparatus for accurately and automatically analyzing an unstructured log having an arbitrary structure.

From the standpoint of an administrator who manages many servers, the numerous logs generated by each of the servers and the equipment connected to them are one of the things that greatly perplexes the administrator. It would be fine if the structure and contents of the logs are known, but if the following unstructured logs occur, it is difficult for the administrator who needs to perform debugging based on them to know the meaning.

<UTM log>

3`0`1`1`3238c5`1020`20160927`10:45:23`3`6`120216154246`99.6.101.3`52152`99.6.1.111`8001`eth6`eth8``''1658`10`858 `8``31`1``S sa A / fa A FA a```199`N/A`4``''

<MF2 log>

<14>1 2016-10-11T02:02:00.268873Z [fw4_allow] [127.0.0.1]start_time="2016-10-11 11:01:48" end_time="2016-10-11 11:01:53" duration=5 machine_name=Saha-FW fw_rule_id=5 nat_rule_id=4 src_ip=1.223.252.190 src_port=37770 dst_ip=192.168.20.10 dst_port=80 protocol=TCP ingres_if=EXT packets_forward=5 packets_backward=4 bytes_forward=969 bytes_backward=427 fragment_info bytes_backward=427 fragment_info flag_record=3Way OK / FIN2 [SAF:SAF] user_auth=- terminate_reason=-

Therefore, such unknown atypical logs have no meaning of information by themselves, and can be regarded as having meaning as information only when analyzed as follows.

Field name UTM log normalization result MF2 normalization result Origin IP (src_ip) 99.6.101.3 1.223.252.190 Origin port (src_port) 52152 37770 Destination IP (dstn_ip) 99.6.1.111 192.168.20.10 Destination port (dstn_port) 8001 80 Protocol (prtc) - TCP

In order to analyze the above unknown unstructured logs, it is necessary to designate parsers used for text analysis by category of each unstructured log. Until now, engineers have been manually performing these assignments, but in this case, it takes a long time, and since it is a human task, there are problems that many mistakes occur. If there is an efficient method for automatically matching parsers, the above problem can be solved, but such a method seems to have not been studied much.

It is an object of the present invention to solve the above-described problems.

In addition, an object of the present invention is to provide a parser matching method based on a normalization rate for accurately and automatically analyzing unstructured logs having an arbitrary structure, so that an administrator can easily analyze unstructured logs.

In addition, the present invention provides a method for calculating the normalization rate for the result of analyzing each unstructured log so that a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure can be performed. The purpose.

A characteristic configuration of the present invention for achieving the object of the present invention as described above and realizing the characteristic effects of the present invention described later is as follows.

According to an aspect of the present invention, in a parser matching method based on a normalization rate for accurately automatically analyzing an unstructured log having an arbitrary structure, (a) a parser matching device includes at least one unstructured text obtained from at least one service. When a difficult analysis log list including logs is obtained, obtaining policy type information corresponding to category information of each of the unstructured text logs by referring to the difficult analysis log list; (b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And (d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the K-th normalization rate information. Disclosed is a method comprising:

As an example, in step (c), after the parser matching device normalizes the specific unstructured text log using the K-th candidate log parsing policies, the number of cases in which the adequacy evaluation of the normalized result is greater than or equal to a threshold. And generating the Kth normalization rate information with reference to ratio information between the number of K_1 to K_Mth log parsing policies.

As an example, in step (c), the parser matching device generates a specific normalization result, which is a result of normalizing the specific unstructured text log using the K-th specific candidate log parsing policy, which is one of the K-th candidate log parsing policies. For reference, by obtaining log data type data corresponding to each section of the specific normalization result, and comparing (i) the log data type data with (ii) policy data type data corresponding to the K-th specific candidate log parsing policy, Disclosed is a method comprising generating a specific adequacy rating for the specific normalization result, indicating whether it is appropriate to use the K-th specific candidate log parsing policy to normalize the specific unstructured text log.

As an example, in the step (b), the parser matching device, in a state in which the K_1th to K_Mth log parsing policies corresponding to the Kth log parser are classified for each policy type, the K_1th to the K_1th Disclosed is a method comprising selecting the candidate log parsing policy including the K-th candidate log parsing policy from among the K_M log parsing policies corresponding to the specific policy type information.

As an example, (e) in a state in which the parser matching device determines whether each of the first to Nth log parsers is included in the specific log parser, the user can provide the specific log parser to the user. Disclosed is a method comprising the step of supporting at least one of the specific log parsers to be used for the unstructured text log for testing included in the specific category information corresponding to the specific unstructured text log.

As an example, (f) the parser matching device determines whether or not each of the first to Nth log parsers is included in the specific log parser, and the normalization rate information is the largest among the specific log parsers. The method further comprises the step of designating specific category information corresponding to the specific log unstructured text log and adding it to an analysis available log list.

As an example, (g) when the parser matching device obtains a test unstructured text log, analyzes it to obtain test category information of the test unstructured text log, and then obtains the test category information and the analysis available log. Select a predetermined log parser to be used to normalize the test unstructured text log with reference to the list, but if the similarity between the test category information and the specific category information is greater than or equal to a threshold, the normalization rate information among the specific log parsers The method is disclosed, further comprising the step of using the largest one to normalize the test unstructured text log.

According to another aspect of the present invention, there is provided a parser matching device for performing a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure, comprising: at least one memory for storing instructions; And one or more processors configured to perform the instructions, wherein the processor includes: (I) when an egg analysis log list including at least one unstructured text log obtained from at least one service is obtained, the egg A process of obtaining policy type information corresponding to category information of each of the unstructured text logs with reference to an analysis log list; (II) With reference to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, N log parsers-N is an integer greater than or equal to 1-One of the K-th log parsers-K is 1 or more A process of selecting a K-th candidate log parsing policy that is at least a part of an integer of N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (III) a process of generating K-th normalization rate information for the specific unstructured text log of the K-th log parser with reference to the information on the K-th candidate log parsing policy; And (IV) performing a process of determining whether the Kth log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the Kth normalization rate information. The device is started.

As an example, in the (III) process, after the processor normalizes the specific unstructured text log using the K-th candidate log parsing policies, the number of cases where the adequacy evaluation of the normalized result is greater than or equal to a threshold value and the Disclosed is a device, characterized in that the K-th normalization rate information is generated with reference to ratio information between the number of K_1 to K_M-th log parsing policies.

As an example, the process (III) refers to a specific normalization result, which is a result of normalizing the specific unstructured text log using the K-th specific candidate log parsing policy, which is one of the K-th candidate log parsing policies. Thus, by obtaining log data type data corresponding to each section of the specific normalization result, and comparing (i) the log data type data with (ii) policy data type data corresponding to the K-th specific candidate log parsing policy, A device is disclosed for generating a specific adequacy rating for the specific normalization result indicating whether it is appropriate to use the K-th specific candidate log parsing policy to normalize an unstructured text log.

As an example, in the (II) process, in a state in which the K_1 to K_Mth log parsing policies corresponding to the Kth log parser are classified for each policy type, the K_1 to K_M log Disclosed is a device, wherein the candidate log parsing policy including the K-th candidate log parsing policy is acquired by acquiring the K-th candidate log parsing policy corresponding to the specific policy type information from among the parsing policies.

As an example, the processor provides the specific log parser to the user in a state in which (V) it is determined whether each of the first to Nth log parsers is included in the specific log parser, Disclosed is a device, characterized in that the device further performs a process of supporting the use of at least one of the parsers for a test unstructured text log included in specific category information corresponding to the specific unstructured text log.

As an example, in a state in which it is determined whether or not each of the first to Nth log parsers is included in the specific log parser, the processor may determine that the normalization rate information is the largest among the specific log parsers. Disclosed is a device, characterized in that, after designating specific category information corresponding to a log unstructured text log, a process of adding it to an analysis available log list is performed.

As an example, the processor, when (VII) a test unstructured text log is obtained, analyzes it to obtain test category information of the test unstructured text log, and then provides the test category information and the analysis log list. For reference, a predetermined log parser to be used to normalize the test unstructured text log is selected, but when the similarity between the test category information and the specific category information is greater than or equal to a threshold, the normalization rate information among the specific log parsers is the most. A device is disclosed, characterized by further performing a process of using a large one to normalize the test unstructured text log.

The present invention provides a parser matching method based on a normalization rate for accurately and automatically analyzing unstructured logs having an arbitrary structure, thereby enabling an administrator to easily analyze unstructured logs.

In addition, the present invention can provide a method for calculating a normalization rate for a result of analyzing each unstructured log so that a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure can be performed. There is an effect.

1 is a diagram showing a configuration of a parser matching device that performs a parser matching method based on a normalization rate for accurately automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention.
2A and 2B are one of first to Nth log parsers used to perform a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention. A diagram showing the configuration of a K log parser.
3 is a flowchart illustrating a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention.
4A and 4B illustrate an example of supporting a user to use an appropriate log parser by performing a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention. It is a drawing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily implement the present invention.

1 is a diagram showing a configuration of a parser matching device that performs a parser matching method based on a normalization rate for accurately automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention.

Referring to FIG. 1, the parser matching device 100 may include first to Nth log parsers 130-1 to 130 -N. In this case, input/output and operation processes of the first to Nth log parsers 130-1 to 130 -N may be performed by the communication unit 110 and the processor 120, respectively. However, in FIG. 1, a specific connection relationship between the communication unit 110 and the processor 120 is omitted. In addition, the memory 115 may be in a state in which various instructions to be described later are stored, and the processor 120 may execute the instructions stored in the memory, thereby performing processes to be described later to perform the present invention. Just because the parser matching device 100 is described in this way does not exclude the case where the parser matching device 100 includes an integrated processor in which a medium, a processor, and a memory are integrated for carrying out the present invention. As the configuration of the parser matching device 100 has been described above, each of the first to Nth log parsers 130-1 to 130 -N will be described with reference to FIGS. 2A and 2B.

2A and 2B are one of first to Nth log parsers used to perform a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention. A diagram showing the configuration of a K log parser.

First, referring to FIG. 2A, the K-th log parser 130-K (K is an integer of 1 or more and N or less), which is one of the first to Nth log parsers 130-1 to 130-N, is It can be seen that the K_1 to K_Mth log parsing policies (M is an integer greater than or equal to 1) are included. Each of the K_1 to K_Mth log parsing policies may mean respective parsing rules that can be used when the Kth log parser 130 -K performs a parsing operation on logs. The K_1th to K_Mth log parsing policies may be classified according to categories, and FIG. 2B is referred to in order to examine them.

Referring to FIG. 2B, as an example, it can be seen that Security, Traffic, System, and SMS categories exist under the LogGenerator parser, and log parsing policies such as ncac_esa_ids exist under the LogGenerator parser. Here, Security, Traffic, System, and SMS are categories of each log parsing policy, and those located below them may be log parsing policies included in each category.

As described above, the parser matching device 100 and the first to Nth log parsers 130-1 to perform a parser matching method based on a normalization rate for accurately automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention. 130-N) has been described. Hereinafter, a parser matching method will be examined in earnest with reference to FIG. 3.

3 is a flowchart illustrating a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention.

Referring to FIG. 3, first, when the parser matching device 100 obtains an egg analysis log list including at least one unstructured text log obtained from at least one service, the random analysis log list is referred to. Thus, policy type information corresponding to category information of each unstructured text log may be obtained (S01). Thereafter, the parser matching device 100 refers to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, and is one of the first to Nth log parsers (N is an integer greater than or equal to 1). A K-th candidate log parsing policy that is at least a part of K_1 to K_M-th log parsing policies (M is an integer of 1 or more) corresponding to the K-th log parser (K is an integer of 1 or more and N or less) may be selected (S02). In addition, the parser matching device 100 may generate Kth normalization rate information for a specific unstructured text log of the Kth log parser with reference to information on the Kth candidate log parsing policy (S03). Next, the parser matching device 100 may determine whether or not the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the K-th normalization rate information ( S04).

Hereinafter, each step will be described in detail.

First, the parser matching device 100 may obtain an egg analysis log list including at least one unstructured text log obtained from at least one service. Here, each of the services may be performed by respective servers and equipment operating in conjunction with the parser matching device 100. In addition, the unstructured text log is generated while each server and equipment performs services, and contains the information necessary for the engineer to manage and maintain the services, but the structure has not been analyzed, so the engineer It may be in a difficult state to obtain easily.

In addition, the random analysis log list may be a data set including such unstructured text logs, and may be in the form of a list, or may have a form of any other data structure. Each unstructured text log included in the unstructured log list is not easy to be analyzed, but it is possible to record which service each was created while executing, so each unstructured text log contains their category information. May be in a tagged state. For example, some unstructured text logs may be generated from a security server and tagged with security category information, while other unstructured logs are generated from a messenger server that performs a messenger service between users and tagged SMS category information. I can. The parser matching device 100 may obtain policy type information corresponding to category information of each of the unstructured text logs by referring to a difficult analysis log list including such information.

Thereafter, the parser matching device 100 refers to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, and K_1 to K_Mth respectively corresponding to the Kth log parser 130-K. The K-th candidate log parsing policy, which is at least a part of the log parsing policy, may be selected. Specifically, in a state in which the parser matching device 100 has the Kth log parsing policy corresponding to the Kth log parser 130 -K classified for each policy type, a specific policy type among the Kth log parsing policies The candidate log parsing policy including the K-th candidate log parsing policy may be selected by acquiring the K-th candidate log parsing policy corresponding to the information. For example, if a specific unstructured text log is a log included in the security category, specific policy type information will correspond to the security type. In this situation, the parser matching device 100 A K-th candidate log parsing policy corresponding to the security category may be selected from among the K_1 to K_M-th log parsing policies.

Thereafter, the parser matching device 100 may generate Kth normalization rate information for a specific unstructured text log of the Kth log parser 130 -K with reference to the information on the Kth candidate log parsing policy. Specifically, after the parser matching device 100 normalizes a specific unstructured text log using the K-th candidate log parsing policies, the number of cases where the adequacy evaluation degree of the normalized result is equal to or greater than a threshold value and the K_1th to K_Mth The Kth normalization rate information may be generated with reference to information on the ratio between the number of log parsing policies.

For example, as described above, when a specific unstructured text log is included in the security category, and accordingly, the K-th candidate log parsing policy also falls under the security category, the K-th candidate log parsing policy is used to normalize a specific unstructured text log. After that, by evaluating the normalized results using each K-th candidate log parsing policy, the adequacy evaluation degree is calculated, and the ratio of cases where they are greater than or equal to the threshold value is calculated, thereby generating the K-th normalization rate information. As an example, as in the LogGenerator parser of FIG. 2A, if there are 5 out of 8 K-th candidate log parsing policies corresponding to the security category, and 3 of them have an adequacy rating exceeding the threshold, the K-th normalization rate information is It can be calculated as 3/8. Of course, it is not limited thereto, and of course, the K-th normalization rate information may be calculated as 3/5 in some cases.

Here, an example of calculating the adequacy evaluation will be described. That is, the parser matching device 100 refers to a specific normalization result, which is a result of normalizing a specific unstructured text log using the K-th specific candidate log parsing policy, which is one of the K-th candidate log parsing policies, By obtaining log data type data corresponding to each section, and comparing (i) log data type data with (ii) policy data type data corresponding to the K-th specific candidate log parsing policy, to normalize a specific unstructured text log It is possible to generate a specific adequacy evaluation degree for the specific normalization result indicating whether it is appropriate to use the candidate log parsing policy. For example, assume that an IP address is included in the second section of the specific normalization result, and the K-th specific candidate log parsing policy is a parsing policy that assumes that the port number information is included in the second section. In this case, the log data type data may include a first indication indicating that there will be four three-digit numbers separated by periods in the second section. Since an integer value may be included, a second indication indicating this may be included. In this case, since the first indication and the second indication will be different from each other, a specific adequacy evaluation degree for the K-th specific candidate log parsing policy may be calculated below a threshold.

This method of calculating the adequacy evaluation is only an example, and any other method may be used as a part of the present invention if each log parsing policy provides a measure to determine whether or not the unstructured text log is well normalized. . For example, if the number of sections of a specific unstructured text log and the number of sections assumed by the K-th specific candidate log parsing policy are the same, it may be determined that normalization is well performed.

When the K-th normalization rate information is generated in this way, the parser matching device 100 refers to the K-th normalization rate information, and the K-th log parser 130-K is used to analyze a specific unstructured text log. You will be able to determine whether or not to be included in the log parser. That is, after generating first to Nth normalization rate information for each of the first to Nth log parsers 130-1 to 130-N, respectively, a specific log to be used to analyze a specific unstructured text log The parser can be determined.

Thereafter, the parser matching device 100 provides a specific log parser to the user so that the user can use at least one of the specific log parsers for a test unstructured text log included in specific category information corresponding to a specific unstructured text log. You can apply. To describe this, refer to FIGS. 4A and 4B.

4A and 4B illustrate an example of supporting a user to use an appropriate log parser by performing a parser matching method based on a normalization rate for accurately and automatically analyzing an unstructured log having an arbitrary structure according to an embodiment of the present invention. It is a drawing.

Referring to FIG. 4A, a user may parse a log parser to be used for category information corresponding to a corresponding device type and receive recommendations from the matching device 100 by clicking a recommendation request 200 button in a predetermined device management tab.

In addition, referring to FIG. 4B, as an example, it can be seen that through the above-described process, the LogGenerator parser and the Security parser were selected as specific log parsers, and their normalization rates were calculated as 1/8 and 2/9 in the table, respectively. have. In this case, the user can more easily analyze the category information by selecting a security parser with a higher normalization rate.

Alternatively, you can fully automate the log parser selection process for specific unstructured text logs. That is, the parser matching device 100 may designate a specific log parser having the largest normalization rate information for the specific category information corresponding to the specific log unstructured text log, and then add it to the list of analyzeable logs. If described according to the above-described example, the parser matching device 100 may select a security parser and designate it for specific category information.

After parser matching is completed in this way, the parser matching device 100, when the test unstructured text log is obtained, analyzes it to obtain the test category information of the test unstructured text log, and then, the test category information and analysis are possible. You can select a predetermined log parser to use to normalize the unstructured text log for testing by referring to the log list. Here, if the similarity between the test category information and the specific category information is greater than or equal to the threshold value, the largest normalization rate information among the specific log parsers may be used to normalize the test unstructured text log.

Through this process, even if an unstructured text log whose structure is unknown is obtained, the user can easily analyze it.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Anyone with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from such a machine.

Accordingly, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all modifications equivalently or equivalently to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. I would say.

Claims (14)

In the normalization rate-based parser matching method for accurately and automatically analyzing an unstructured log having an arbitrary structure,
(a) When the parser matching device obtains a random analysis log list including at least one unstructured text log obtained from at least one service, each of the unstructured text logs with reference to the random analysis log list Obtaining policy type information corresponding to the category information of;
(b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more;
(c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And
(d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the K-th normalization rate information
Including,
The step (c),
After the parser matching device normalizes the specific unstructured text log using the K-th candidate log parsing policies, the number of cases where the adequacy evaluation degree of the normalized result is equal to or greater than a threshold value and the K_1 to K_Mth log parsing policies And generating the K-th normalization rate information with reference to the ratio information between the number of units. delete The method of claim 1,
The step (c),
The parser matching device refers to a specific normalization result, which is a result of normalizing the specific unstructured text log, using a K-th specific candidate log parsing policy, which is one of the K-th candidate log parsing policies, and each of the specific normalization results. To normalize the specific unstructured text log by obtaining log data type data corresponding to the section, and comparing (i) the log data type data with (ii) policy data type data corresponding to the K-th specific candidate log parsing policy. And generating a specific adequacy rating for the specific normalization result indicating whether it is appropriate to use the K-th specific candidate log parsing policy. In the normalization rate-based parser matching method for accurately and automatically analyzing an unstructured log having an arbitrary structure,
(a) When the parser matching device obtains a random analysis log list including at least one unstructured text log obtained from at least one service, each of the unstructured text logs with reference to the random analysis log list Obtaining policy type information corresponding to the category information of;
(b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more;
(c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And
(d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the K-th normalization rate information
Including,
The step (b),
When the parser matching device is in a state in which the K_1 to K_Mth log parsing policies corresponding to the Kth log parser are classified for each policy type, the specific policy type information among the K_1 to K_Mth log parsing policies And selecting the candidate log parsing policy including the K-th candidate log parsing policy corresponding to the K-th candidate log parsing policy. In the normalization rate-based parser matching method for accurately and automatically analyzing an unstructured log having an arbitrary structure,
(a) When the parser matching device obtains a random analysis log list including at least one unstructured text log obtained from at least one service, each of the unstructured text logs with reference to the random analysis log list Obtaining policy type information corresponding to the category information of;
(b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more;
(c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And
(d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the K-th normalization rate information
Including,
(e) The parser matching device provides the specific log parser to the user in a state in which it is determined whether each of the first to Nth log parsers is included in the specific log parser, so that the user can use the specific log parser. Supporting at least one of the unstructured text logs for testing included in specific category information corresponding to the specific unstructured text log
The method further comprising a. In the normalization rate-based parser matching method for accurately and automatically analyzing an unstructured log having an arbitrary structure,
(a) When the parser matching device obtains a random analysis log list including at least one unstructured text log obtained from at least one service, each of the unstructured text logs with reference to the random analysis log list Obtaining policy type information corresponding to the category information of;
(b) The parser matching device refers to the specific policy type information corresponding to the specific unstructured text log, which is one of the unstructured text logs, and is the Kth log of one of N log parsers-N is an integer greater than or equal to 1- Selecting a K-th candidate log parsing policy that is at least a part of the parser-K is an integer of 1 or more and N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more;
(c) generating, by the parser matching device, Kth normalization rate information for the specific unstructured text log of the Kth log parser with reference to the information on the Kth candidate log parsing policy; And
(d) determining, by the parser matching device, whether the K-th log parser is included in a specific log parser to be used to analyze the specific unstructured text log with reference to the K-th normalization rate information
Including,
(f) In a state in which the parser matching device determines whether each of the first to Nth log parsers is included in the specific log parser, it is determined that the normalization rate information is the largest among the specific log parsers. Specifying specific category information corresponding to the text log and adding it to the list of logs that can be analyzed
The method further comprising a. The method of claim 6,
(g) When the parser matching device obtains a test unstructured text log, it analyzes it to obtain test category information of the test unstructured text log, and then refers to the test category information and the analysis log list. To select a predetermined log parser to be used to normalize the test unstructured text log,
When the similarity between the test category information and the specific category information is greater than or equal to a threshold, using the largest normalization rate information among the specific log parsers to normalize the test unstructured text log
The method further comprising a. In a parser matching device that performs a parser matching method based on a normalization rate to accurately and automatically analyze an unstructured log having an arbitrary structure,
One or more memories for storing instructions; And
Including one or more processors configured to perform the instructions, wherein the processor is, (I) when a random analysis log list including at least one unstructured text log obtained from at least one service is obtained, the random analysis A process of obtaining policy type information corresponding to category information of each of the unstructured text logs with reference to a log list; (II) With reference to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, N log parsers-N is an integer greater than or equal to 1-One of the K-th log parsers-K is 1 or more A process of selecting a K-th candidate log parsing policy that is at least a part of an integer of N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (III) a process of generating K-th normalization rate information for the specific unstructured text log of the K-th log parser with reference to the information on the K-th candidate log parsing policy; And (IV) performing a process of determining whether or not the Kth log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the Kth normalization rate information,
The (III) process,
After the processor normalizes the specific unstructured text log using the K-th candidate log parsing policies, the number of cases where the adequacy evaluation degree of the normalized result is equal to or greater than a threshold value and the number of K_1 to K_M-th log parsing policies And generating the K-th normalization rate information with reference to the ratio information of the liver. delete The method of claim 8,
The (III) process,
The processor refers to a specific normalization result, which is a result of normalizing the specific unstructured text log, using a K-th specific candidate log parsing policy, which is one of the K-th candidate log parsing policies, in each section of the specific normalization result. The K-th to normalize the specific unstructured text log by obtaining corresponding log data type data, and comparing (i) the log data type data with (ii) policy data type data corresponding to the K-th specific candidate log parsing policy. A device for generating a specific adequacy evaluation degree for the specific normalization result, indicating whether it is appropriate to use a specific candidate log parsing policy. In a parser matching device that performs a parser matching method based on a normalization rate to accurately and automatically analyze an unstructured log having an arbitrary structure,
One or more memories for storing instructions; And
Including one or more processors configured to perform the instructions, wherein the processor is, (I) when a random analysis log list including at least one unstructured text log obtained from at least one service is obtained, the random analysis A process of obtaining policy type information corresponding to category information of each of the unstructured text logs with reference to a log list; (II) With reference to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, N log parsers-N is an integer greater than or equal to 1-One of the K-th log parsers-K is 1 or more A process of selecting a K-th candidate log parsing policy that is at least a part of an integer of N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (III) a process of generating K-th normalization rate information for the specific unstructured text log of the K-th log parser with reference to the information on the K-th candidate log parsing policy; And (IV) performing a process of determining whether or not the Kth log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the Kth normalization rate information,
The (II) process,
In a state in which the K_1 to K_Mth log parsing policies corresponding to the Kth log parser are classified for each policy type, the processor corresponds to the specific policy type information among the K_1 to K_Mth log parsing policies. And selecting the candidate log parsing policy including the K-th candidate log parsing policy. In a parser matching device that performs a parser matching method based on a normalization rate to accurately and automatically analyze an unstructured log having an arbitrary structure,
One or more memories for storing instructions; And
Including one or more processors configured to perform the instructions, wherein the processor is, (I) when a random analysis log list including at least one unstructured text log obtained from at least one service is obtained, the random analysis A process of obtaining policy type information corresponding to category information of each of the unstructured text logs with reference to a log list; (II) With reference to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, N log parsers-N is an integer greater than or equal to 1-One of the K-th log parsers-K is 1 or more A process of selecting a K-th candidate log parsing policy that is at least a part of an integer of N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (III) a process of generating K-th normalization rate information for the specific unstructured text log of the K-th log parser with reference to the information on the K-th candidate log parsing policy; And (IV) performing a process of determining whether or not the Kth log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the Kth normalization rate information,
The processor,
(V) In a state in which it is determined whether each of the first to Nth log parsers is included in the specific log parser, by providing the specific log parser to the user, the user can specify at least one of the specific log parsers. A device, further comprising: performing a process of supporting the use of the unstructured text log for testing included in specific category information corresponding to the unstructured text log. In a parser matching device that performs a parser matching method based on a normalization rate to accurately and automatically analyze an unstructured log having an arbitrary structure,
One or more memories for storing instructions; And
Including one or more processors configured to perform the instructions, wherein the processor is, (I) when a random analysis log list including at least one unstructured text log obtained from at least one service is obtained, the random analysis A process of obtaining policy type information corresponding to category information of each of the unstructured text logs with reference to a log list; (II) With reference to specific policy type information corresponding to a specific unstructured text log, which is one of the unstructured text logs, N log parsers-N is an integer greater than or equal to 1-One of the K-th log parsers-K is 1 or more A process of selecting a K-th candidate log parsing policy that is at least a part of an integer of N or less-K_1 to K_Mth log parsing policies respectively corresponding to-M is an integer of 1 or more; (III) a process of generating K-th normalization rate information for the specific unstructured text log of the K-th log parser with reference to the information on the K-th candidate log parsing policy; And (IV) performing a process of determining whether or not the Kth log parser is included in a specific log parser to be used to analyze the specific unstructured text log, with reference to the Kth normalization rate information,
The processor,
(VI) In a state in which it is determined whether each of the first to Nth log parsers is included in the specific log parser, it is determined that the normalization rate information is the largest among the specific log parsers. The process of specifying category information and adding it to the list of available logs for analysis
A device, characterized in that to further perform. The method of claim 13,
The processor,
(VII) When an unstructured text log for test is obtained, analyze it to obtain test category information of the test unstructured text log, and then refer to the test category information and the list of analyzeable logs, and refer to the test unstructured text log. Select a certain log parser to use to normalize the text log,
When the similarity between the test category information and the specific category information is greater than or equal to a threshold, the process of using the largest normalization rate information among the specific log parsers to normalize the test unstructured text log
A device, characterized in that to further perform.

Images