GB2440359A

GB2440359A - Rules for data traffic assembled in single data structure

Info

Publication number: GB2440359A
Application number: GB0614335A
Authority: GB
Inventors: Stephen Robinson
Original assignee: Chronicle Solutions UK Ltd
Current assignee: Chronicle Solutions UK Ltd
Priority date: 2006-07-19
Filing date: 2006-07-19
Publication date: 2008-01-30
Also published as: WO2008009990A1; GB0614335D0

Abstract

A single data structure is assembled from a plurality of rules, the rules being for the treatment of objects in electronic data traffic. Each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value. The single data structure is arranged such that at least some of the component parts are shared between rules. The merging of rules into a more unified data structure, in which common sub-expressions are combined or eliminated, speeds up checking of rules. A new rule is added by comparing the terms and values of the new rule to terms and values of rules already stored. If a term of the new rule is already stored, but the value of that term is not stored, the value is then stored in the structure with a link to the already stored term.

Description

SYSTEM

The present invention relates to a system for monitoring electronic data traffic, and particularly to a system which watches for and alerts a network administrator to specific data traffic on the computer network. For example an administrator can define rules to instruct the system to watch for particular content and actions can be triggered when matching content is found.

An example of a known such alerting system is found in many e-mail clients. An administrator, or a user, can create logical rules from a set of predefined rules which can be matched against incoming c-mails. If a match to the rule is found in an e-mail then a pre-selected action is performed automatically by the e-mail client. For example, a rule may be created in which any c-mails with the subject heading containing the word "confidential" are automatically moved to a folder labelled "private" in the e-mail client.

Multiple rules may be defined for a system, each rule having a combination of predefined terms and resulting actions. If multiple rules are defined then each rule must be tested against each incoming object.

A problem with known such alerting systems occurs when the number of defined rules becomes large. The more defined rules there are, the longer it takes to test the rule set against each incoming object. This is particularly problematic when the testing is performed in systems with a high rate of incoming objects, for example when testing rules against network traffic.

The present invention seeks to provide a system for taking all of the alert rules defined by the administrator and merging them into a single combined rule represented by a data structure that allow's those rules to be more efficiently and quickly matched against incoming content in each object.

Accordtng to a first aspect ot the present invention there is provided a computer system for assembling a single electronic data structure from a plurality of logical rules for the treatment of objects in electronic data traffic, wherein each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value; means for user input of a plurality of said logical rules; means for dividing each rule into its sub-expressions and dividing each sub-expression into its component parts; a term database fr storing the terms of the rules with links to each of the other component parts of the sub-expression of the rule; means for comparing a term of a newly input rule with the terms stored in the term database, and if said term is not found in the term database then storing said term with a link to each of the other component parts ol' the related sub- expression of the newly input rule; and if said term is found in the term database, then following the stored links to the first of the previous comparisons associated with the stored term.

comparing the associated stored value of the comparison with the value of the term in the newly input rule; if the value is stored then continuing to the next sub-expression; if the value is not stored then storing the value as a link to said term: then continuing for the next sub-expression of the newly input rule.

The rule set state for an object being tested may be stored using an array of tn-state values (unknown. true or false) wherein one value corresponds with each node in the data structure and one value corresponds to each comparison. Two bits per node is preferably used for storage.

According to a second aspect of the present invention there is provided a computer system for automatically processing logical rules governing the treatment of objects in electronic data traffic, the system comprising: means for combining a plurality of user input logical rules into a single data structure comprising a term database, a comparison operator database and a rule graph; means for intercepting data traffic comprising a plurality of said objects; means for scanning the intercepted traffic and extracting one or more terms from said objects; means for comparing an intercepted term with terms stored in the term database; if the term does not find a match, then stopping; if the term does find a match, then following stored links associated with the term to each comparison that includes the term and checking the state of the comparison; if the state is known then stopping; if the state is not known then evaluating the comparison, and triggering the associated rule if the comparison is true.

Preferably the object state is updated to reflect whether the term evaluated to true or false, and the parents of each node are evaluated in turn. Advantageously means are provided for triggering a particular rule if a node in the data structure (rule set) is evaluated to true and the node is the head node for that rule.

Action may be taken when a rule is triggered such as details being logged, e-mail alerts being sent, running command line programs and/or adjusting the logging level for the object.

According to a third aspect of the invention there is provided a data structure for processing a plurality of logical rules for the treatment of objects in electronic data traffic, wherein each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value; wherein the data structure takes the form of' a rule graph in which any one term appears only once.

and any sub-expression occurs only once.

Corresponding methods, apparatus, computer programs and other associated aspects are also provided as can be envisaged by a person skilled in the art.

The invention will now be described, by way of example only, with reference to the accompanying drawing in which: Figure 1 is a schematic illustration of one embodiment of a rule graph used in the present invention; and Figure 2 is a flow diagram illustrating a sequence of operations of a system according to one aspect of the invention, used to evaluate a received term: and Figure 3 is a flow diagram illustrating a sequence of operations of one step of figure 2 (the step evaluate graph").

In a computer network an administrator will define rules which are used to trigger actions when matching content is captured from data traffic on the network. These rules will typically monitor client and server details such as username. IP address and MAC address, the type of the objects content, the size of the object, the protocol used to transport it, the direction (upload or download), and data information regarding the transfer of the object. In addition rules would typically monitor keywords within the objects content, and its check sum (e.g. the MD4 check sum). Other properties dependent on the content type or protocol, can also be included and also the ID of a previous alert that was triggered. However this is not an exhaustive list.

When a rule is matched then various actions may he taken by the system, such as logging the access and object details that triggered the alert rule into a database for later review by the administrator. Alternatively or in addition the system might notify the administrator of the details e.g. by e-mail, SMS or MSN. It could automatically take a particular action such as blocking the access immediately to prevent the object being transferred, or redirect it to a predefined resource such as a quarantine area.

Alternatively the system could inject predefined data into the communication channel.

for example to allow the access but accompany it with a warning message to the user.

Evidently a large number of rules will usually be generated on a typical network and checking each rule against each data transmission in the network utilises network computing capacity and memory and can slow the speed of the network.

The system of the present invention can speed up the process of checking the rules because it provides for effectively merging the rules into a more unified data structure in which common sub-expressions are combined or eliminated.

The unified data structure compiled from the administrators rules contains three main parts: * Terms -a collection of all the terms present in the complete merged rule set.

* Comparisons -a collection of all the comparisons in the complete merged rule set.

* Rule Graph -a graph representing the logic that joins the comparisons together.

Figure 1 shows a shows a schematic illustration of a rule graph for three rules: * Rulel:DstJP=192l68.l.l * Rule 2: SrcIP 192.168.1.1 & Type = Off & Protocol = FTP * Rule 3: SrcIP = 192.168.1.1 & Type = GIF & Protocol = I-ITTP The terms of the rules comprise the nodes: DstlP (Destination IP address). SrcIP (Source IP address), Type and Protocol. The comparisons comprise the node denoted by "=" and the remainder of the nodes form the rule graph.

The rule graph can be constructed using techniques used to create compilers and interpreters and will be familiar to persons skilled in the art. Tools such as lex. yacc, flex and bison can be employed to transform the text of the rules into the graph.

These tools are adapted so that with each additional rule which is added by the administrator, the system does not start from the beginning again but instead continues where it finished with the previous rule. In this way the single large graph can be generated containing all the rules instead of lots of different graphs. The assembly of such a rule graph is described in more detail below.

Once the data structure and rule graph have been established for an existing set of rules it will be used in monitoring accesses and content in network traffic. The steps in this procedure are illustrated in figures 2 and 3.

First, each term in received content or accesses is looked up in the term collection. If the term is not found, then the rule set does not contain the term and therefore that term cannot cause a rule to be matched. If the term is not found then there is no need to waste processing time evaluating the term and therefore the processing stops, as indicated by the terminator box "done".

If the term is found then ii means that one or more comparisons contain the term.

Each term contains a list of links to each comparison that includes that term so it is possible to very efficiently locate the comparisons which need to be evaluated. l'he state of the first comparison in the list is checked to see if it already has a value other than unknown'. If the comparison does have a value then the processing of that comparison stops. and the system proceeds to the next comparison until all comparisons have been processed.

If, on the other hand, the comparison does not have a value (value is unknown) the term is evaluated to either true' or false' and the object state is updated accordingly.

At this stage the rule graph is evaluated according to the sequence of steps illustrated in figure 3. Each of the parent nodes in the graph arc evaluated against the new term value. As each node is evaluated, its parent node is then also evaluated, and this process repeats until the head node for the original rule has been evaluated. The head nodes represent the original rules as created by an administrator, therefore if a head node is evaluated as true' then the corresponding rule is triggered.

As described below it is also necessary to evaluate whether other child nodes can affect the outcome and if so then the state of other child nodes and sub-child nodes are set to false until the graph branches.

When a node in the graph is evaluated, an additional action can be taken depending on the type of the node. For example. assume a rule reads: SrcIP = 192.168.1.0/24 I DstlP = 192.168.1.0/24 (meaning if the source OR destination address is 192.168. 1.0/24).

If either side of this rule is evaluated to true then there is no need to evaluate the other side of the rule as the OR will be true regardless. Similarly, if the rule reads: SrcIP 192.168.1.0/24 & DstIP = 10.0.0.0/8 (meaning if the source address is 192.168.1.0/24 AND destination address is 10.0.0.0/8).

Then either side evaluating to false would mean that there is no need to evaluate the other side of the rule. The alerting system takes advantage of this fact and when it decides that there is no need to evaluate one side of a node it will mark all child nodes (as far as it can) on that side to either true or false so that if the system receives a value for one of these terms it will not evaluate it, knowing in advance that its value is irrelevant to the final outcome.

The current evaluation state of the rule set for each access being monitored is maintained by the alerting system using an array of tn-state values (unknown, true or false), one for each node in the graph and for each comparison. This array uses two bits per node to reduce the overhead in tracking this state and that combined with the common sub-expression elimination allows a very large rule set to be used before memory consumption becomes a problem.

A further complication while processing the rule set is that many of the terms can have more than one value. For example the keyword term. most content includes hundreds, if not thousands of words so when a rule wants to know when the keyword is abc' it is important to ignore cases where the comparison evaluates to false because it could evaluate to true later on.

The keyword term is a special case in another way. It is not desirable to have the alerter informed about the value of the keyword term for every word in every item of content monitored. Thereibre, the alerting system has a mechanism to inform the content source which words it's interested in. This way the alert rules are only evaluated for words which are included in the rule set. The alerting system calls these terms External terms' because the comparison is done outside the alerting system.

An administrator may define alert rules such that more than one may trigger for a given access or object. For this reason each rule can be given a precedence value. This value is an integer, lower values have a higher precedence and higher values have a lower precedence. An administrator is then able to define which rule should take effect if more than one nile is triggered by a single access or object.

When an alert rule is submitted to the alerting system, that rule contains one or more actions that should be taken if and when the rule is triggered. These actions are submitted to the alerting system in the form of pointers to either an access object, which defines the access that triggered the alert, or to a layer object which defines the particular object within the access that caused the alert to he triggered. There are several categories of alert actions, including "Log action", "e-mail action", "Command action" and "log-level action", as described below.

A log action simply logs the details of the triggered alert to a database. The log action enters a single record in the database each time it is run.

An email action causes an e-mail to be sent to the recipient specified by the alert rule whenever it is run. There are two types of e-mail which can be sent by this action: full descriptions and summaries. Full Descriptions are sent when the quantity of alerts are not hitting the rate limit. One e-mail is sent per alert and each notification to an administrator includes a full description of the alert that has just fired. Summaries are sent when the quantity of alerts are hitting the rate limit. The number of times each alert is triggered is remembered by the action and one e-mail is sent whenever the rate limit allows. This e-mail contains a summary of the alerts that have been triggered since the last notification. The format of the bodies and subjects of these c-mails can be hard coded into the email action for speed of processing, or alternatively, an administrator may specify the content of both the subject and the body by using templates. To minimise the processing requirements of the action, if the e-mail fails to be sent it should not be retried. An entry can he placed in the main log file so an administrator can see why the c-mails are not being sent.

The command action causes a command line program to be run whenever the action is run. The parameter specified in the alert rule defines the program that should be run and the parameters that should be passed to that program. The parameters are passed by string substitution within the parameter string supplied by the alert rule.

The log-level action causes the logging level to be adjusted for the access that triggered the alert. The parameter specified by the alert rule defines the new logging level. The log-level action makes use of the rule precedence to decide which rule applies when more than one nile matches a given access. Each rule has a unique precedence. this precedence is an integer between 0 and OxFFFFFFFF, the lower the number the higher the precedence. The log level action also has a stored default logging level, which is used if no alert rule is triggered for the object or access. The log-level action always starts by using the default logging level with a precedence of OxFFFFFFFF. Whenever a rule triggers the precedence of that rule is examined. If the precedence is higher (a lower value) then the current logging level is changed to that specified by the rule. If the precedence is lower (a higher value) then it is ignored. As it is not possible to know in advance which alert rules will trigger it is not possible to know when the final logging level is reached until the object or access has completed.

Some rules may only include comparisons of properties that are global to all objects within an access, for example if the rule only references addresses or the protocol used to transport the objects. Other rules may only reference properties specific to one object within the access, for example the object's type. Still others may reference both properties global to the access and properties specific to one or more objects within that access. It is necessary to have a separate logging level for each object within an access as the administrator may wish to log different amounts of data for different object types. For example the administrator may wish to log all data for word documents, but only log the first 100kb of an MP3 file and not log any CSS stylesheets. Therefore if a rule triggers for an access and not for a specific object, the logging level for all objects within the access must be adjusted, provided that the rule has a higher precedence than the individual objects.

Claims

CLAIMS

I. A computer system for assembling a single electronic data structure from a plurality of logical rules for the treatment of objects in electronic data traffic, whercin each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value: means for user input of a plurality of said logical rules; means for dividing each rule into its sub-expressions and dividing each sub-expression into its component parts; a term database for storing the terms of the rules with links to each of the other component parts of the sub-expression of the rule; means for comparing a term of a newly input rule with the terms stored in the term database, and if said term is not found in the term database then storing said term with a link to each of the other component parts of the related sub-expression of the newly input rule; and if said term is found in the term database, then following the stored links to the first of the previous comparisons associated with the stored term, comparing the associated stored value of the comparison with the value of the term in the newly input rule; if the value is stored then continuing to the next sub-expression; if the value is not stored then storing the value as a link to said term; then continuing for the next sub-expression of the newly input rule.

2. A system according to claim I comprising storing the data structure using an array of tn-state values wherein one value corresponds with each node in the data structure and one value corresponds to each comparison.

3. A system according to claim 2 comprising using two bits per node for storage.

4. A data structure for processing a plurality of logical rules for the treatment of objects in electronic data traffic, wherein each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value; wherein the data structure takes the form of a rule graph in which any one term appears only once, and any sub-expression occurs only once.

5. The data structure of claim 4 comprising an array of tn-state values for each object wherein one value corresponds with each node in the data structure and one value corresponds to each comparison.

6. A computer system for automatically processing logical rules governing the treatment of objects in electronic data traffic, the system comprising: means for combining a plurality of user input logical rules into a single data structure comprising a term database, a comparison operator database and a rule graph; means for intercepting data traffic comprising a plurality of said objects; means for scanning the intercepted traffic and extracting one or more terms from said objects; means for comparing an intercepted term with terms stored in the term database; if the term does not find a match, then stopping; if the term does find a match, then following stored links associated with the term to each comparison that includes the term and checking the state of the comparison; if the state is known then stopping; if the state is not known then evaluating the comparison, and triggering the associated rule if the comparison is true.

7. A system according to claim 6 wherein the object state is updated to reflect whether the term evaluated to true or false.

8. A system according to claim 7 wherein the parents of each node are evaluated in turn.

9. A system according to claim 8 comprising means for triggering a particular rule if a node in the data structure is evaluated to true and the node is the head node for said rule.

10. A system according to any one of claims 6 to 9 comprising means for taking action in response to the triggering of a rule.

11. A system according to claim 10 wherein said action includes any one or more of: logging details of the object which triggered the rule; sending an e-mail to alert a recipient that the nile has been triggered; running a command line program; and/or adjusting the logging level for the object.

12. A system substantially as hereinbefore described with reference to the accompanying drawings.

13. A method for assembling a single electronic data structure from a plurality of logical rules for the treatment of objects in electronic data traffic, wherein each rule comprises at least one sub-expression which comprises component parts including a term, a comparison operator and a value the method comprising the steps: input of a plurality of said logical rules; dividing each rule into its sub-expressions and dividing each sub-expression into its component parts; storing the terms of the rules in a term database with links to each of the other component parts of the sub-expression of the rule; comparing a term of a newly input rule with the terms stored in the term database, and if said term is not found in the term database then storing said term with a link to each of the other component parts of the related sub-expression of the newly input rule; and if said term is found in the term database, then following the stored links to the first of the previous comparisons associated with the stored term, comparing the associated stored value of the comparison with the value of the term in the newly input rule; if the value is stored then continuing to the next sub-expression; if the value is not stored then storing the value as a link to said term: then continuing for the next sub-expression of the newly input rule.

14. A method according to claim 13 further comprising the step of storing the data structure herein one array of tn-state values exists for each object wherein one value corresponds with each node in the data structure and one value corresponds to each comparison.

15. A method of processing logical rules governing the treatment of objects in electronic data traffic, the method comprising: combining a plurality of user input logical rules into a single data structure comprising a term database, a comparison operator database and a rule graph; intercepting data traffic comprising a plurality of said objects; scanning the intercepted traffic and extracting one or more terms from said objects; 1 5 comparing an intercepted term with terms stored in the term database; if the term does not find a match, then stopping; if the term does find a match, then following stored links associated with the term to each comparison that includes the term and checking the state of the comparison; if the state is known then stopping; if the state is not known then evaluating the comparison, and triggering the associated rule if the comparison is true.

16. A method according to claim 15 further comprising the step of taking action when a rule is triggered.

17. A method according to claim 16 wherein said action includes any one or more of: logging details of the object which triggered the rule; sending an e-mail to alert a recipient that the rule has been triggered; running a command line program; and/or adjusting the logging level for the object.