JP5856988B2 - Communication classification apparatus, method, and program - Google Patents

Description

  The present invention relates to a communication classification apparatus, method, and program, and in particular, classifies communication directly caused by a user's operation and communication mechanically transmitted in conjunction with the communication from various types of communication flowing on a network. The present invention relates to a communication classification apparatus, method, and program.

  Specifically, the present invention relates to a technology that facilitates a communication analysis for each user by aggregating a plurality of communications issued from the same user and classifying them in a communication analysis.

  In recent years, in order to efficiently analyze diversified communication traffic, various communication classification methods according to analysis applications have been devised.

  Examples of general communication aggregation include flows that aggregate packets with the same source and destination addresses, and bi-directional communication nodes from the start to the end of protocol state transition, such as TCP. There is a method of aggregating a series of packets in the form of a session. These methods aggregate packets into more meaningful communication units based on the nature of the protocol. As a result, specific analysis processing can be simplified, and the storage size for saving the communication log can be reduced. On the other hand, the number of flows and sessions generated varies greatly due to the influence of the communication software, the network of the communication destination, and the content structure. Therefore, when attempting higher-dimensional analysis such as user behavior analysis, it is necessary to further aggregate communication.

  The technology of Non-Patent Document 1 attempts to concentrate higher-order communications by paying attention to the communication log of the Web server and the user's behavior pattern. Specifically, based on the time information recorded in the log of the Web server, the user's IP address, and the information on the content to be accessed, a measurement is made of how long each user continues to access. Then, the duration of the Web session for each user and the boundary of the session are detected, and communication aggregation based on the user session is attempted.

D He, A Goker, "Detecting session boundaries from web user logs", Proceedings of the BCS-IRSG 22nd annual colloquium on information retrieval research, pp.57--66, 2000.

  However, the technique of Non-Patent Document 1 has the following three points.

  1) In this previous study, the purpose of separating sessions using time intervals is to measure the duration of the session. The purpose of the communication cluster proposed by the present invention is to visualize the relationship between communications by aggregating a plurality of communications as a communications cluster, and not to obtain the duration of communications. The purpose of the present invention is different from that of previous research in that communication is simply distinguished using time intervals.

  2) In this previous study, only the time interval between adjacent communication logs was used as a criterion for distinguishing each session for each user. The communication is not divided based on the determination of “which communication is triggered by each session”. In this prior research method, communication that triggers a session cannot be identified, and thus communication that is actually performed by a user cannot be distinguished from communication that is caused by software caused by the operation.

  3) Since this previous research uses Web server logs, the communication measurement point must be a server installed at the end of the network. Therefore, it cannot be applied to measurements at intermediate nodes in the network, and the protocol to be measured is limited to HTTP (Hyper Text Transfer Protocol) / HTTPS (Hyper Transfer Protocol over Secure Socket Layer). It is scarce.

  The present invention has been made in view of the above points. Based on a communication log collected at an arbitrary measurement point on a network, communication based on user actions and mechanical causes are not limited to a specific communication protocol. It is an object of the present invention to provide a communication classification apparatus, method, and program capable of obtaining a communication classification result suitable for user behavior analysis by discriminating between the above-described communications and aggregating a plurality of communications.

In order to solve the above-described problem, the present invention (Claim 1) is directed to communication (hereinafter referred to as “representative communication”) directly resulting from a user's operation from various types of communication flowing on the network. A communication classification device for classifying communication that is mechanically transmitted in conjunction with the communication (hereinafter referred to as “subordinate communication”),
When the communication data obtained by capturing the communication flowing on the network or the collected communication log is input, the input means stores in the communication data storage means in the input order;
Two continuous communication data X _n and X _{n + 1} are acquired from the communication data storage means, and when the time interval between the communication data X _n and X _{n + 1} is _{equal to} or greater than a predetermined threshold Tc, the two communication data The data are different communication clusters, and the communication data X _{n + 1} is a representative communication. If the data is smaller than the threshold value Tc, it is the same communication cluster, and the X _{n + 1} is determined as a dependent communication. Time interval comparison means;
The communication data X _{n + 2} next to the communication data X _{n + 1} that has become the representative communication is acquired from the communication data storage means, and the communication data X _{n + 2} and the communication data X _{n + 1} If the difference is smaller than a predetermined representative communication identification threshold Tf, representative communication identification means that makes the communication data X _{n + 1} a dependent communication,
A result storage unit that stores the classification results of the time interval comparison unit and the representative communication identification unit together with a communication identifier that uniquely indicates the communication data in a classification result storage unit.

  In the present invention (Claim 2), when there is an incomplete cluster in which representative communication is missing in the communication cluster of the classification result storage means, a dependent communication group of another communication cluster of the classification result storage means The communication class supplement means further includes a representative communication of a complete communication cluster having the highest similarity as the representative communication of the incomplete cluster.

Further, according to the present invention (Claim 3), in the communication data storage means,
If communication data for each user is not observed for a certain time or longer, a means for deleting is included.

  As described above, according to the present invention, communication based on a user's action is distinguished from communication caused by a mechanical action based on a communication log collected at an arbitrary measurement point on a network without being limited to a specific communication protocol. In addition, by collecting a plurality of communications, it is possible to apply the technique to a communication classification result suitable for user behavior analysis.

It is a figure which shows the outline | summary 1 of the communication analysis method in one embodiment of this invention. It is a figure which shows the outline | summary 2 of the communication analysis method in one embodiment of this invention. It is a figure which shows the outline | summary 3 of the communication analysis method in one embodiment of this invention. It is an example of composition of a communication classification device in one embodiment of the present invention. It is a flowchart of operation | movement of the communication classification | category part in one embodiment of this invention. It is an example of output data of the communication classification part and communication classification complementation part in one embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the outline of the present invention will be described.

  In the communication to which the present invention is applicable, when attention is paid to communication transmitted from a specific user, the communication is continuously and intensively generated in the form following the specific communication. This is a recognized communication. For example, connection requests in TCP (Transmission Control Protocol) communication, DNS (Delivery Status Notification) query request packets, HTTP sessions, and the like are applicable.

  The present invention is to identify a first trigger communication and a communication that occurs dependently on the first communication among a series of communication generated by a user to achieve a specific purpose.

  When the target communication is a stateful protocol, the first communication start request packet in a series of packets that exercise each session is extracted and used as an input to the system of the present invention. For example, the packet is a syn flag in the case of TCP communication, and is a GET request packet in the case of HTTP communication.

  When the target communication is a stateless protocol, each packet that the user requests from the server is input to the system of the present invention. For example, in the case of DNS communication, it is a DNS query request packet.

  Next, the communication classification method of the present invention will be described.

  FIG. 1 shows an outline of a communication analysis method according to an embodiment of the present invention.

  In a series of continuous communications with the same purpose by the same user, the first trigger communication is referred to as “typical communication”, and the subsequent dependent communication is referred to as “subordinate communication”.

  The communication cluster in the figure is composed of one representative communication and a plurality of subordinate communications. The final purpose of the present invention is to accurately classify communication clusters from communication logs.

  When attention is paid to communication from A1 to A3 in FIG. 1, since communication of A2 and A3 occurs continuously after communication of A1, A1 is representative communication, and A2 and A3 are Dependent communication. The same applies to communications from C1 to C3.

  The communication B1 is not continuous with the preceding and succeeding communications and has sufficient time for the preceding and succeeding communications, so that one representative communication belongs to an independent communication cluster (communication cluster B). . The “sufficient time” here is an important parameter for classifying communication clusters. This will be described later with reference to FIG.

  There are cases where all communications are regarded as dependent communications, such as D1 to D4. In this case, the cluster is an incomplete communication cluster. Details of such a case will be described in the description of the figure. An application technique using an incomplete cluster will be described later.

[1] Communication classification method:
The communication classification method will be described below.

  FIG. 2 shows an outline 2 of the communication analysis method according to the embodiment of the present invention. In the figure, a thick line arrow shows typical communication, and a broken line arrow shows dependent communication.

  In FIG. 2, for communication from X1 to X10, intervals between the communication are shown from t1 to t9, respectively. Assume that the communication cluster to which each communication belongs is still unknown.

(1) Classification into communication clusters:
When classifying communication clusters for communication, a threshold value Tc between communication clusters is set as a parameter. Tc represents the minimum time interval in which each communication cluster is considered to be different in two consecutive communications.

  When the time interval between any two consecutive communications is larger than Tc, the two communications are considered to be different communication clusters. For example, in FIG. 2, when t1 ≧ Tc, X1 and X2 are regarded as different communications. On the other hand, when t2 <Tc, it is assumed that X2 and X3 belong to the same communication cluster.

  In the above method, when a certain user performs a plurality of communications in parallel at the same time, a plurality of different types of communications of the user may be mixed into one communication cluster, and the communication cluster may not be correctly identified. In the above method, a rough classification result of communication is obtained by using time difference information between communications, and reliable accuracy of communication classification is not guaranteed. Therefore, erroneous communication cluster classification determination is not considered in the scope of the present invention, but is considered in a higher-level analysis system using the present invention.

(2) Detection of representative communications:
When classifying the communication cluster, a representative communication identification threshold value Tf is set as a parameter. Tf is a time interval and is used to detect a communication cluster in which no representative communication exists. However, Tf <Tc.

  In principle, the first communication of each communication cluster obtained by the above-described method of classifying into communication clusters (1) is regarded as representative communication. For example, in the communication cluster B of FIG. 2, X2 is communication that appears first, and therefore is considered as representative communication. After X3, it becomes dependent communication. Exceptions that assume that there is no representative communication are discussed in the next section.

  When the time interval between the representative communication and the dependent communication is smaller than the threshold value Tf, it is assumed that the representative communication is missing, and the representative communication is changed to the dependent communication. In addition, the communication cluster is set as an incomplete cluster. For example, in the communication cluster C of FIG. 2, when t7 <Tf, X7 is regarded as dependent communication. The reason for this will be described later with reference to FIG.

(3) Parameter selection method:
The threshold values Tc and Tf are determined by prior trials in accordance with the network environment and protocol to be measured, and optimal values for appropriately classifying the communication cluster or the representative communication. Tc and Tf are not necessarily fixed values. It is also possible to dynamically determine the optimum Tc or Tf by providing a certain learning period.

[2] Supplementary methods for typical communications:
Next, as an extension of the above technique, a supplementary method when representative communication is not observed will be described.

  FIG. 3 shows an outline 3 of the communication analysis method according to the embodiment of the present invention. In the figure, black circles represent representative communications, and white circles represent dependent communications.

  In communication, there are cases in which representative communication cannot be selected by the method of FIGS. The case is a case where the communication that should be originally transmitted is not transmitted because the caller has cached the communication. Specifically, for example, when a record to be inquired in DNS communication already exists in the local cache on the sender side, a DNS inquiry is not performed.

  The demerit caused by the loss of the representative communication is that it is difficult to determine the attribute of the communication cluster due to the lack of characteristic communication that triggered the generation of the communication cluster.

  A procedure for complementing representative communication will be described with reference to FIG. In FIG. 3, the upper half communication cluster is an incomplete communication cluster in which the representative cluster is missing, and the lower half three communication clusters are complete communication clusters in which the representative cluster has been detected.

  1) When the representative communication is missing, the similarity between another dependent communication group of the communication cluster and a dependent communication group of another communication cluster is determined. Here, whether or not the representative communication is missing can be determined by recording the presence or absence of the representative communication at the time of the operation of changing the representative communication to the dependent communication and referring to the information.

  2) As a result, representative communication of a complete communication cluster having a similarity exceeding a certain threshold and having the highest similarity is regarded as provisional representative communication of an incomplete communication cluster.

  The communication classification apparatus of the present invention will be described below.

  FIG. 4 shows an example of the configuration of the communication classification apparatus according to an embodiment of the present invention.

  The communication classification device 10 shown in FIG. 1 includes a communication classification unit 11, memories 12 </ b> A and 12 </ b> B, a communication log database 13, a classification result database 14, and a communication classification complement unit 15.

  As a method of inputting data to the communication classification device 10, there are a method of inputting communication flowing on the network in real time from the communication capture device 1, or a method of using already collected communication or communication log as input data.

  The memory 12A temporarily stores input communication data (for example, transmission source address) for each user, and the memory 12B previously stores parameters of the communication cluster threshold Tc and the representative communication identification threshold Tf, And the classification result by the process of the communication classification part 11 is stored.

  The communication classification unit 11 classifies the communication into communication clusters from the input data by the method described above, and stores the classification result in the classification result database 14 together with the communication identifier. At that time, the transmission source address (IP address, etc.) of the user is held in the memory 12A for a certain period, and the classification of communication for each user is realized. Here, the “communication identifier” is a tuple of information that can be uniquely identified, depending on a target communication protocol. The communication for each user stored on the memory 12 is deleted from the memory 12 when the communication of the user is not observed for a certain period of time. This time is determined in consideration of the communication volume of the target network and the performance of the system.

  FIG. 5 is a flowchart of the operation of the communication classification unit according to the embodiment of the present invention.

When communication capture data is input from the communication capture device 1 or a communication log is input from an external device and stored in the memory 12A (step 101), the communication classification unit 11 determines that there is communication data in the memory 12A (step 101). 102, No) stores the communication data in the communication log database 13, and takes out two consecutive communication data _Xn , _{Xn + 1} from the memory 12A (step 103). In addition, when storing in the communication log database 13, the communication identifier and time information which can identify the said communication uniquely at least shall be provided. If the time interval (X _{n + 1} -X _n ) between the two communication data is smaller than the communication cluster threshold Tc stored in the memory 12B (step 104, No), X _n and X _{n + 1} are the same. The communication cluster is determined, _{Xn + 1} is set as a dependent communication (step 105), and the classification result is stored in the memory 12B (step 110).

On the other hand, if the time interval (X _{n + 1} −X _n ) between the two communication data is greater than or equal to the communication cluster threshold Tc stored in the memory 12B (step 104, Yes), X _n and X _{n + 1} Is determined to be another communication cluster, and X _{n + 1} is determined to be representative communication (step 106). Then, the next data X _{n + 2} of X _{n + 1} from the memory 12A is taken out (step 107), X _{n + 2} -X when _{n + 1} value is less than the typical communication identification threshold Tf (step 108, Yes), X _{n + 1} determined as representative communication in step 106 is changed to “subordinate communication”, and the classification result in the memory 12B is updated (step 110). On the other hand, _if the value of X _{n + 2} −X _{n + 1} is greater than or equal to the representative communication identification threshold value Tf (step 108, No), the “typical communication” determined in step 106 is stored in the memory 12B (step 110). The value of n is incremented by 1, and the process proceeds to step 102 (step 111). If there is no communication capture data or log data to be read into the memory 12A (step 102, Yes), the classification result stored in the memory 12B is stored in the classification result database 14 (step 112), and the process is terminated.

  The communication classification complementing unit 15 refers to the classification result database 14 and extracts a record in which no representative communication exists, and with respect to the incomplete communication cluster, another communication cluster of the transmission user of the communication, or another Complement missing typical communications from the user's communications cluster.

  The procedure for complementing the missing representative communication is shown using the example of FIG. FIG. 6 shows data examples of the output result (A) of the communication classification unit and the output (B) of the communication complementing unit. In the output of the communication classification unit (A), the communication cluster A is composed of communication X0, X1,... Xn, and the communication clusters B and C are the same. However, it is assumed that the communication cluster B is in a state where it is determined that the representative communication Y0 is missing by the procedure described above in “[1] (2) Detection of representative communication”. Here, the communication complementing unit compares each communication of the communication cluster B lacking the representative communication with the communication of another communication cluster.

  Next, a comparison method between communication clusters will be described. As an index used for communication comparison, a parameter that appears in common even when the communication destination server is the same and the user is different is selected. Here, an index used for comparison is Pn (n: number of each index). Examples of the index are the transmission destination address and the number of communication bytes of each communication, or the total communication data amount and the total number of communication of all the communication except the representative communication of the communication cluster. When these indices are P1 to Pn, the weight (W1 to Wn) corresponding to each index is defined. When the similarity between communication clusters is defined as S, the similarity is expressed as a total value of values obtained by multiplying the comparison result of each index by a weight. The similarity Sab between the communication cluster a and the communication cluster b is derived as follows.

ここで、指標の数はNであり、Pkaは通信クラスタaにおける指標kの値を表す。またEqは与えられた2つの指標が一致していた場合は１，そうでなければ0とする関数であり、Wは指標kに対応する重みである。
通信クラスタBと、他の通信クラスタとのそれぞれの類似度Sを求めた結果、Sが一定閾値(任意に定める)以上であり、かつ複数の通信クラスタが該当した場合はSが最も大きい通信クラスタを、補完元通信クラスタとして選択する。いずれのSも閾値を超えなかった場合は、類似した代表的通信が存在しないと見なす。図6において通信クラスタCが補完元通信クラスタとして選択されたと仮定した場合、通信クラスタCの代表的通信Z0を通信クラスタBの代表的通信Y0として補完する。

Here, the number of indices is N, and Pka represents the value of the index k in the communication cluster a. Eq is a function that is 1 if the two given indices match, and 0 otherwise, and W is a weight corresponding to the index k.
As a result of obtaining the similarity S between communication cluster B and another communication cluster, if S is equal to or greater than a certain threshold (arbitrarily determined) and multiple communication clusters are applicable, the communication cluster with the largest S Is selected as a complement source communication cluster. If neither S exceeds the threshold, it is considered that there is no similar representative communication. If it is assumed that the communication cluster C is selected as the complement source communication cluster in FIG. 6, the representative communication Z0 of the communication cluster C is supplemented as the representative communication Y0 of the communication cluster B.

  In the communication classification complementing unit 15 described above, there is a possibility that the missing representative communication is erroneously complemented by a communication that is not directly related to the missing communication. If it is less than a certain value, the incomplete communication cluster is discarded in the subsequent analysis process (outside the scope of the present invention), and the problem is handled by a method that does not affect other communication clusters.

  The operation of the components of the communication classification apparatus shown in FIG. 4 can be constructed as a program and installed in a computer used as the communication classification apparatus for execution or distributed through a network. .

  The present invention is not limited to the above-described embodiments, and various modifications and applications can be made within the scope of the claims.

DESCRIPTION OF SYMBOLS 1 Communication capture device 10 Communication classification device 11 Communication classification part 12A, 12B Memory 13 Communication log database 14 Classification result database 15 Communication classification complement part

Claims (7)

From a wide variety of communications that flow on the network, communications that are directly attributable to user operations (hereinafter referred to as “typical communications”), and communications that are mechanically transmitted (hereinafter referred to as “subordinate”). A communication classifying apparatus for classifying the data into “communication”),
When the communication data obtained by capturing the communication flowing on the network or the collected communication log is input, the input means stores in the communication data storage means in the input order;
Two continuous communication data X _n and X _{n + 1} are acquired from the communication data storage means, and when the time interval between the communication data X _n and X _{n + 1} is _{equal to} or greater than a predetermined threshold Tc, the two communication data The data are different communication clusters, and the communication data X _{n + 1} is a representative communication. If the data is smaller than the threshold value Tc, it is the same communication cluster, and the X _{n + 1} is determined as a dependent communication. Time interval comparison means;
The communication data X _{n + 2} next to the communication data X _{n + 1} that has become the representative communication is acquired from the communication data storage means, and the communication data X _{n + 2} and the communication data X _{n + 1} If the difference is smaller than a predetermined representative communication identification threshold Tf, representative communication identification means that makes the communication data X _{n + 1} a dependent communication,
A result storage means for storing the classification results of the time interval comparison means and the representative communication identification means together with a communication identifier uniquely indicating the communication data in a classification result storage means;
A communication classification apparatus comprising: If there is an incomplete cluster in which representative communication is missing in the communication cluster of the classification result storage means, the similarity of the subordinate communication groups of other communication clusters of the classification result storage means exceeds a predetermined threshold, and The communication classification apparatus according to claim 1, further comprising: a communication classification complementing unit that sets the representative communication of the complete communication cluster having the highest similarity as the representative communication of the incomplete cluster. The communication data storage means includes
The communication classification device according to claim 1, further comprising means for deleting when communication data for each user is not observed for a certain period of time. From a wide variety of communications that flow on the network, communications that are directly attributable to user operations (hereinafter referred to as “typical communications”), and communications that are mechanically transmitted (hereinafter referred to as “subordinate”). A communication classification method for classifying a communication
In an apparatus having input means, time interval comparison means, representative communication identification means, and result storage means,
An input step in which the input means stores the communication data obtained by capturing the communication flowing on the network, or when the collected communication log is input, and stores it in the communication data storage means in the order of input;
The time interval comparing means, two communication data X _n continuing from the communication data storage means, obtains the X _{n + 1,} the communication data X _n, the time interval of X _{n + 1} is greater than a predetermined threshold value Tc The two communication data are different communication clusters, and the communication data X _{n + 1} is a representative communication. If the communication data is smaller than the threshold Tc, the two communication data are the same communication cluster, and the X _{n + 1} Is a time interval comparison step for determining a subordinate communication;
The representative communication identification means acquires communication data X _{n + 2} next to the communication data X _{n + 1} that has become the representative communication from the communication data storage means, and the communication data X _{n + 2} If the difference between the communication data X _{n + 1} is smaller than a predetermined representative communication identification threshold Tf, a representative communication identification step in which the communication data X _{n + 1} is a dependent communication;
A result storage step of storing the classification results of the time interval comparison step and the representative communication identification step together with a communication identifier uniquely indicating the communication data in a classification result storage unit;
The communication classification method characterized by performing. When there is an incomplete cluster in which representative communication is missing in the communication cluster of the classification result storage unit, the communication classification complementing unit has a predetermined similarity degree of a dependent communication group of another communication cluster of the classification result storage unit The communication classification method according to claim 4, further comprising a communication classification complementing step in which the representative communication of the complete communication cluster having the highest similarity and exceeding the threshold is set as the representative communication of the incomplete cluster. In the input step,
The communication classification method according to claim 4, wherein when communication data for each user is not observed for a certain time or longer, the data in the communication data storage means is deleted. Computer
The communication classification program for functioning as each means of the communication classification apparatus of any one of Claims 1 thru | or 3.

Images