CN106156026B - A method for online anomaly detection of virtual assets based on data flow - Google Patents

Abstract

The invention discloses a method for online abnormal discovery of virtual assets based on data flow, which mainly includes data processing, offline analysis and online analysis. The user operation behavior log data stream flows into the data window and is preprocessed to extract the data summary. The data in the database is regularly mined by the pattern generation algorithm to mine the user's normal behavior pattern and abnormal behavior pattern. The system analyzes the data in the sliding window in real time and extracts the current Behavior patterns match normal and abnormal behavior patterns in the pattern library. The invention applies the technology of data flow to the abnormal discovery of virtual assets, and designs a technical framework of online abnormal discovery of virtual assets based on data flow, so that the system can realize real-time abnormal detection more quickly and effectively, so as to better prevent the loss of users.

Description

A method for online anomaly detection of virtual assets based on data flow

technical field

The invention belongs to the field of Internet technology, and in particular relates to a method for online anomaly detection of virtual assets based on data flow.

Background technique

The rapid development of the Internet has given birth to the prosperity of e-commerce, among which the growth of virtual asset transactions is particularly rapid. Virtual assets refer to items that are competitive, persistent and can be exchanged or traded in the online world, including online banking, online account numbers. , online game equipment weapons, virtual currency, etc.

At present, my country has carried out research on eID-based virtual asset management and security technology in cyberspace to achieve standardized and unified management of virtual assets. The virtual asset security system comprehensively and accurately records various operations on virtual assets, but how to mine abnormal transaction behaviors from these recorded data still faces many challenges. In view of the huge scale and very fast growth rate of online virtual asset transaction information, it is extremely urgent to automatically discover and predict abnormal behaviors from massive virtual asset transaction information, so as to effectively detect criminal behaviors that have occurred and may occur.

The main purpose of anomaly discovery is to train and build an anomaly detection model based on known anomalous data. Anomaly detection methods mainly include anomaly detection techniques based on statistics, information theory, spectrum-based, and machine learning-based, among which anomaly detection techniques based on machine learning mainly include cluster-based, classification-based, and sequence-based pattern-based anomaly detection techniques. Cluster-based anomaly detection technology can only be used for offline analysis. After clustering all data, those groups whose number of individuals is less than a certain threshold are regarded as anomalies. The advantage of clustering algorithm is that it does not require historical data with Label. Anomaly detection can be regarded as a classification problem in essence, which is to classify data into normal or abnormal. The anomaly detection technology mainly uses the labeled historical data for training to obtain a classifier, and then uses the classifier to classify the new data. Sequence pattern-based anomaly discovery technology mainly mines some normal behavior patterns and abnormal behavior patterns of users through multi-user operation time series data, and then extracts behavior patterns from new user data, and compares them with the normal behavior patterns and abnormal behavior patterns in the database. Match to see if the current operation is an exception.

Quan Yong et al. [1] proposed an anomaly detection method for e-commerce transaction logs based on co-occurrence matrix. The algorithm uses co-occurrence matrix to model user's transaction behavior, and establishes co-occurrence matrix space by PCA method, so as to obtain user's normal transaction. model. In the detection stage, the co-occurrence matrix generated by the data to be treated is revised and the user's transaction pattern is obtained, and the distance between the user's transaction pattern and its normal pattern is calculated by the matrix 2-norm, and the user's transaction behavior is judged whether the user's transaction behavior is abnormal.

Ji Bingshuai et al. [2] proposed another user behavior anomaly detection method in e-commerce. First, according to the characteristics of user behavior log data, it was divided into static attribute sets and operation sequence sets, and then the Apriori algorithm and GSP sequence based on axis attributes were used. The pattern mining algorithm performs pattern mining on these two types of data sets respectively, establishes the normal behavior pattern of the user on this basis, and finally uses the sequence-based pattern comparison method to match the current behavior pattern of the user with its historical normal behavior pattern. This is to determine whether the user's transaction behavior is abnormal.

Zhao Xueliang [3] proposed a data stream outlier detection method based on the sliding window model. This method uses a simple sliding window to effectively manage the change of new and old data in the data stream, and the data structure used in the algorithm effectively reduces the number of neighbors. The amount of calculation in statistics makes the algorithm performance better.

However, the anomaly detection methods in the first two [1, 2] virtual assets above are both offline analysis, and offline analysis is based on historical data analysis. Therefore, the timeliness is very low.

The outliers found by the third method [3] above are the outliers in the current sliding window, not the global outliers, and there is no framework for outlier discovery technology based on data flow.

[1] Quan Yong, Li Shudong, Jia Yan, et al. Anomaly Detection of E-Commerce Transaction Logs Based on Symbiosis Matrix [J]. China Electronic Commerce: Communication Market, 2013(4):39-45.

[2] Ji Bingshuai, Li Hu, Han Weihong, et al. Research on abnormal user behavior detection for e-commerce [J]. Information Network Security, 2014(9):80-85.

[3] Zhao Xueliang. Research on data flow outlier detection based on sliding window model [D]. Chongqing University, 2012.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention provides a method for online anomaly detection of virtual assets based on data flow, which can detect anomalies in real time, and is suitable for real-time detection of abnormal behaviors in virtual asset operations.

The technical scheme of the present invention is as follows:

A method for online anomaly discovery of virtual assets based on data flow, comprising the following steps:

(1) Data processing: The user operation behavior log data stream flows into the data window, and the data summary is extracted by preprocessing the data in the data window, and the processed data stream directly flows out of the data window and is stored in the permanent storage;

(2) Offline analysis: The data in the database is calculated once regularly, and the pattern generation algorithm is used to mine the normal behavior patterns and abnormal behavior patterns of users;

(3) Online analysis: The system analyzes the data in the sliding window in real time, extracts the current behavior pattern, and matches the normal behavior pattern and abnormal behavior pattern in the pattern library to see if it is abnormal. If it is judged to be abnormal, Perform alarm processing.

Wherein, the described step (2) comprises the following steps:

1. Data storage: When the data flow from the data window flows into the permanent storage, the default is the normal behavior label. When the real-time analysis module detects that a user operation is abnormal, it adjusts the corresponding data label in the database. At the same time, adjusting the labels of the corresponding data in the database also includes manual feedback adjustments. For example, when the system judges that a user behaves abnormally and issues an alarm and is manually confirmed as a false alarm, the information needs to be fed back to the database to adjust the labels of the corresponding data. . To deal with the storage of massive data of virtual asset user operation behavior, nosql database storage is generally used, such as Cassandra.

2. Pattern generation: For the data in the database of the offline analysis module, the system uses the pattern generation algorithm to periodically calculate once, and obtains the normal behavior pattern library and abnormal behavior pattern library of each user. The pattern generation algorithm uses a variety of algorithms, such as association rules, sequential patterns, spectral theory, and spatiotemporal sequence mining;

3. Mode update: When calculating the update mode for the data in the database, only use all the operation behavior data before the last logout of the user for analysis.

Wherein, described step (3) comprises the following steps:

1) Extract data summary: only process the data between user login and logout, record only the time of the login operation, save memory space and ensure that important information is not lost, and the data structure used is conducive to subsequent calculations;

2) Extract the current user behavior pattern: every time the user has new operation behavior data to enter, the current user behavior pattern extraction is performed on the data summary corresponding to the user;

3) Behavior pattern matching: The extracted behavior pattern is matched with the normal behavior pattern library and abnormal behavior pattern generated in the offline analysis module.

Further, the described step 1) also includes the following steps:

Step 1: First create a new HashMap named dataProfile to store the data summary;

Step 2: Read a record in the buffer, verify whether the user ID field in the record is empty, if it is empty, skip directly to step 5; otherwise, go to the next step;

Step 3: Verify whether there is a record with the key as the current user ID in the current data profile dataProfile. If not, add a record with the key as the current user ID in the dataProfile. In this case, the operation type must be a login operation, and the login needs to be recorded. time; otherwise, go to the next step;

Step 4: Check what the current operation type is. If it is a logout operation, delete the record whose key is the current user ID in the dataProfile; if it is other operations, in the value of the record whose key is the current user ID in the dataProfile Add the current operation type and the corresponding product ID to the operation sequence;

Step 5: Read the next record in the buffer and enter the loop.

Further, the described step 3) also includes the following steps:

Step a: Match with the abnormal behavior pattern in the abnormal behavior pattern library;

Step b: If the match is successful, it is judged as a known abnormality;

Step c: If the match is not successful, then match with the normal behavior pattern, if the match is successful, it is judged as normal, if the match is not successful, it is judged as an unknown abnormality;

Step d: After it is confirmed to be abnormal, perform the following four operations: ①Real-time feedback to the front end, and an abnormal alarm is issued; ②Delete the user's record in the data summary; ③Add the user to an abnormal user queue, and do not Then perform anomaly detection on it until the user sends a logout behavior, and delete it from the abnormal user queue. ④ Feed the anomaly back to the database and adjust the corresponding label.

The beneficial effects of the present invention are: when the data flowing from the data window is used to flow into the permanent storage, the default is the normal behavior label, and when the real-time analysis module detects that a certain user operation is abnormal, it adjusts the label of the corresponding data in the database, which can The data in the data window can flow out directly without waiting for the detection operation to be completed and to determine which label it belongs to, which can save memory space and prevent data from being blocked in the data window.

Since the abnormal operation of the user can be judged before the logout operation, and the real-time analysis module can immediately feedback the abnormal operation to the offline analysis module to adjust the label of the corresponding data in the database, it can ensure that the user's last logout before the last time. All data are labelled updated.

Compared with the prior art, the present invention applies the technology of data flow to the abnormal discovery of virtual assets, and designs a technical framework for online abnormal discovery of virtual assets based on data flow, so that the system can detect abnormalities in real time more quickly and effectively, So as to better prevent the loss of users.

Description of drawings

FIG. 1 is a frame diagram of an online anomaly discovery based on a data stream virtual asset according to the present invention.

FIG. 2 is a flow chart of an extraction data summary generation algorithm of the present invention.

FIG. 3 is a diagram of a hardware deployment environment of the present invention.

Detailed ways

In order to facilitate the understanding of the present invention, the present invention will be further described below with reference to the accompanying drawings and embodiments.

The present invention provides a method for online anomaly detection of virtual assets based on data flow, the frame diagram of which is shown in FIG. 1 , including an online analysis module and an offline analysis module. First, the user operation behavior log data stream flows into the data window, and the data summary is extracted by preprocessing the data in the data window. The processed data stream directly flows out of the data window and is stored in permanent storage. In the offline analysis module, the data in the database will be calculated regularly, and the pattern generation algorithm will be used to mine the user's normal behavior patterns and abnormal behavior patterns. In the online analysis module, the system will analyze the data in the sliding window in real time, extract the current behavior pattern, and then match with the normal behavior pattern and abnormal behavior pattern in the pattern library to see if it is abnormal. If it is judged to be abnormal, alarm processing will be performed.

Online analysis module: The online analysis module mainly has three tasks, namely, extracting data summary, extracting current user behavior pattern, and behavior pattern matching. Table 1 is a simple example of the user operation behavior log stream in a certain period of time. The data stream includes 12 records, the time span is mostly 50 seconds, and three users participate. This example only shows five fields of user, IP address, time, operation behavior type, and related product ID. Real data will be much more complicated. The purpose of extracting the data summary is to save valuable memory space as much as possible without losing important information, and the data structure used needs to be conducive to subsequent calculations. Therefore, the data extraction summary mode of the present invention mainly adheres to the following two requirements:

A. Only process the data between the user's login and logout;

B. Only record the time of the login operation.

Table 1 Simple example of user operation behavior log stream

Table 2 is a simple example of user operation behavior data summary generated according to the data example in Table 1. The data summary mainly includes four fields: user ID, IP address, login time, and operation sequence. The data summary is stored in the List in terms of each user, and the operation sequence in this field is also a List. When the user has new operation behavior data into the data window, the operation type and related product ID are extracted and added to the operation sequence list. middle.

Table 2 Simple example of user operation behavior data summary

The specific algorithm for extracting the data summary is shown in Figure 2. The main steps are:

Step 1: First create a new HashMap named dataProfile to store the data profile.

Step 2: Read a record in the buffer, verify whether the user ID field in the record is empty, if it is empty, skip directly to step 5; otherwise, go to the next step.

Step 3: Verify whether there is a record with the key as the current user ID in the current data profile dataProfile. If not, add a record with the key as the current user ID in the dataProfile. In this case, the operation type must be a login operation, and the login needs to be recorded. time; otherwise, go to the next step.

Step 4: Check what the current operation type is. If it is a logout operation, delete the record whose key is the current user ID in the dataProfile; if it is other operations, in the value of the record whose key is the current user ID in the dataProfile The current operation type and the corresponding product ID are added to the operation sequence.

Step 5: Read the next record in the buffer and enter the loop.

Every time a user enters new operation behavior data, the current behavior pattern is extracted from the data summary corresponding to the user, and the extracted behavior pattern is matched with the normal behavior pattern library and abnormal behavior pattern generated in the offline analysis module. The matching process is as follows: first match the abnormal behavior pattern in the abnormal behavior pattern library. If the match is successful, it will be judged as a known abnormality; if the match is not successful, it will be matched with the normal behavior pattern. It is judged as normal, and if the match is not successful, it is judged as an unknown abnormality. After it is confirmed to be abnormal, four operations need to be done: ①Real-time feedback to the front end, and an abnormal alarm is issued; ②Delete the user's record in the data summary; ③Add the user to an abnormal user queue, and no longer Anomaly detection is performed until the user sends a logout behavior, and it is deleted from the abnormal user queue; ④ Feedback the anomaly to the database and adjust the corresponding label.

Table 3 is an example of a simple behavior pattern extracted from the user user1 according to the data summary in Table 2, which indicates that the user user1 logs in at the IP address 220.79.15.21 for less than 30 minutes at around 19:00, and the price of the related product is In the range of 0-100 yuan, the operation sequence is login---browsing the goods whose similarity is 0.84 with the goods added to the shopping cart---browsing the goods added to the shopping cart---adding to the shopping cart.

Table 3 Examples of simple behavior patterns extracted by user user1

Table 4 is an example of some normal behavior patterns of user user1 in the behavior pattern database, including two association rules of IP addresses and time; the percentage of the price range of the products concerned, in the example, 80% of the products that user user1 follows are 0-100 yuan , 19% are 100-200 yuan, 1% are 200-500 yuan; the frequent pattern of three operation sequences.

Table 4 Examples of normal behavior patterns of user user1 part

In the pattern matching stage, the steps are: ① First, compare the static attributes (IP address and time, commodity price) in the user's current operation behavior pattern with all the association rules in the normal behavior pattern library. The second behavior is judged to be normal; ② Otherwise, the operation sequence in the current user operation behavior pattern is compared with all the operation sequences in the normal behavior pattern library. When the similarity exceeds the set threshold, the behavior is judged to be normal. Otherwise, it is judged as abnormal. In the given example, when the static attributes are matched, it is found that the "IP address and time" does not match. The normal behavior mode is that the login time with the IP address 220.79.15.21 is generally around 11:00, and this time it appears at around 19:00. Enter Matching of operation sequences; There are many methods for calculating the similarity of operation sequences, which is not the focus of the present invention. In this example, the Deep-Simi algorithm is used to calculate the operation sequence in the current behavior pattern and the given example is normal The similarity of the first operation sequence in the behavior pattern is 0.7, and the threshold is generally set between 0.4 and 0.6, so this behavior is judged to be normal.

Offline analysis module: mainly includes data storage and pattern generation. To deal with the storage of massive data of virtual asset user operation behavior, nosql database storage is generally used, such as Cassandra. It is worth noting that when the data flowing from the data window flows into the permanent storage, it defaults to the normal behavior label. When the real-time analysis module detects that a user operation is abnormal, it adjusts the label of the corresponding data in the database. One advantage of this is that the data in the data window does not need to wait for the detection operation to complete and determine which label it belongs to, and can flow directly out, which saves memory space, otherwise a lot of data will be blocked in the data window. At the same time, adjusting the labels in the database should also include manual feedback adjustments. For example, when the system judges that a user's behavior is abnormal and an alarm is manually confirmed as a false alarm, then we need to feed this information back to the database. Adjust the labels of the corresponding data.

For the data in the database of the offline analysis module, the system will use the pattern generation algorithm to periodically calculate it once, and obtain the normal behavior pattern library and abnormal behavior pattern library of each user. Pattern generation algorithms can use a variety of algorithms, such as association rules, sequential patterns, spectral theory, and spatiotemporal sequence mining. When calculating the update mode for the data in the database, we only use all the operation behavior data before the last logout of the user for analysis. Because some of the latest data in the database has not adjusted the label, the label is normal by default, and we can ensure that all the data before the user's last logout is the updated label, this is because if the user is The abnormal operation will definitely be judged before the logout operation. After the real-time analysis module detects the abnormality, it can immediately feed back to the offline analysis module to adjust the label of the corresponding data in the database.

The hardware deployment environment diagram of the present invention is shown in FIG. 3 . The hardware of the present invention has strong scalability. When the demand increases, it is only necessary to add cluster nodes.

Example 1

A method for online anomaly discovery of virtual assets based on data flow, the hardware specific information of the virtual asset management system is as follows:

Virtual asset data stream processing cluster: 2 nodes, the nodes are configured as 4-core CPU, 32G memory, Centos6.564-bit system;

Behavior mode computing cluster: 5 nodes, the nodes are configured as 4-core CPU, 16G memory, Centos6.564 system;

Virtual asset operation log database: 1 node, the node is configured with 2-core CPU, 8G memory, 2TB hard disk, Centos6.564-bit operating system;

Behavior pattern library: 1 node, the node is configured with 2-core CPU, 8G memory, 2TB hard disk, and Centos6.564-bit operating system.

The hardware configuration environment as above can cope with the concurrent operations of 1W level users. The virtual asset data stream processing cluster extracts data summaries from the continuously flowing data in real time, stores the data summaries in memory, and the processed data flows directly out of the sliding window and stores them in the virtual asset operation log database. The behavior pattern computing cluster regularly and continuously accesses the data in the virtual asset operation log database, calculates the user behavior pattern, and updates the behavior pattern database after obtaining a new behavior pattern. At the same time, the virtual asset data stream processing cluster extracts the current behavior pattern of the user according to the information in the data summary, and then accesses the normal behavior pattern and abnormal behavior pattern of the user in the behavior pattern library, and matches them respectively to verify whether the current operation is are abnormal. If it is judged to be abnormal, the abnormal label needs to be fed back to the virtual asset operation log database.

Compared with the prior art, the present invention applies the technology of data flow to the abnormal discovery of virtual assets, and designs a technical framework for online abnormal discovery of virtual assets based on data flow, so that the system can detect abnormalities in real time more quickly and effectively, So as to better prevent the loss of users.

The above is an exemplary description of the present invention. Obviously, the realization of the present invention is not limited by the above-mentioned methods, as long as various improvements made by the technical solutions of the present invention are adopted, or the ideas and technical solutions of the present invention are directly used without improvement. Those applied to other occasions are all within the protection scope of the present invention.

Claims (6)

1. a method for online abnormal discovery of virtual assets based on data flow, is characterized in that, comprises the following steps: Step 1: Data processing: the user operation behavior log data stream flows into the data window, and the data in the data window is preprocessed to extract the data summary, and the processed data stream directly flows out of the data window and is stored in permanent storage; Step 2: Offline analysis: The data in the database is calculated regularly, and the pattern generation algorithm is used to mine the normal and abnormal behavior patterns of users, including the following steps: Step A: Data storage: When the data stream flowing from the data window flows into the permanent storage, it defaults to the normal behavior label. When the real-time analysis module detects that the user operation is abnormal, it adjusts the corresponding data label in the database; Step B: Pattern generation: For the data in the offline analysis module database, the system uses the pattern generation algorithm to periodically calculate to obtain the normal behavior pattern library and abnormal behavior pattern library of each user; Step C: Mode update: When calculating the update mode for the data in the database, only use all the operation behavior data before the last logout of the user for analysis; Step 3: Online analysis: The system analyzes the data in the sliding window in real time, extracts the current behavior pattern, matches the normal behavior pattern and abnormal behavior pattern in the pattern library, and judges whether it is abnormal. Alarm processing, including the following steps: Step D: Extract data summary: only process the data between the user's login and logout, and only record the time of the login operation; Step E: extracting the current user behavior pattern: every time the user has new operation behavior data to enter, the current user behavior pattern extraction is performed on the data summary corresponding to the user; Step F: Behavior pattern matching: the extracted behavior pattern is matched with the user's normal behavior pattern library and abnormal behavior pattern library generated in the offline analysis module. 2 . The method for online anomaly detection based on data stream virtual assets according to claim 1 , wherein the adjustment of the labels of the corresponding data in the database in the step A further includes manual feedback adjustment. 3 . 3. The method for online anomaly discovery based on data flow virtual assets according to claim 1, wherein the pattern generation algorithm in the step B comprises association rules, sequential patterns, spectral theory, mining based on time-space sequences . 4. a kind of method based on data flow virtual asset online abnormal discovery according to claim 1, is characterized in that, also comprises the following steps in described step D: Step (1): Create a new HashMap named dataProfile to store the data summary; Step (2): read a record in the buffer, verify whether the user ID field in the record is empty, if it is empty, directly enter step 5; otherwise, enter the next step; Step (3): verify whether the key is the record of the current user ID in the current data profile dataProfile, if not, then add the record of the current user ID in the dataProfile, record the login time; Otherwise, enter the next step; Step (4): Check the current operation type. If it is a logout operation, delete the record whose key is the current user ID in the dataProfile; if it is other operations, then in the dataProfile, the key is the value of the record with the current user ID. Add the current operation type and the corresponding product ID to the operation sequence in ; Step (5): Read the next record in the buffer and enter the loop. 5. a kind of method for online anomaly discovery based on data stream virtual assets according to claim 1, is characterized in that, in described step F, also comprises the following steps: Step (1): match the abnormal behavior pattern in the abnormal behavior pattern library; Step (2): The matching is successful, and it is judged as a known abnormality; Step (3): not matching successfully, then matching with the normal behavior pattern, if the matching is successful, then it is judged as normal, if it is not matched successfully, then it is judged as an unknown abnormality; Step (4): Confirmed as abnormal. 6. a kind of method for online abnormal discovery based on data stream virtual assets according to claim 5, is characterized in that, in described step (4), also comprises the following steps: Step (1): real-time feedback to the front end, and an abnormal alarm is issued; Step (2): delete the user's record in the data summary; Step (3): add the user to the abnormal user queue, and no longer perform abnormal detection on it, until the user sends a logout behavior, delete it from the abnormal user queue; Step (4): Feed the abnormality back to the database and adjust the corresponding label.

Images