CN107992469A - A kind of fishing URL detection methods and system based on word sequence - Google Patents

Abstract

The present invention provides a kind of fishing URL detection methods and system based on word sequence, for detecting fishing URL.By being segmented to URL character strings, and then obtain the vector representation of word sequence, then the contextual information and feature in word sequence are learnt automatically using deep learning model, it is not necessary to manually include the relevant text feature of word in extraction URL, be used for detecting fishing URL using trained model.So as to solve the problems, such as to run into the fishing URL detections of above-mentioned existing word-based feature.

Description

A method and system for detecting phishing URLs based on word sequences

technical field

The invention relates to the field of information security, in particular to a method and system for detecting phishing URLs based on word sequences.

Background technique

Phishing URLs are a type of phishing attempt to obtain sensitive user information such as usernames, passwords, and credit card details by masquerading as a reputable corporate media website. Phishing URLs usually claim to be from popular social networking sites (including YouTube, Facebook, Twitter, etc.), auction sites (eBay), electronic shopping sites (PayPal, Alibaba, etc.), or network managers (Google, Yahoo, Internet service providers, etc.) business), etc., in order to lure the credulity of the victim. The deception method often used by attackers is to embed keywords that confuse users in URLs. For example, attackers use URLs such as "login.mydomain.tld/paypal" to lure PayPal users.

At present, there are many phishing URL detection methods and security products, both in the research field and in commercial products. Most of the main principles are based on manually extracting the characteristics of URL-related data, building a classification model, and classifying URLs to detect phishing URLs. URL. According to the analysis data, existing detection methods can be divided into two categories: detection methods based on multi-source information and detection methods based on URL itself.

Detection methods based on multi-source information need to collect various data related to URLs, including Alexa rankings, WHOIS information, web page content, etc., and construct complex models to train marked data to detect whether unknown URLs are phishing URLs. This method usually has a relatively high accuracy rate, but because collecting these various data requires a lot of extra overhead such as resources and time, it is not suitable for real-time detection in high-speed networks.

The detection method based on the URL itself only analyzes the text features of the URL string itself and is used to build a classification model. It is a lightweight detection method and is suitable for real-time detection.

Specifically, the phishing detection method based on the URL itself extracts the text features of the URL string and trains a classification model to detect phishing URLs. The text features of the URL string itself can be divided into character features and word features. Character features mainly consider the characteristics of the characters that make up the URL text string, including the length of characters, the proportion of vowels and consonants, the number of numbers, the number of special symbols, and the entropy value of character distribution. Word features mainly analyze the words with semantic information contained in the URL and their occurrence frequency features, such as the commonly used words login, update, etc. in URLs, as well as popular well-known brands such as paypal, alibaba, etc.

Lightweight phishing detection based on the URL itself is more in line with the need for real-time response in high-speed networks. Character-based features ignore the semantic information contained in URLs. URLs are used to facilitate human memory, so they are usually readable and easy to remember, and contain multiple meaningful common words. Moreover, in phishing attacks, attackers often use keywords to confuse users.

Most of the current phishing URL detection methods based on word features use words and frequency of occurrence as features, without considering the word sequence features contained in the URL, and these features are all based on manual proposals, which have certain limitations. First of all, manually extracting features requires a lot of manpower and resources to statistically analyze and verify the effectiveness of features; secondly, manually extracted features are usually only valid for a certain type of data and have poor robustness; moreover, attackers use Keywords are usually similar to normal URLs, which can confuse users and reduce the detection efficiency of classification models.

Contents of the invention

In view of the deficiencies in the above-mentioned prior art, the purpose of the present invention is to provide a method and system for detecting phishing URLs based on word sequences, which are used to detect phishing URLs. By segmenting the URL string, the vector representation of the word sequence is obtained, and then the deep learning model is used to automatically learn the context information and features in the word sequence, without the need to manually extract the text features related to words in the URL, and the trained model is used Used to detect phishing URLs. Thereby, the aforementioned problems encountered in the existing detection of phishing URLs based on word features are solved.

For reaching above-mentioned purpose, the technical scheme that the present invention takes is:

A method for detecting phishing URLs based on word sequences, comprising the following steps:

Convert labeled URLs into word sequence vectors as training data;

Using the training data to train the classification model;

Convert unknown URLs into word sequence vectors and input them into the trained classification model for labeling.

Further, converting marked URLs or unknown URLs into word sequence vectors includes:

Filter out protocols and gTLDs in marked URLs or unknown URLs;

Segment the remaining part after filtering, and use the dictionary to segment the character string of each segment obtained by the segmentation through the forward maximum matching method to obtain the word sequence;

All the words in the above dictionary are numbered starting from 1, so that each word has a unique number, and the word sequence of each marked URL or unknown URL is converted into a fixed-length vector represented by a number.

Further, the protocol includes http, https, ftp, ftps, gopher; the gTLD includes com, org, net, edu, gov.

Further, said using a dictionary to perform word segmentation by means of forward maximum matching includes:

Determine whether the entire string is in the dictionary, and if so, no word segmentation is performed;

If not, remove the last character and determine whether the remaining string is in the dictionary;

Repeat the above judgment process until the word in the dictionary is matched, and then remove the word in the match;

Continue to perform the above steps on the remaining part of the string until all the strings are processed;

If the string does not contain words in the dictionary, it is broken into individual characters.

Further, the dictionary selects the Google English word corpus disclosed by Peter Norvig.

Furthermore, the classification model trained with training data is trained with a bidirectional LSTM model based on word sequences.

Further, using the training data to train the classification model includes:

The training data is randomly divided into a training part and a verification part, and the bidirectional LSTM model is trained by setting parameters such as hyperparameters and activation functions of the neural network model.

Further, the bidirectional LSTM model includes a four-layer neural network including an embedding layer, a bidirectional LSTM layer, a dropout layer, and a sigmoid layer. Using training data to train the classification model also includes: using the dropout function on the output of the bidirectional LSTM layer to prevent overfitting.

A phishing URL detection system based on word sequences, including:

Conversion modules and classification training models;

The conversion module is used to convert the marked URL into a word sequence vector as the training data of the training classification model; and is used to convert the unknown URL into a word sequence vector and input it into the trained classification model for labeling.

As mentioned above, the method and system provided by the present invention do not need to manually extract any features, only need to convert the URL into a word sequence vector representation, and automatically learn the context information and features in the word sequence through a deep neural network (two-way LSTM model), Used to detect phishing URLs.

Compared with traditional techniques for detecting phishing URLs, it has the following advantages:

First of all, there is no need to additionally collect URL-related data and manually extract URL text features, and automatically learn the word sequence context information and features of URLs by using deep learning models, and use them to detect phishing URLs; significantly reducing overhead.

In addition, by deeply mining the contextual information and features contained in the word sequence of the URL, compared with the machine learning model based on manually extracted word features and the deep learning model based on character sequences, both have better results. The detection effect is better.

Finally, with the method and system of the present invention, using the trained model, on a common server, the single-thread prediction speed can reach no less than 600 URLs per second. On the premise of improving the detection accuracy, it can meet the needs of real-time detection at the same time.

Description of drawings

FIG. 1 is a schematic flowchart of a method for detecting phishing URLs based on word sequences in an embodiment of the present invention.

FIG. 2 is a schematic structural diagram of a bidirectional LSTM model used in a method for detecting phishing URLs based on word sequences in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention.

In one embodiment of the present invention, a kind of phishing URL detection method and system based on word sequence are provided, and the main steps of the method include:

(1) word sequence vector representation, at first, adopt the method based on dictionary matching to obtain the keyword sequence contained in the URL, then obtain the vector representation of URL word sequence based on dictionary encoding;

(2) Model training, for the word sequence vector obtained in the previous step, use the marked training data to train the bidirectional LSTM model based on the word sequence;

(3) Phishing URL detection, using a trained bidirectional LSTM model based on word sequences to detect whether an unknown URL is phishing.

The system includes: conversion module and classification training model;

The conversion module is used to convert the marked URL into a word sequence vector representation as the training data of the training classification model; and is used to convert the unknown URL into a word sequence vector representation and input it into the trained classification model for labeling.

The word sequence vector representation step in the method is mainly to obtain the vector representation of the URL word sequence, mainly including the following steps:

i) First, filter out known protocols and gTLDs in the URL. Commonly used protocols include http, https, ftp, ftps, gopher, etc., and gTLDs include com, org, net, edu, gov, etc. 14;

ii) For the remaining part, first use symbols to segment, and then use the pre-prepared dictionary for each segment to perform word segmentation through the method of forward maximum matching, combined with the pseudo code shown in Algorithm 1 in the following table, the specific word segmentation process is : First judge whether the entire string is in the dictionary, if it is, no word segmentation is needed; if not, remove the last character, and judge whether the remaining string is in the dictionary until it matches the word in the dictionary, Then remove the words in the match, and continue the above steps for the rest of the string until all the strings are processed. If the string does not contain the words in the dictionary, it is divided into individual characters.

The dictionary used in the above word segmentation process is the Google English word corpus released by Peter Norvig (including 333,333 English words); other English word dictionaries are not applicable. This dictionary is Peter Norvig's statistics of commonly used words in web pages, which is more in line with the URL naming method.

iii) Then, all words in the above-mentioned dictionary are numbered from 1, and each word has a unique number, and the word sequence of each URL is converted into a fixed-length vector represented by numbers;

In the model training step in the method, the vector set obtained in the previous step is used as the training data to train the bidirectional LSTM model based on the word sequence. The training sample set is randomly divided into two parts: training and verification (respectively accounting for about 80% and 20% of all labeled data), by setting the hyperparameters of the neural network model (the output dimension of each layer, etc.) Bidirectional LSTM model for training. The deep learning model used includes a multi-layer neural network, which is an embedding layer, a bidirectional LSTM layer, a dropout layer, and a sigmoid layer. The dropout function is used to prevent overfitting for the output of the bidirectional LSTM layer.

The phishing URL detection step in the method is mainly to detect whether the unmarked data, that is, the unknown URL, is phishing. Input the word sequence vector of the unknown URL into the trained bidirectional LSTM model for labeling. If the output is 1, it means that it is a phishing URL, otherwise it is a normal URL.

To further illustrate with an example: the overall process of the word sequence-based phishing URL detection method is shown in Figure 1, and the structure of the word-sequence-based bidirectional LSTM model is shown in Figure 2.

Take the phishing URL: http://shen.mansell.tripod.com/games/gameboy.html as an example. The URL label status is 1, and the URL is represented by a fixed-length word sequence vector and the bidirectional LSTM model is trained, and the training is used A good model detects the unknown URL: http://fly-project.net//yahoo.link/Yah/T/Y.html.

1) First, perform word sequence vector representation on the input URL, and first use the pre-prepared dictionary to segment the URL:

Then number the words in the dictionary, and the word sequence is expressed as a fixed-length vector with a length of N. The value of N can be obtained through statistics. After statistics, it is found that more than 90% of URLs contain 13 words, so set N=13, then the two URLs get vectors (1,4,5,6,7,11,13,0,0,0,0,0,0) and (2,19,3,9,12, 8,14,0,0,0,0,0,0).

Use the same method to obtain the word sequence vector representations of all URLs in the sample set. The sample set contains marked normal URL and phishing URL data.

2) Use the data marked in the word sequence vector set as training data to input into the two-way LSTM model based on word sequence as shown in Figure 2 for training. First, the word sequence vector of the URL is input to the Embedding layer for dimensionality reduction processing, and then input to Learning is carried out in the bidirectional LSTM layer, and the learning result is input to the dropout layer to prevent overfitting, and the last layer of sigmoid function outputs the detection result. Marking 1 indicates a phishing URL, and marking 0 indicates a normal URL, which is actually a binary classification problem, so the model output uses the sigmoid function for 0-1 classification.

Input all the labeled data into the model training data, and output the trained model.

3) For unlabeled data, input its vector into the trained model, and output the labeled result, if the output is 1, it means it is a phishing URL, otherwise it is a normal URL.

Therefore, from the above example, we can see that the method in this example does not need to manually extract any features, but only needs to convert the URL into a word sequence vector representation, and automatically learn the context information and features in the word sequence through a deep neural network (bidirectional LSTM model) , used to detect phishing URLs.

Its main steps include: 1) word sequence vector representation, URL is carried out participle at first, the URL here comprises labeled and unknown. All URLs are converted to vectors, and then the model is trained with the labeled data. Then use the method of filling the sequence to obtain a fixed-length vector representation; "fixed length" means that the length of the word sequence vector obtained by each URL is the same. The padding sequence method is used to handle vectors of different lengths, converting them to the same length.

2) Model training, for the vector obtained in the previous step, use the marked training data to train the bidirectional LSTM model.

3) Phishing URL detection. For unlabeled URLs, input their vector representations into the trained two-way LSTM model for labeling. Those marked as 1 are phishing URLs.

Step 1) at first by word sequence vector representation, obtain the fixed-length vector representation of URL character string, this method trains and analyzes the vector representation of URL;

Step 2) To the preprocessed data, use the marked data to train the bidirectional LSTM model based on the word sequence;

Step 3) Input the vector representation of the unknown URL into the trained two-way LSTM model for labeling, and detect whether it is a phishing URL;

Use the above method to detect phishing URLs; the context information and features contained in the word sequence of the URL can be deeply mined, compared with the machine learning model based on manually extracted word features and the deep learning model based on character sequences. The detection effect on the same data set is shown in Table 1;

Moreover, this method is a lightweight phishing URL detection method. Using a trained model, on a common server, the single-thread prediction speed can reach no less than 600 URLs per second. It can meet the requirements of real-time detection while improving the detection accuracy.

Table 1 Comparison of detection results of four different detection models

Model Precision recall F1 Decision Tree Model Based on Word Features 0.8803 0.8700 0.8751 Random Forest Model Based on Word Features 0.8981 0.8965 0.8973 Bidirectional LSTM Model Based on Character Sequence 0.9553 0.9474 0.9513 Bidirectional LSTM model based on word sequence 0.9808 0.9716 0.9762

Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Claims (9)

1. A method for detecting phishing URLs based on word sequences, comprising the following steps: Convert labeled URLs into word sequence vectors as training data; Using the training data to train the classification model; Convert unknown URLs into word sequence vectors and input them into the trained classification model for labeling. 2. the phishing URL detection method based on word sequence as claimed in claim 1, is characterized in that, converting the marked URL or unknown URL into a word sequence vector comprises: Filter out protocols and gTLDs in marked URLs or unknown URLs; Segment the remaining part after filtering, and use the dictionary to segment the character string of each segment obtained by the segmentation through the forward maximum matching method to obtain the word sequence; All the words in the above dictionary are numbered starting from 1, so that each word has a unique number, and the word sequence of each marked URL or unknown URL is converted into a fixed-length vector represented by a number. 3. the phishing URL detection method based on word sequence as claimed in claim 2, is characterized in that, described agreement comprises http, https, ftp, ftps, gopher; Described gTLD comprises com, org, net, edu, gov. 4. the phishing URL detection method based on word sequence as claimed in claim 2, is characterized in that, described using dictionary carries out word segmentation by the mode of forward maximum matching and comprises: Determine whether the entire string is in the dictionary, and if so, no word segmentation is performed; If not, remove the last character and judge whether the remaining string is in the dictionary; Repeat the above judgment process until the word in the dictionary is matched, and then remove the word in the match; Continue to perform the above steps on the remaining part of the string until all the strings are processed; If the string does not contain words in the dictionary, it is broken into individual characters. 5. the phishing URL detection method based on word sequence as claimed in claim 4, is characterized in that, described dictionary selects the Google English word corpus disclosed by Peter Norvig for use. 6. the phishing URL detection method based on word sequence as claimed in claim 2, is characterized in that, the classification model that adopts training data training selects the two-way LSTM model based on word sequence to train. 7. the phishing URL detection method based on word sequence as claimed in claim 1, is characterized in that, adopts training data training classification model to comprise: The training data is randomly divided into a training part and a verification part, and the bidirectional LSTM model is trained by setting parameters such as hyperparameters and activation functions of the neural network model. 8. the phishing URL detection method based on word sequence as claimed in claim 7, is characterized in that, two-way LSTM model comprises embedding layer, two-way LSTM layer, dropout layer and sigmoid layer four-layer neural network, adopts training data training classification model to also Including: using the dropout function on the output of the bidirectional LSTM layer to prevent overfitting. 9. A phishing URL detection system based on word sequences, comprising: Conversion modules and classification training models; The conversion module is used to convert the marked URL into a word sequence vector as the training data of the training classification model; and is used to convert the unknown URL into a word sequence vector and input it into the trained classification model for labeling.