CN112764882B - Docker-based onion address and hidden service content collection method - Google Patents

Abstract

The invention discloses a Docker-based onion address and hidden service content searching method, which mainly solves the problems of long time consumption and low searching speed in the prior art. The scheme is as follows: 1) obtaining a small number of onion addresses by using two different methods, and collecting hidden service home pages corresponding to the addresses; 2) constructing a deep neural network, training the deep neural network by using a training set to obtain a classifier, and calculating the probability that the hidden service home page belongs to each category; 3) referring to the probability and the number of onion addresses in the home page of the hidden service, prejudging the number of onion addresses embedded in all pages of a corresponding website, and setting the number of collected mirror images; 4) a mirror image starting generation container is used for searching all page embedded onion addresses and content of the website corresponding to the home page; 5) repeating 1) to 4) to finish the collection of the Tor onion address and the hidden service content. The method has the advantages of less time consumption and high data collecting speed, and can be used for monitoring the onion routing network Tor.

Description

Docker-based onion address and hidden service content collection method

technical field

The invention belongs to the field of computer technology, and further relates to a method for collecting onion addresses and hidden service content, which can be used for monitoring the onion routing network Tor and perceiving the changing trend of hidden service content in the network.

Background technique

In recent years, with the development of the Internet and the popularization of various computing devices, the total amount of information in the Internet has reached an unprecedented magnitude. According to statistics, as of December 2020, the total number of web pages on the Internet has exceeded 1.8 billion, and the average daily traffic has reached nearly 6EB. The Internet can be subdivided into the surface network and the corresponding deep network. The surface web refers to websites that can be indexed by standard search engines, while the deep web refers to websites that are not indexed by standard search engines. The content in the surface network only accounts for 10% of the total content in the entire Internet, and the rest belongs to the deep network. Similar to the surface network, it also includes common mail services, online banking, social forums and other services, but requires a password or is accessed by Grant permission to access. Anonymous networks such as Tor, I2P, and Freenet in the deep network all require special software, special configuration, and authentication; or use a unique communication protocol to access them, thus obtaining good anonymity. These networks are easily abused by illegal activities, and the network is full of various sensitive content and illegal activities.

Among the various anonymous networks, Tor is the most widely used one. As of December 2020, Tor has about 7,000 active nodes, about 2.5 million active users, and more than 170,000 hidden services. Tor's anonymity relies on the relay nodes and unique communication mechanisms that volunteers and related organizations voluntarily provide in the network. When users use Tor services, the client usually uses three Tor relay nodes as communication agents, and uses public key encryption to negotiate session keys with these three nodes in turn. In the process of communication, firstly, the transmitted content is encrypted in three layers at the sending end, and then transmitted to the receiving end in turn through the selected three relay nodes. When the encrypted content reaches a node, the node performs a layer of decryption through the negotiated key, obtains the IP address of the next relay node, and continues to transmit. During the whole delivery process, each relay node can only know the IP addresses of its adjacent nodes, but cannot know the IP addresses of the sender and the receiver at the same time, thus realizing the anonymous transmission of content.

Illegal content in Tor is mostly served through hidden services. The operation of the hidden service and the user's access to the service are based on the three-hop communication mechanism, and a more sophisticated design has been carried out. In the realization of hidden service, there are five important components, namely hidden server, directory server, hidden service directory server, client, introduction node and session node.

Hidden server: Provides hidden services in Tor, such as web services, instant messaging services, etc. When the hidden service is created, it will generate public key and other related information.

Directory server: The directory server stores information such as IP addresses, bandwidth, flag bits, and fingerprints of relay nodes. After the Tor client is started, it will send a request to the directory server to obtain information about a part of the relay nodes, and randomly select qualified nodes from it to establish a communication link.

Hidden service directory server: responsible for storing the hidden service descriptor of the hidden service. The descriptor contains the information such as the introduction node and public key selected by the hidden service. Responsible for responding to user requests for access to specific hidden services sent from clients. A relay node needs to provide sufficient bandwidth and guarantee continuous operation for 96 hours before it may be determined as a hidden service directory server by the authoritative hidden service directory server. The hidden service directory server functions like the DNS server of the surface network.

Client: Provides a program for users to access Tor. During the connection process, the client selects and connects to the three-hop relay node with the help of the relevant information provided by the directory server.

Introducing nodes: Hidden services usually choose multiple introductory nodes through which to provide services. Hidden servers establish connections with them through a three-hop communication mechanism. When the client accesses the hidden service, it needs to obtain the introduction node information from the hidden service directory server through the three-hop communication mechanism, and contact the introduction node again through the three-hop communication mechanism to exchange the information of the session node.

Session node: The key communication node negotiated between the client and the hidden server. Taking the session node as the central node, the client and the hidden server establish two three-hop communication links to form six communication links for communication.

Access to hidden services requires the use of onion addresses. The onion address is calculated by performing secure hashing algorithm 1, namely SHA 1, on the 1024bits hidden service RSA public key, intercepting the first 80bits of the result, performing base 32 encoding to obtain a string of length 16bytes, and then splicing it with the .onion suffix. The working process of the hidden service is as follows:

1. The hidden service selects 3 relay nodes as its introduction nodes, and provides hidden services on these nodes. The hidden server establishes a connection with the introduction node through the 3-hop relay node.

2. The hidden server uploads the hidden service descriptor to the 6 hidden service directory servers, and the descriptor contains the selected incoming node and public key information.

3. When the client accesses the hidden service, it randomly sends a request to 6 hidden service directory servers to obtain the descriptor of the hidden service and learn the information of the imported node.

4. The client selects a session node and establishes a connection with it through the three-hop relay node.

5. The client establishes a connection with the incoming node through the three-hop relay node, and sends the relevant information of the session node.

6. If the hidden server can provide services, establish a three-hop connection to the session node and authenticate the link.

7. The client and the hidden server establish a 6-hop communication link through the session node to communicate.

Users need to know the onion address of the hidden service in order to access it. Onion addresses are only publicized in private forums, hidden service content, and only a small part is indexed by search engines on the surface or deep web. difficult to obtain on a large scale. Due to the multi-hop node communication mechanism of Tor hidden services, its access speed is slow, which limits the large-scale and rapid acquisition of onion addresses and hidden service content. How to obtain data effectively and quickly has become a research hotspot in the field of Tor research.

Song Shengnan of Beijing Jiaotong University proposed a method for collecting onion addresses and hidden service content in his master's thesis. This method deploys the collection program on a rented overseas server. Due to the low configuration of the general server, large-scale and rapid collection is required. It is expensive to purchase and deploy more servers, and after the collection is completed, a large number of files need to be downloaded from the server, which increases extra time. At the same time, because the collection program runs directly on the server, the degree of concurrency is low, so the computing power of the rented server cannot be fully utilized, resulting in a waste of resources.

The hidden service content acquisition method proposed by Ding Xiang of Southeast University in his master's thesis uses multiple threads for concurrent collection. Although it reduces the time-consuming of collection to a certain extent and improves the degree of concurrency, the scale of crawlers and agents can be expanded by expanding the scale of crawlers and agents. Increase the acquisition speed to improve efficiency, but it needs to modify the configuration file and project code, which is cumbersome and cannot be quickly expanded.

The above existing methods for obtaining onion addresses and hidden service content all use the file system of the same host, but cannot achieve file system isolation between different projects, so the robustness is low; in addition, the upgrade of project dependencies is relatively low. Trouble, you need to manually update each dependency, and you may encounter the problem of multiple dependency version conflicts. Although the project runs for a long time, the number of onion addresses and hidden service content obtained is small, and the efficiency is low.

SUMMARY OF THE INVENTION

The purpose of the present invention is to aim at the above-mentioned deficiencies of the prior art, and to propose a method for efficiently collecting Tor onion addresses and hidden service contents based on Docker virtualization technology, so as to collect more onion addresses in a short time and reduce the number of acquisitions. Hiding the time-consuming service content improves the efficiency of obtaining onion addresses and hiding service content.

In order to achieve the above object, the technical scheme of the present invention is: 1) use two different methods to obtain a small number of onion addresses, and collect the corresponding hidden service homepages of these addresses; 2) construct a deep neural network to train it using a training set to obtain a classification 3) With reference to the probability and the number of onion addresses in the homepage of the hidden service, predict the number of embedded onion addresses on all pages of the corresponding website, and set the number of collected mirrors; 4) The image is started to generate a container, and all pages of the website corresponding to the above home page are searched for embedded onion addresses and content; 5) Repeat 1) to 4) to complete the collection of Tor onion addresses and hidden service content. The specific implementation steps include the following:

(1) Get the onion address:

1a) utilize deep network search engine and surface layer network search engine to search specific keywords respectively, extract the onion address in the search result, obtain the onion address Z ₁ relevant to this keyword and indexed by the search engine;

1b) Deploy a relay node in Tor to make it a hidden service directory server. By modifying the source code, the hidden service public key in the server needs to be hashed and encoded to generate the onion address, and the corresponding public key is obtained. onion address Z ₂ ;

Store the onion address obtained by the above two methods as a file;

(2) Collect hidden service homepages corresponding to all onion addresses obtained by the two methods in (1), and obtain a collection of hidden service homepages to be classified;

(3) The darknet address text data set DUTA is preprocessed in sequence by file deduplication, data cleaning, text segmentation, and vectorization, and a multi-category text category classifier is constructed:

(3a) Perform the preprocessing of file deduplication, data cleaning, text segmentation, and vectorization on the darknet address text data set DUTA that belongs to C categories and already has category tags to form a training set word vector;

(3b) Set the parameters of the deep neural network consisting of one embedding layer, two one-dimensional convolutional layers, two one-dimensional pooling layers, one fully connected layer and one softmax layer, and pass the DUTA dataset through (3a) The training set word vector obtained after preprocessing is used as training data, and is input into the deep neural network for training to obtain a multi-category text category classifier;

(4) Perform the preprocessing of file deduplication, data cleaning, text segmentation, and vectorization for each unknown hidden service homepage in the set of hidden service homepages to be classified in (2) to obtain the test set word vector, and test Set the word vector as the input of the trained multi-category text category classifier, and use the classifier to obtain the probability P _i that the preprocessed hidden service home pages belong to the i-th category in the C categories respectively, 1≤i ≤C, and count the number N of onion addresses in each home page;

(5) Set the threshold value T=100 to distinguish the number of onion addresses embedded in the homepage of the hidden service, and set the threshold value K=500 to distinguish the number of onion addresses contained in all pages of the website corresponding to the homepage; Quantity N is compared with threshold T or probability P:

If the number of onion addresses in a homepage of a hidden service is N≥T or the probability P _i of the homepage belonging to each category is close to 1≤i≤C, it indicates that the homepage may contain content of multiple categories, and the homepage corresponds to The website is a directory type, predict that the number M of onion addresses contained in all pages in the website corresponding to the home page is greater than K, and execute (6);

If the number of onion addresses in the homepage of a hidden service is N<T or the probability Pj of the homepage belonging to the _jth category in the C category is much greater than the probability that the homepage belongs to the C category except the jth category, it means that the homepage belongs to the jth category. The home page may only belong to the jth category, predict that the number M of onion addresses contained in all pages of the website corresponding to the home page is less than the threshold K, and execute (7);

(6) Set D=50 Docker images for hidden service content collection, start these Docker images to generate 50 containers, run the built hidden service content collection code in these containers, and embed all pages in the website corresponding to each home page Collect the hidden service content corresponding to the onion address;

(7) Set D=20 Docker images for hidden service content collection, start these Docker images to generate 20 containers, run the built hidden service content collection code in these containers, and embed all pages in the website corresponding to each home page The hidden service content corresponding to the onion address is collected.

(8) Count the total number W of hidden service contents collected in (6) and (7), parse and extract all embedded onion addresses Z ₃ from these hidden service contents, and compare these onion addresses with (1a), ( 1b) The obtained onion addresses are added, and the total amount of the obtained onion addresses is: Z=Z ₁ +Z ₂ +Z ₃ ;

(9) Set the number of onion target addresses to be obtained as X and the number of hidden service content targets as Y according to actual needs, and compare the sizes of X and Z and Y and W:

If X>Z or Y>W, it indicates that the currently obtained number of onion addresses Z or the number of hidden service contents W has not reached the set target, repeat (1)-(8) until the number of onion addresses Z and the number of hidden service contents W reach the set number;

If X≤Z and Y≤W are satisfied at the same time, it indicates that the currently obtained number Z of onion addresses and the number W of hidden service contents have reached the set number, and the data collection is stopped.

Compared with the prior art, the present invention has the following advantages:

First, it is conducive to the rapid deployment of the development environment to the production environment.

Since the invention uses Docker technology to collect hidden service content, it avoids the tedious steps of building a development environment in the existing method. It only needs to download the relevant dependent images for configuration, and then package them to form a project image, and the project is directly in the production environment. The image is deployed, avoiding the tedious steps of migrating the development environment to the production environment.

Second, the time for each container to complete data collection is relatively similar, which further shortens the collection time.

The present invention predicts the probability that the hidden service home page corresponding to the onion address belongs to each set category, counts the number of onion addresses in each hidden service home page, and predicts that all pages in the website corresponding to the hidden service home page contain the number of onion addresses. By setting the number of Docker images, the time required for each container to complete the collection of onion addresses in all pages is relatively similar, avoiding the excessive number of onion addresses in a website that needs to be processed by a container and causing the collection process to take too long. Collection efficiency of onion addresses and hidden service content.

Description of drawings

Fig. 1 is the realization general flow chart of the present invention;

FIG. 2 is a sub-flow chart of acquiring hidden service content in the present invention.

Detailed ways

The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

Referring to Fig. 1, the implementation steps of the present invention are as follows:

Step 1. Collect onion addresses through two methods.

Since the use of Tor requires the use of special software and configuration, in order to allow users to use the Tor service more conveniently, Tor2web was established by volunteers in 2008. The project provides users with specific keywords under the premise of sacrificing some anonymity. For example, .sh, .guide, .city, users need to splicing keywords after the onion address represented by the project, and then they can use a common browser to access the hidden service website corresponding to the onion address. The specific implementation is as follows:

1.1) utilize deep web search engine and surface web search engine to search specific keywords respectively, extract the onion address in the search result, obtain the onion address Z ₁ relevant to this keyword and indexed by the search engine;

1.2) By modifying the Tor source code, perform secure hashing algorithm 1, ie SHA 1, on the hidden service public key in the server, and perform base 32 encoding on the first 80 bits of the calculation result to obtain a string with a length of 16 bytes. After the string Splicing .onion to get onion addresses, although this method is more complicated and time-consuming, it is an indispensable way to collect onion addresses, and can obtain onion addresses Z ₂ that are not indexed by search engines or other hidden service content.

Store the onion addresses obtained by the above two methods as a file.

In this example, relay nodes are deployed in the Tor anonymous network, providing bandwidth of more than 200KB/S, obtaining the fast Fast and stable stable flags and running for 96 hours to obtain the hidden service directory server flag.

Step 2: Write the homepage collection code of the hidden service, obtain the dependent image, and integrate and package the hidden service homepage collection code and the dependent image to form a project image.

2.1) Write hidden service homepage collection code;

2.2) Obtain the Ubuntu operating system image and the Tor image respectively, configure the Tor image, and set the network proxy;

2.3) Integrate and package the hidden service home page collection code with the above-mentioned Ubuntu operating system image and Tor image, to form a project image for the hidden service home page collection;

2.4) Create a Docker configuration file and specify the operations that need to be completed in the process of starting the project image to generate the container: copy the file containing the onion address obtained in step 1 into the container, and run the hidden service home page to obtain the code;

2.5) Start the project image, read the created Docker configuration file, and complete the set operation;

2.6) Run the hidden service home page acquisition code, collect the hidden service home pages corresponding to the onion addresses in the onion address file obtained in step 1, and obtain the set of hidden service home pages to be classified.

Step 3: Preprocess the darknet address text data set DUTA in sequence.

3.1) File deduplication:

Calculate the MD5 value of the fifth-generation information digest of each hidden service content file in the DUTA data set in turn, and store it in the set. If the MD5 value of a file already exists in the set, it indicates that this file and the file whose MD5 value calculation has been completed The content contained is exactly the same. For the same content, only one file to save the page content is reserved, and redundant files are deleted to obtain a set of deduplicated files;

3.2) Data cleaning:

Use regular expressions to remove the useless decoration content in each hidden service content file of the deduplication file set, and obtain the file set after data cleaning;

3.3) Text segmentation:

Use the natural language processing tool NLTK to perform text segmentation on the hidden service content of each file in the file collection after data cleaning, that is, segment each sentence in the hidden service content to get all the words that make up the sentence, and then remove the words in these words. adverbs, prepositions, conjunctions, auxiliary verbs, and retain the content words such as nouns, verbs, adjectives, and quantifiers;

3.4) Vectorization:

Use the vector generator word2Vec to encode the real words obtained in 3.3) to obtain vectorized real words and form the training set word vector.

Step 4, build a multi-category text category classifier.

4.1) Establish a deep neural network consisting of an embedding layer, the first convolution layer, the first pooling layer, the second convolution layer, the second pooling layer, the fully connected layer and the softmax layer. The parameter settings are as follows:

Embedding layer: the input dimension is 200, and the output dimension is 128;

The first convolution layer: the number of convolution kernels is 64, the size of the convolution kernel is 7, the sliding step size is 2, and the activation function is set to relu;

The first pooling layer, the pooling window size is 7, and the pooling step size is 2;

The second convolution layer: the number of convolution kernels is 64, the size of the convolution kernel is 3, the sliding step size is 2, and the activation function is set to relu;

The second pooling layer, the pooling window size is 20, and the pooling step size is 1;

Fully connected layer: the activation function is set to relu;

Softmax layer: The number of output categories is 8.

4.2) Use the training set word vector obtained after the preprocessing of the DUTA data set in step 3 as the training data, and input it into the deep neural network established in 4.1) for training to obtain a multi-category text category classifier:

4.2.1) set the loss function of the deep neural network to be the cross entropy function Ceem, and set the target output threshold data to be R _i ;

4.2.2) Each layer in the deep neural network model operates on the input training data to obtain the output data E _i , 1≤i≤C;

4.2.3) Calculate the deviation between the output data E _i and the target output data R _i given by the DUTA training set through the Ceem function, and use the Adam optimization algorithm to update the parameters of the deep neural network to continuously reduce E _i and R _i degree of deviation;

4.2.4) Repeat steps 4.2.2) to 4.2.3) for 1000 rounds of training to complete the optimization of the parameters of the deep neural network, and obtain a multi-category text category classifier.

Step 5, using a multi-category text category classifier to calculate the probability that the home page of the hidden service belongs to each of the C categories.

5.1) Perform the same preprocessing as step 3 on the hidden service home page obtained in step 2 to obtain the test set word vector;

5.2) The test set word vector obtained after processing in 5.1) is used as the input of the multi-category text category classifier, and the i-th output result zi is obtained through the classifier operation, _i represents the text category number, 1≤i≤C, C represents the number of text categories;

5.3) According to the operation result zi of the classifier, calculate the probability P _i of the hidden service homepage belonging to the _ith category in the C categories:

Step 6, obtain hidden service content.

Referring to Fig. 2, the concrete realization of this step is as follows:

6.1) Count the number N of onion addresses in the home page of each hidden service obtained in step 2;

6.2) Set the threshold value T=100 to distinguish the number of onion addresses embedded in the homepage of the hidden service, and set the threshold value K=500 to distinguish the number of onion addresses contained in all pages in the website corresponding to the homepage;

6.3) According to the statistical number N of onion addresses and the set threshold value T of the number of onion addresses, and the probability P _i of the hidden service homepage belonging to the i-th category in the C categories, determine the content to be searched:

If the number of onion addresses in the homepage of a hidden service N≥T or the probability P _i of the homepage belonging to each category is similar, indicating that the homepage may contain content of multiple categories, then the website corresponding to the homepage is a directory type. It is predicted that the number M of onion addresses contained in all pages of the website corresponding to the home page is greater than K, and step 6.4) is executed;

If the number of onion addresses in the homepage of a hidden service is N<T or the probability Pj of the homepage belonging to the _jth category in the C category is much greater than the probability that the homepage belongs to the C category except the jth category, it indicates that the homepage belongs to the jth category. It may only belong to the jth category, then it is predicted that the number M of onion addresses contained in all pages of the website corresponding to the home page is less than the threshold K, and step 6.5) is executed;

6.4) Set D = 50 Docker images for hidden service content collection, start these Docker images to generate 50 containers, run the built hidden service content collection code in these containers, and use the method in step 7 for each homepage corresponding website. Collect the hidden service content corresponding to the embedded onion address in all pages, and obtain the hidden service content W;

6.5) Set D = 20 Docker images for hidden service content collection, start these Docker images to generate 20 containers, run the built hidden service content collection code in these containers, and use the method in step 7 for each homepage corresponding website. Collect the hidden service content corresponding to the embedded onion address in all pages, and obtain the hidden service content W.

Step 7: Collect hidden service content corresponding to the embedded onion addresses in all pages of the website corresponding to each home page.

7.1) Write code to collect the hidden service content corresponding to the embedded onion address on all pages in the website corresponding to the hidden service home page;

7.2) Obtain the Ubuntu operating system image and the Tor image respectively, configure the Tor image, and set the network proxy;

7.3) Integrate and package the code written in 7.1) with the two images of the above-mentioned Ubuntu operating system image and Tor image to form a project image;

7.4) Create a Docker configuration file and specify the operations that need to be completed in the process of starting the project image to generate the container: run the code written in 7.1);

7.5) Start the project image, read the created Docker configuration file, and complete the set operation;

7.6) Run the code written in 7.1), and collect the hidden service content corresponding to the onion address embedded in all pages in the website corresponding to each home page obtained in step 2.

Step 8: Parse and extract all embedded onion addresses in the hidden service content, and count the total number of onion addresses and hidden service content currently obtained.

8.1) Read the hidden service content W obtained in step 6;

8.2) According to the format characteristics of the onion address: that is, the prefix is http or https, and the suffix is .onion. Between the prefix and the suffix is a string composed of 16-bit letters or numbers, using a regular expression: https*://\ w+\.onion, perform regular matching on the hidden service content obtained by reading, get all the embedded onion addresses Z ₃ in the hidden service content, and compare these embedded onion addresses with the onion addresses obtained in 1.1) and 1.2) Add, the total amount of onion addresses obtained is: Z=Z ₁ +Z ₂ +Z ₃ .

Step 9: Determine whether the number of onion addresses and the number of hidden service contents reach the set target.

9.1) According to actual needs, set the number of onion target addresses to be obtained as X and the number of hidden service content targets as Y;

9.2) Compare the magnitudes of X with Z and Y with W:

If X>Z or Y>W, it means that the currently obtained number of onion addresses Z or the number of hidden service contents W has not reached the set target, repeat steps 1 to 8 until the number of onion addresses Z and the number of hidden service contents W reach the set value a fixed quantity;

If X≤Z and Y≤W are satisfied at the same time, it means that the currently obtained number of onion addresses Z and the number of hidden service contents W have reached the set number, stop data collection, and obtain the final number of onion addresses and hidden services searched out. amount of content.

The above description is only a specific example of the present invention, and does not constitute any limitation to the present invention. Obviously, for those skilled in the art, after understanding the content and principles of the present invention, they may not deviate from the principles and structures of the present invention. Under the circumstance of the present invention, various modifications and changes in form and details are made, but these modifications and changes based on the idea of the present invention are still within the scope of protection of the claims of the present invention.

Claims (8)

1. A method for collecting onion addresses and hidden service contents based on Docker is characterized by comprising the following steps:

(1) obtaining an onion address:

1a) respectively searching specific keywords by using a deep network search engine and a surface network search engine, extracting onion addresses in search results to obtain onion addresses Z which are related to the keywords and indexed by the search engines ₁ ；

1b) Arranging a relay node in Tor to enable the relay node to become a hidden service directory server, and performing Hash calculation and encoding required by generating an onion address on a hidden service public key in the server in a source code modification mode to obtain an onion address Z corresponding to the public key ₂ ；

Storing the onion addresses obtained by the two methods as files;

(2) collecting hidden service home pages corresponding to all the onion addresses obtained by the two methods in the step (1) to obtain a set of hidden service home pages to be classified;

(3) sequentially carrying out file duplicate removal, data cleaning, text word segmentation and vectorization on a dark web address text data set DUTA to construct a multi-classification text category classifier:

(3a) sequentially carrying out preprocessing of file duplicate removal, data cleaning, text word segmentation and vectorization on a dark web address text data set DUTA which belongs to C categories and has category marks to obtain a training set word vector;

(3b) setting parameters of a deep neural network sequentially consisting of an embedding layer, a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a full-link layer and a softmax layer, inputting training set word vectors obtained after the DUTA data set is preprocessed in the step (3a) as training data into the deep neural network for training to obtain a multi-classification text class classifier;

(4) sequentially carrying out file duplication removal, data cleaning, text word segmentation and vectorization preprocessing on each unknown hidden service home page in the hidden service home page set to be classified in the step (2) to obtain a test word set vector, taking the test word set vector as the input of a trained multi-classification text class classifier, and obtaining the probability P that the preprocessed hidden service home pages respectively belong to the ith class in the C classes by using the classifier _i I is more than or equal to 1 and less than or equal to C, and the number N of onion addresses in each home page is counted;

(5) setting a threshold value T of the number of onion addresses embedded in a home page of the hidden service to be 100, and setting a threshold value K of the number of onion addresses contained in all pages in a website corresponding to the home page of the hidden service to be 500; comparing the number N of onion addresses in each hidden service home page to a threshold T or probability P:

if the number N of onion addresses in a certain hidden service home page is more than or equal to T or the probability P that the home page belongs to each category _i If i is more than or equal to 1 and less than or equal to C is close, the home page possibly comprises contents of a plurality of categories, the website corresponding to the home page is of a directory type, the number M of onion addresses contained in all pages in the website corresponding to the home page is judged to be more than K, and the step (6) is executed;

if the number N of onion addresses in a certain hidden service home page is less than T or the probability P that the home page belongs to the jth category in the C categories _j If the probability that the home page belongs to the C categories except the jth category is far greater than the probability that the home page belongs to the jth category, the home page is possibly only the jth category, the number M of onion addresses contained in all pages in a website corresponding to the home page is judged to be smaller than a threshold value K, and then the step (7) is executed;

(6) setting D as Docker images collected by 50 hidden service contents, starting the Docker images to generate 50 containers, running constructed hidden service content collection codes in the containers, and collecting the hidden service contents corresponding to all page embedded onion addresses in a website corresponding to each home page;

(7) setting D as 20 Docker images collected by hidden service contents, starting the Docker images to generate 20 containers, running constructed hidden service content collection codes in the containers, and collecting the hidden service contents corresponding to all page embedded onion addresses in a website corresponding to each home page;

(8) counting the total number W of the hidden service contents collected in (6) and (7), and analyzing and extracting all the embedded onion addresses Z from the hidden service contents ₃ And adding the onion addresses with the onion addresses obtained in (1a) and (1b) to obtain the total number of the onion addresses as follows: z ═ Z ₁ +Z ₂ +Z ₃ ；

(9) Setting the number of onion target addresses to be obtained as X and the number of hidden service content targets as Y according to actual needs, and comparing the sizes of X and Z and Y and W:

if X is larger than Z or Y is larger than W, the number Z of the onion addresses obtained at present or the number W of the hidden service contents does not reach the set target, and repeating the steps (1) - (8) until the number Z of the onion addresses and the number W of the hidden service contents reach the set number;

if X is less than or equal to Z and Y is less than or equal to W, the number Z of the onion addresses and the number W of the hidden service contents which are obtained at present are indicated to reach the set number, and then data collection is stopped.

2. The method of claim 1, wherein the step (2) of collecting the hidden service header page corresponding to the onion address obtained in the step (1) is to use a Docker technique to obtain the hidden service header page, and is implemented as follows:

(2a) compiling hidden service home page collection codes;

(2b) respectively acquiring a Ubuntu operating system image and a Tor image, configuring the Tor image and setting a network agent;

(2c) integrating and packaging a hidden service home page collecting code and the two images of the Ubuntu operating system image and the Tor image to form an item image for collecting the hidden service home page;

(2d) creating a Docker configuration file, and specifying operations to be completed in the process of starting the project image to generate the container: copying the file containing the onion address obtained in the step (1) to the container, and operating a hidden service home page acquisition code;

(2e) starting the project mirror image, reading the established Docker configuration file, and finishing the set operation;

(2f) and (3) operating the hidden service home page acquisition code, and collecting the hidden service home page corresponding to the onion address in the onion address file obtained in the step (1) to obtain a hidden service home page set to be classified.

3. The method according to claim 1, wherein in the step (3a), the dark web address text data set DUTA is preprocessed by document de-duplication, data cleaning, text word segmentation and vectorization in sequence, and the method is implemented as follows:

(3a1) file deduplication:

sequentially calculating the MD5 value of a fifth generation information summary of each hidden service content file in the DUTA data set, storing the values in the set, if the MD5 value of a certain file exists in the set, indicating that the content of the file is completely the same as the content of the file which completes the MD5 value calculation, only reserving a file for storing the page content for the same content, deleting redundant files, and obtaining a duplicate removal file set;

(3a2) data cleaning:

removing useless modification content in each hidden service content file of the duplicate removal file set by using a regular expression to obtain a data-cleaned file set;

(3a3) text word segmentation

Using a natural language processing tool NLTK to perform text segmentation on hidden service contents of each file in a file set after data cleaning, namely segmenting each sentence in the hidden service contents to obtain all words forming the sentence, removing adverbs, prepositions, conjunctions and auxiliary verbs in the words, and keeping real words of nouns, verbs, adjectives and quantitative words in the words;

(3a4) vectorization

And (4) using a vector generator word2Vec to encode the real words obtained in the step (3c) to obtain vectorized real words and form training set word vectors.

4. The method of claim 1, wherein the setting of the structural parameters of the deep neural network in (3b) is as follows:

embedding layer: an input dimension of 200 and an output dimension of 128;

first convolutional layer: the number of convolution kernels is 64, the size of the convolution kernels is 7, the sliding step length is 2, and the activation function is set to relu;

the first largest pooling layer, the size of a pooling window is 7, and the pooling step length is 2;

a second convolutional layer: the number of convolution kernels is 64, the size of the convolution kernels is 3, the sliding step length is 2, and the activation function is set to relu;

the second largest pooling layer, the size of the pooling window is 20, and the pooling step length is 1;

full connection layer: the activation function is set to relu;

softmax layer: the number of output classes is 8 classes.

5. The method of claim 1, wherein the deep neural network is trained in (3b) as follows:

(3b1) setting a loss function of the deep neural network as a cross entropy function Ceem, and setting target output threshold data R _i ；

(3b2) Each layer in the deep neural network model operates input training data to obtain output data E _i ，1≤i≤C；

(3b3) Computing output data E by Ceem function _i Target output data R given with DUTA training set _i And updating the parameters of the deep neural network by using an Adam optimization algorithm, and continuously reducing E _i And R _i The degree of deviation of (a);

(3b4) repeating the steps 3b1) -3 b3) for 1000 rounds of training, completing the optimization of the deep neural network parameters, and obtaining the multi-classification text category classifier.

6. The method of claim 1, wherein (4) the probability P that the hidden top of service page belongs to each of the C classes is calculated using a multi-class text class classifier _i ：

(4.1) carrying out the same pretreatment as the pretreatment of the (3a) on the hidden service home page obtained in the step (2) to obtain a test set word vector;

(4.1) using the test set word vector obtained after the processing of the step 4.1) as the input of a multi-classification text category classifier, and obtaining the ith output result z through the operation of the classifier _i I represents the number of text categories, i is more than or equal to 1 and less than or equal to C, and C represents the number of the text categories;

(4.1) operation result z according to the classifier _i Calculating the probability P that the hidden service home page respectively belongs to the ith category in the C categories _i ：

7. The method of claim 1, wherein the hidden service contents corresponding to all page embedded onion addresses in the website corresponding to each home page are collected in step (6) as follows:

(6a) compiling codes, and collecting hidden service contents corresponding to the onion addresses embedded in all pages in the website corresponding to the hidden service home page;

(6b) respectively acquiring a Ubuntu operating system image and a Tor image, configuring the Tor image and setting a network agent;

(6c) integrating and packaging the code written in the step (6a) and the Ubuntu operating system mirror image and the Tor mirror image to form a project mirror image;

(6d) creating a Docker configuration file, and specifying operations to be completed in the process of starting the project image to generate the container: running (6a) the written code;

(6e) starting the project mirror image, reading the established Docker configuration file, and finishing the set operation;

(6f) and (3) running the code compiled in the step (6a), and collecting hidden service contents corresponding to all page embedded onion addresses in the website corresponding to each home page obtained in the step (2) to obtain a hidden service content set.

8. The method of claim 1, wherein (8) parsing extracts all the embedded onion addresses in the hidden service content as follows:

(8a) reading hidden service content;

(8b) according to the format characteristics of the onion address: i.e. the prefix is http or https, the suffix is onion, between the prefix and the suffix, is a string of 16 letters or numbers, using a regular expression: and (3) performing regular matching on the read hidden service content to obtain all embedded onion addresses in the hidden service content.

Images