CN106202077B - Task distribution method and device - Google Patents

Abstract

The embodiment of the invention discloses a task distribution method and a task distribution device, which are used for improving the crawling efficiency of a web crawler. The method provided by the embodiment of the invention comprises the following steps: analyzing a Uniform Resource Locator (URL) of a first page and a hash value of a parent page of the URL of the first page into a task list; judging whether the URL of the first page is crawled or not; when the URL of the first page is not crawled, determining whether the first page and the second page are adjacent pages according to a sub domain name in the URL of the first page and a hash value of a parent page of the URL of the first page; when the first page and the second page are determined not to be adjacent pages, allocating the first page to a new thread; when the first page and the second page are determined to be adjacent pages, the first page is distributed to a thread where the second page is located; and controlling to execute the downloading task on the distributed threads according to the time sequence.

Description

A task distribution method and device

technical field

The present invention relates to the field of communication technologies, and in particular, to a task distribution method and device.

Background technique

In the web crawler technology, how to obtain the target URL (Uniform Resource Locator, Uniform Resource Locator) web page collection content in batches, at the same time, the web crawler itself is not mistaken by the server where the target web page collection is located as a malicious web crawler, which is a big problem. . To ensure the efficiency of crawling web pages, web crawlers need to open multiple download threads to concurrently send access requests to the web pages in the URL task list. However, the target website server will determine whether the address request is sent by a malicious web crawler by detecting the sequence and frequency of URL access requests sent from the same IP (Internet Protocol, Internet Protocol) address. If the page set in the URL access request sequence of a certain IP address within a certain short period of time conforms to the definition of adjacent pages by the website, it is determined that the IP address is running a malicious web crawler. To avoid this from happening, various techniques or strategies are now being employed.

The prior art provides a distributed web crawler technology that uses priority points to sort a task list. When using distributed web crawler to crawl web pages, it sets a fixed IP address for each node, and sets the maximum number of concurrently executed threads for each node to 3 or less, and executes URL requests on each thread. At the same time, a column of data items named points is added to the task list, the points are determined according to the analysis of the URL or content of the web page, and the task queue is reordered according to the points from large to small. With this approach, the amount of IP address resources required is greatly reduced. At the same time, due to the rearrangement of the task list, the access order of adjacent pages with the same parent page may be disrupted, and the maximum number of concurrent crawling threads from each IP address is 3, and the maximum number of access requests issued is With a value of 3, these access requests are time-controlled so that they are not too dense, so they are not easily misidentified as malicious web crawlers.

With this method, the number of concurrent threads of each node is too small, resulting in a low crawling efficiency (the number of web pages crawled per unit time) of the web crawler; and there are adjacent pages that are intensively accessed in the same time period and are identified as malicious networks The possibility of reptiles.

SUMMARY OF THE INVENTION

The present invention provides a task distribution method and device for improving the crawling efficiency of a web crawler.

A first aspect of the present invention provides a task distribution method, comprising:

Parse the hash value of the URL of the first page and the parent page of the URL of the first page into the task list;

Determine whether the URL of the first page has been crawled;

When the URL of the first page is not crawled, determine the relationship between the first page and the first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page. Whether the two pages are adjacent pages;

when it is determined that the first page and the second page are not adjacent pages, assigning the first page to a new thread;

When it is determined that the first page and the second page are adjacent pages, allocating the first page to the thread where the second page is located;

Execute the download task on the allocated thread according to the timing control.

In combination with the first aspect of the present invention, in the first implementation manner of the first aspect of the present invention, the judging whether the URL of the first page has been crawled specifically includes:

Determine whether the URL of the first page has been crawled according to the URL list of the crawled pages, and if so, delete the URL of the first page from the task list.

In combination with the first aspect of the present invention, in the second implementation manner of the first aspect of the present invention, the hash of the parent page based on the subdomain name in the URL of the first page and the URL of the first page The hash value to determine whether the first page and the second page are adjacent pages specifically includes:

Determine whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and determine whether the hash value of the parent page of the URL of the first page is the same as the URL of the second page. Whether the hash value of the parent page is the same;

When the subdomain names are the same or the hash values are the same, determining that the first page and the second page are adjacent pages;

When the subdomain names are not the same and the hash values are not the same, it is determined that the first page and the second page are not adjacent pages.

With reference to the first aspect of the present invention, or the first implementation manner of the first aspect, or the second implementation manner of the first aspect, in the third implementation manner of the first aspect of the present invention, the third The allocation of a page to a new thread includes:

The first page is allocated to a new thread, and the thread number corresponding to the new thread is written into the thread number item of the URL of the first page; the new thread saves the URL of the first page. The hash value of the parent page and the name of the URL of the first page;

Delete the URL of the first page from the task list and write it into the URL list of crawled pages.

With reference to the first aspect of the present invention, or the first implementation manner of the first aspect, or the second implementation manner of the first aspect, in the fourth implementation manner of the first aspect of the present invention, the The allocation of a page to the thread where the second page is located specifically includes:

Allocate the first page to the thread where the second page is located, and write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of the first page; The thread where the two pages are located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page;

Delete the URL of the first page from the task list and write it into the URL list of crawled pages.

A second aspect of the present invention provides a task distribution device, comprising:

a parsing unit for parsing the URL of the first page and the hash value of the parent page of the URL of the first page into the task list;

a judging unit for judging whether the URL of the first page has been crawled;

A determining unit, configured to determine the said first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page when the URL of the first page is not crawled Whether the first page and the second page are adjacent pages;

a first allocation unit, configured to allocate the first page to a new thread after determining that the first page and the second page are not adjacent pages;

a second allocation unit, configured to allocate the first page to the thread where the second page is located after determining that the first page and the second page are adjacent pages;

The execution unit is used to execute the download task on the allocated thread according to the timing control.

In combination with the second aspect of the present invention, in the first implementation manner of the second aspect of the present invention, the judging unit is specifically configured to judge whether the URL of the first page has been crawled according to the URL list of the crawled page , if yes, delete the URL of the first page from the task list.

With reference to the second aspect of the present invention, in a second implementation manner of the second aspect of the present invention, the determining unit specifically includes:

a judging module for judging whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page when the URL of the first page is not crawled, and judging the first page Whether the hash value of the parent page of the URL of the page is the same as the hash value of the parent page of the URL of the second page;

a first determining module, configured to determine that the first page and the second page are adjacent pages when the subdomain names are the same or the hash values are equal;

A second determining module, configured to determine that the first page and the second page are not adjacent pages when the subdomain names are not the same and the hash values are not the same.

With reference to the second aspect of the present invention, or the first implementation manner of the second aspect, or the second implementation manner of the second aspect, in the third implementation manner of the second aspect of the present invention, the first distribution unit Specifically include:

The first distribution module is used to assign the first page to a new thread, and write the thread number corresponding to the new thread into the thread number item of the URL of the first page; the new thread saves all The hash value of the parent page of the URL of the first page and the name of the URL of the first page;

The first deletion and writing module is used to delete the URL of the first page from the task list and write it into the URL list of the crawled pages.

With reference to the second aspect of the present invention, or the first implementation manner of the second aspect, or the second implementation manner of the second aspect, in the fourth implementation manner of the second aspect of the present invention, the second allocating unit Specifically include:

A second distribution module, configured to allocate the first page to the thread where the second page is located, and write the thread corresponding to the thread where the second page is located in the thread number item of the URL of the first page Thread number; The thread where the second page is located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page;

The second deletion and writing module is configured to delete the URL of the first page from the task list and write it into the URL list of the crawled pages.

It can be seen from the above technical solutions that the embodiment of the present invention has the following advantages: the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page are parsed into the task list; Whether the URL of the first page has been crawled; and when the URL of the first page has not been crawled, according to the subdomain name in the URL of the first page and the parent of the URL of the first page The hash value of the page determines whether the first page and the second page are adjacent pages; when it is determined that the first page and the second page are not adjacent pages, the first page is allocated to In the new thread, and perform the download task according to the timing control; when it is determined that the first page and the second page are adjacent pages, the first page is allocated to the thread where the second page is located, and Execute download tasks according to timing control. Because the present invention reasonably allocates the URL tasks of the distributed web crawler, and allocates adjacent web pages to the same thread, it ensures that each adjacent page assigned to the same IP address is downloaded serially, and the non-adjacent pages are downloaded in series. It is allocated to different threads for concurrent download; therefore, in the case of a limited number of IPs, it not only improves the crawling efficiency of web crawlers, but also effectively prevents web crawlers with limited IP numbers from being identified as malicious web crawlers.

Description of drawings

1 is a schematic flowchart of an embodiment of a task distribution method provided by the present invention;

2 is a schematic flowchart of another embodiment of the task distribution method provided by the present invention;

3 is a schematic flowchart of another embodiment of the task distribution method provided by the present invention;

4 is a schematic flowchart of another embodiment of the task distribution method provided by the present invention;

5 is a schematic structural diagram of an embodiment of a task distribution device provided by the present invention;

6 is a schematic structural diagram of another embodiment of the task distribution device provided by the present invention;

7 is a schematic structural diagram of another embodiment of the task distribution apparatus provided by the present invention;

8 is a schematic structural diagram of another embodiment of the task distribution apparatus provided by the present invention;

FIG. 9 is a schematic structural diagram of another embodiment of the task distribution apparatus provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present invention.

It should be understood that although the terms first, second, etc. may be used to describe each user or terminal in this embodiment of the present invention, the user or terminal should not be limited by these terms. These terms are only used to distinguish users or terminals from one another. For example, without departing from the scope of the embodiments of the present invention, the first user may also be referred to as the second user, and similarly, the second user may also be referred to as the first user; similarly, the second user may also be referred to as It is called a third user, etc., which is not limited in this embodiment of the present invention.

First, some abbreviations and key terms involved in the present invention are defined:

Web crawler: A program or script that automatically crawls information from the World Wide Web according to certain rules.

URL: On the WWW, each information resource has a uniform and unique address on the Internet. The address is called URL (Uniform Resource Locator), which is the uniform resource locator of WWW, which refers to the network address. .

Adjacent pages: Different websites have different definitions of adjacent pages, which usually include: different pages pointed to by multiple URL links contained in the same web page, and pages with the same subdomain name (for example: http://www.very .com/entries/793336/ and http://www.very.com/entries/792442/).

Timing control in the crawler thread: Set the time interval of the URL access request sent by the web crawler download thread to the target website. The time interval is greater than the time interval required for people to click on the webpage, and must be a random value.

IP address; a 32-bit address assigned to each host connected to the Internet.

An embodiment of the present invention provides a task distribution method, which is mainly a method performed by a task distribution device. Please refer to FIG. 1. An embodiment of the task distribution method provided by the present invention includes:

101. Parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list;

It should be noted that the parser is responsible for transmitting the data type to the task list, specifically parsing the obtained uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list ; Therefore, the data type is no longer just the parsed URL string, but contains a binary array corresponding to the URL string and the hash value of the parent page of the URL. It can be understood that, in the task list, the format of the task list is also changed from only listing the URL string to not only listing the URL string, but also listing the hash value that identifies the parent page of the URL.

102. Determine whether the URL of the first page has been crawled;

It should be noted that when a new URL task in the task list enters the task waiting queue, deduplication processing is performed first; therefore, it is first necessary to determine whether the URL of the first page has been crawled.

103. When the URL of the first page is not crawled, determine the first page and the second page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page Whether it is an adjacent page;

When the URL of the first page is not crawled, determine whether the first page and the second page are the same according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page. an adjacent page; wherein the second page is a page in a running thread that is different from the first page.

104. When it is determined that the first page and the second page are not adjacent pages, allocate the first page to a new thread;

When it is determined that the first page and the second page are not adjacent pages, the first page is allocated to a new thread.

105. When it is determined that the first page and the second page are adjacent pages, allocate the first page to the thread where the second page is located;

When it is determined that the first page and the second page are adjacent pages, the first page is allocated to the thread where the second page is located.

106. Execute the download task on the allocated thread according to the time sequence control.

In the embodiment of the present invention, the URL of the first page and the hash value of the parent page of the URL of the first page are parsed into the task list; and it is judged whether the URL of the first page has been crawled and when the URL of the first page is not crawled, determine the first page and the first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page. Whether the two pages are adjacent pages; when it is determined that the first page and the second page are not adjacent pages, the first page is allocated to a new thread; when it is determined that the first page and the second page are adjacent pages When the page is adjacent, the first page is allocated to the thread where the second page is located, and the download task is performed on the allocated thread according to the timing control. Because the present invention reasonably allocates the URL tasks of the distributed web crawler, and allocates adjacent web pages to the same thread, it ensures that each adjacent page assigned to the same IP address is downloaded serially, and the non-adjacent pages are downloaded in series. It is allocated to different threads for concurrent download; therefore, in the case of a limited number of IPs, it not only improves the crawling efficiency of web crawlers, but also effectively prevents web crawlers with limited IP numbers from being identified as malicious web crawlers.

Referring to FIG. 2, another embodiment of the task distribution method provided by the present invention includes:

201. Parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into a task list;

202. Determine whether the URL of the first page has been crawled according to the URL list of the crawled page, and if so, delete the URL of the first page from the task list;

It should be noted that it is determined whether the URL of the first page has been crawled according to the URL list of the crawled pages. If the URL of the first page has been crawled, the URL of the first page will be removed from the task. Deleted from the list.

203. When the URL of the first page is not crawled, determine the first page and the second page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page Whether it is an adjacent page;

204. When it is determined that the first page and the second page are not adjacent pages, allocate the first page to a new thread;

205. When it is determined that the first page and the second page are adjacent pages, allocate the first page to the thread where the second page is located;

It should be noted that, the specific processes of steps 201 and 203 to 205 may correspond to steps 101 and 103 to 105 in the embodiment shown in FIG. 1 respectively, which will not be repeated here.

206. Execute the download task on the allocated thread according to the time sequence control.

It should be noted that the download task is executed according to the timing control through the download thread of the web crawler, and the content of the downloaded page is parsed, and the parsed new URL and the hash value of the parent page of the URL are sent to the URL task list. After regenerating a new URL task waiting queue.

In the embodiment of the present invention, it is determined whether the URL of the first page has been crawled according to the URL list of the crawled pages, and if so, the URL of the first page is deleted from the task list, so as to improve the web crawler's Crawl efficiency.

Referring to FIG. 3, another embodiment of the task distribution method provided by the present invention includes:

301. Parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into a task list;

302. Determine whether the URL of the first page has been crawled according to the URL list of the crawled page, and if so, delete the URL of the first page from the task list;

303. When the URL of the first page is not crawled, determine whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and determine whether the parent page of the URL of the first page is the same. Whether the hash value is the same as the hash value of the parent page of the URL of the second page;

It should be noted that when the URL of the first page is not crawled, it is determined whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and the URL of the first page is determined. Whether the hash value of the parent page of the second page is the same as the hash value of the parent page of the URL of the second page; wherein, the second page is a page in a running thread different from the first page, and the first page The subdomain name of the URL and the hash value of the parent page of the URL of the first page are respectively compared with other pages in all running threads, and it is judged whether they are the same.

304. When the subdomain name is the same or the hash value is the same, determine that the first page and the second page are adjacent pages;

It should be noted that, if it is determined through step 303 that the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, or the hash value of the parent page of the URL of the first page is the same as that of the second page If the hash value of the parent page of the URL is equal, it is determined that the first page and the second page are adjacent pages.

305. When the subdomain names are not the same and the hash values are not the same, determine that the first page and the second page are not adjacent pages;

It should be noted that, if it is determined through step 303 that the subdomain name of the URL of the first page is different from the subdomain name of the URL of the second page, and the hash value of the parent page of the URL of the first page is the same as that of the second page URL If the hash values of the parent pages of the URLs of the pages are not equal, it is determined that the first page and the second page are not adjacent pages.

306. When it is determined that the first page and the second page are not adjacent pages, allocate the first page to a new thread;

307. When it is determined that the first page and the second page are adjacent pages, allocate the first page to the thread where the second page is located;

308. Execute the download task on the allocated thread according to the time sequence control.

It should be noted that the specific processes of steps 301 , 302 , 306 to 308 may correspond to steps 201 , 202 , 204 to 206 in the embodiment shown in FIG. 2 , respectively, which will not be repeated here.

In the embodiment of the present invention, when the URL of the first page is not crawled, the subdomain name of the URL of the first page is judged whether the subdomain name of the URL of the second page is the same as the subdomain name of the URL of the second page. Whether the first page and the second page are adjacent pages is determined by whether the hash value of the parent page of the URL is the same as the hash value of the parent page of the URL of the second page; and the subdomain name is the same or When the hash value is equal, it is determined that the first page and the second page are adjacent pages; when the subdomain name is not the same and the hash value is not equal, it is determined that the first page and the second page are not equal for adjacent pages. Therefore, the URL tasks of the distributed web crawler are reasonably allocated to improve the crawling efficiency of the web crawler.

Referring to FIG. 4, another embodiment of the task distribution method provided by the present invention includes:

401. Parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into a task list;

402. Determine whether the URL of the first page has been crawled according to the URL list of the crawled page, and if so, delete the URL of the first page from the task list;

403. When the URL of the first page is not crawled, determine whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and determine whether the parent page of the URL of the first page is the same. Whether the hash value is the same as the hash value of the parent page of the URL of the second page;

404. When the subdomain name is the same or the hash value is the same, determine that the first page and the second page are adjacent pages;

405. When the subdomain names are not the same and the hash values are not the same, determine that the first page and the second page are not adjacent pages;

406. When it is determined that the first page and the second page are not adjacent pages, assign the first page to a new thread, and write the thread number corresponding to the new thread into the thread number of the URL of the first page item; delete the URL of the first page from the task list and write it into the URL list of crawled pages; the new thread saves the hash value of the parent page of the URL of the first page and the The name of the URL of the first page;

407. When it is determined that the first page and the second page are adjacent pages, assign the first page to the thread where the second page is located, and write the first page in the thread number item of the URL of the first page. The thread number corresponding to the thread where the second page is located; delete the URL of the first page from the task list and write it into the URL list of the crawled page; the thread where the second page is located saves the first page The hash value of the parent page of the URL and the name of the URL of the first page;

408. Execute the download task on the allocated thread according to the time sequence control.

It should be noted that the specific processes of steps 401 to 405 and 408 may correspond to steps 301 to 305 and 308 in the embodiment shown in FIG. 3 , respectively, which will not be repeated here.

In the embodiment of the present invention, when it is determined that the first page and the second page are not adjacent pages, the first page is allocated to a new thread, and the thread number corresponding to the new thread is written into the first page. In the thread number item of the URL; when it is determined that the first page and the second page are adjacent pages, the first page is allocated to the thread where the second page is located, and the thread number of the URL of the first page is Write the thread number corresponding to the thread where the second page is located in the item; at the same time, delete the URL of the first page from the task list, and write it into the URL list of the crawled page; Allocated to the same thread to ensure serial download of each adjacent page assigned to the same IP address, while non-adjacent pages are assigned to different threads for concurrent download; therefore, when the number of IPs is limited, not only does it improve It improves the crawling efficiency of web crawlers, and can effectively avoid web crawlers with a limited number of IPs from being identified as malicious web crawlers.

For ease of understanding, the task distribution method in the embodiment of the present invention is specifically described below with a specific application scenario:

Send a request to obtain the first page to the target web server, and parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list;

Perform deduplication processing on the URL of the first page in the task list, specifically: judging whether the URL of the first page in the task list has been crawled according to the URL list of crawled pages, if the first page has been crawled If the URL of the page has been crawled before, delete the URL of the first page from the task list;

When the URL of the first page is not crawled, the following judgment conditions are used to specifically allocate the first page: judging whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, And judge whether the hash value of the parent page of the URL of the first page is the same as the hash value of the parent page of the URL of the second page; can be understood as the subdomain name of the URL of the first page and the first page The hash value of the parent page of the URL is compared with other pages in all running threads, and judges whether they are the same;

After the above judgment, the obtained result is processed as follows: if it is judged that the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, or the hash value of the parent page of the URL of the first page is the same as that of the first page. If the hash values of the parent pages of the URLs of the two pages are equal, it is determined that the first page and the second page are adjacent pages; at this time, the first page is allocated to the thread where the second page is located, and the first page is assigned to the thread where the second page is located. Write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of a page; delete the URL of the first page from the task list, and write it into the URL list of the crawled page; Wherein, the thread where the second page is located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page;

If it is determined that the subdomain name of the URL of the first page is different from the subdomain name of the URL of the second page, and the hash value of the parent page of the URL of the first page is the same as the hash value of the parent page of the URL of the second page If the values are not equal, it is determined that the first page and the second page are not adjacent pages; at this time, the first page is allocated to a new thread, and the thread number corresponding to the new thread is written into the first page. In the thread number item of the URL; delete the URL of the first page from the task list, and write it into the URL list of the crawled page; wherein, the new thread saves the parent page of the URL of the first page. The hash value and the name of the URL of the first page;

After the above-mentioned adjacent pages and non-adjacent pages are reasonably allocated, the download task is executed according to the timing control through the web crawler download thread, and the content of the downloaded page is parsed. The hash value is sent to the URL task list, and a new URL task waiting queue is generated after deduplication.

The task distribution method has been described above, and will be described below from the perspective of a task distribution device. Specifically, the task distribution device may be integrated in a chip, and the chip may be loaded in a terminal. Please refer to FIG. 5 , and the device includes:

Parsing unit 501, for parsing the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into a task list;

Judging unit 502, for judging whether the URL of the first page has been crawled;

Determining unit 503, configured to determine the first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page when the URL of the first page is not crawled Whether it is adjacent to the second page;

a first allocation unit 504, configured to allocate the first page to a new thread after determining that the first page and the second page are not adjacent pages;

A second allocation unit 505, configured to allocate the first page to the thread where the second page is located after determining that the first page and the second page are adjacent pages;

The execution unit 506 is configured to execute the download task on the allocated thread according to the time sequence control.

In this embodiment of the present invention, the parsing unit 501 parses the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list; the judging unit 502 judges the URL of the first page Whether it has been crawled; when the URL of the first page has not been crawled, the determining unit 503 determines the hash value according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page. Whether the first page and the second page are adjacent pages; when determining that the first page and the second page are not adjacent pages, the first allocation unit 504 allocates the first page to a new thread; when determining that the first page and the second page are not adjacent pages When the first page and the second page are adjacent pages, the second allocation unit 505 allocates the first page to the thread where the second page is located, and the execution unit 506 executes the download task on the allocated thread according to timing control. Because the present invention reasonably allocates the URL tasks of the distributed web crawler, and allocates adjacent web pages to the same thread, it ensures that each adjacent page assigned to the same IP address is downloaded serially, and the non-adjacent pages are downloaded in series. It is allocated to different threads for concurrent download; therefore, in the case of a limited number of IPs, it not only improves the crawling efficiency of web crawlers, but also effectively prevents web crawlers with limited IP numbers from being identified as malicious web crawlers.

Based on the task distribution device in the above embodiment, optionally, the determining unit 502 is specifically configured to determine whether the URL of the first page has been crawled according to the URL list of the crawled pages, and if so, then determine whether the URL of the first page has been crawled. The URL is deleted from the task list to improve the crawling efficiency of the web crawler.

Based on the task distribution apparatus in the foregoing embodiment, optionally, as shown in FIG. 6 , the foregoing determining unit 503 specifically includes:

The judgment module 601 is used to judge whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page when the URL of the first page is not crawled, and to judge the URL of the first page Whether the hash value of the parent page is the same as the hash value of the parent page of the URL of the second page;

a first determining module 602, configured to determine that the first page and the second page are adjacent pages when the subdomain name is the same or the hash value is the same;

The second determining module 603 is configured to determine that the first page and the second page are not adjacent pages when the subdomain names are not identical and the hash values are not identical.

In this embodiment of the present invention, when the URL of the first page is not crawled, the judgment module 601 judges whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and judges whether the subdomain name of the URL of the first page is the same Whether the hash value of the parent page of the URL of the page and the hash value of the parent page of the URL of the second page are the same to determine whether the first page and the second page are adjacent pages; the first determination module 602 When the subdomain names are the same or the hash values are the same, determine that the first page and the second page are adjacent pages; when the subdomain names are not the same and the hash values are not equal, the second determining module 603, It is determined that the first page and the second page are not adjacent pages. Therefore, the URL tasks of the distributed web crawler are reasonably allocated to improve the crawling efficiency of the web crawler.

Based on the task distribution apparatus in the foregoing embodiment, optionally, as shown in FIG. 7 , the foregoing first distribution unit 504 specifically includes:

The first distribution module 701 is used to allocate the first page to a new thread, and write the thread number corresponding to the new thread into the thread number item of the URL of the first page; the new thread saves the first page The hash value of the parent page of the URL and the name of the URL of the first page;

The first deletion and writing module 702 is configured to delete the URL of the first page from the task list and write it into the URL list of the crawled pages.

In this embodiment of the present invention, when it is determined that the first page and the second page are not adjacent pages, the first distribution module 701 allocates the first page to a new thread, and writes the thread number corresponding to the new thread into the thread number item of the URL of the first page; the first deletion and writing module 702 deletes the URL of the first page from the task list, and writes it into the URL list of the crawled page; Adjacent pages are assigned to different threads for concurrent download; therefore, in the case of limited IP number, not only the crawling efficiency of web crawlers is improved, but also web crawlers with limited IP number can be effectively avoided from being identified as malicious web crawlers.

Based on the task distribution apparatus in the foregoing embodiment, optionally, as shown in FIG. 8 , the second distribution unit 505 specifically includes:

The second distribution module 801 is configured to allocate the first page to the thread where the second page is located, and write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of the first page ; The thread where the second page is located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page;

The second deletion and writing module 802 is configured to delete the URL of the first page from the task list and write it into the URL list of the crawled pages.

In this embodiment of the present invention, when it is determined that the first page and the second page are adjacent pages, the second distribution module 801 allocates the first page to the thread where the second page is located, and assigns the first page to the thread where the second page is located. Write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of the In the URL list; thus, the adjacent web pages are allocated to the same thread to ensure serial download of each adjacent page assigned to the same IP address; therefore, in the case of a limited number of IPs, it not only improves the crawling of web crawlers It is efficient, and can effectively avoid web crawlers with a limited number of IPs from being identified as malicious web crawlers.

The embodiments shown in FIG. 5 to FIG. 8 describe the specific structure of the task distribution device from the perspective of functional units. The specific structure of the task distribution device is described below with reference to the embodiment shown in FIG. 9 from the hardware perspective:

As shown in FIG. 9 , the task distribution apparatus includes: a receiver 901 , a transmitter 902 , a processor 903 and a memory 904 .

The task dispatching device involved in this embodiment of the present invention may have more or less components than those shown in FIG. 9 , may combine two or more components, or may have different component configurations or settings, and each component may be Hardware, software or a combination of hardware and software implementation including one or more signal processing and/or application specific integrated circuits.

The processor 903 is configured to read the instructions stored in the memory 904 to perform the following operations:

Parse the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list;

Determine whether the URL of the first page has been crawled;

When the URL of the first page is not crawled, determine whether the first page and the second page are not according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page for adjacent pages;

When it is determined that the first page and the second page are not adjacent pages, the first page is allocated to a new thread;

When it is determined that the first page and the second page are adjacent pages, the first page is allocated to the thread where the second page is located;

Execute the download task on the allocated thread according to the timing control.

The processor 903 is specifically configured to perform the following operations:

Whether the URL of the first page has been crawled is determined according to the URL list of the crawled pages, and if so, the URL of the first page is deleted from the task list.

The processor 903 is specifically configured to perform the following operations:

Determine whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and determine whether the hash value of the parent page of the URL of the first page is the same as that of the parent page of the URL of the second page. Whether the hash value is the same;

When the subdomain name is the same or the hash value is the same, determine that the first page and the second page are adjacent pages;

When the subdomain names are not the same and the hash values are not the same, it is determined that the first page and the second page are not adjacent pages.

The processor 903 is specifically configured to perform the following operations:

The first page is allocated to a new thread, and the thread number corresponding to the new thread is written into the thread number item of the URL of the first page; the new thread saves the hash of the parent page of the URL of the first page The hash value and the name of the URL of the first page;

Delete the URL of the first page from the task list and write it into the URL list of crawled pages.

The processor 903 is specifically configured to perform the following operations:

Allocate the first page to the thread where the second page is located, and write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of the first page; the thread where the second page is located Save the hash value of the parent page of the URL of the first page and the name of the URL of the first page;

Delete the URL of the first page from the task list and write it into the URL list of crawled pages.

In the embodiment of the present invention, the processor 903 parses the uniform resource locator URL of the first page and the hash value of the parent page of the URL of the first page into the task list; Deduplication processing; and determine whether the first page and the second page are adjacent pages according to the hash value of the subdomain name in the URL of the first page and the parent page of the URL of the first page; when determining whether the first page and the second page are adjacent pages; When a page and the second page are not adjacent pages, the first page is allocated to a new thread, and the download task is executed according to the timing control; when it is determined that the first page and the second page are adjacent pages, The first page is allocated to the thread where the second page is located, and the download task is executed according to time sequence control. Because the present invention reasonably allocates the URL tasks of the distributed web crawler, and allocates adjacent web pages to the same thread, it ensures that each adjacent page assigned to the same IP address is downloaded serially, and the non-adjacent pages are downloaded in series. It is allocated to different threads for concurrent download; therefore, in the case of a limited number of IPs, it not only improves the crawling efficiency of web crawlers, but also effectively prevents web crawlers with limited IP numbers from being identified as malicious web crawlers.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above may refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions described in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. a task distribution method, is characterized in that, comprises: Parse the hash value of the URL of the first page and the parent page of the URL of the first page into the task list; Determine whether the URL of the first page has been crawled; When the URL of the first page is not crawled, determine the relationship between the first page and the first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page. Whether the two pages are adjacent pages, the first page and the second page are pages allocated at the same IP address; when it is determined that the first page and the second page are not adjacent pages, assigning the first page to a new thread; When it is determined that the first page and the second page are adjacent pages, allocating the first page to the thread where the second page is located; Execute the download task on the allocated thread according to the timing control. 2. The task distribution method according to claim 1, wherein the judging whether the URL of the first page has been crawled specifically comprises: Determine whether the URL of the first page has been crawled according to the URL list of the crawled pages, and if so, delete the URL of the first page from the task list. 3. The task distribution method according to claim 1, characterized in that, determining said Whether the first page and the second page are adjacent pages specifically includes: Determine whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page, and determine whether the hash value of the parent page of the URL of the first page is the same as the URL of the second page. Whether the hash value of the parent page is the same; When the subdomain names are the same or the hash values are the same, determining that the first page and the second page are adjacent pages; When the subdomain names are not the same and the hash values are not the same, it is determined that the first page and the second page are not adjacent pages. 4. The task distribution method according to any one of claims 1 to 3, wherein the allocating the first page to a new thread specifically comprises: The first page is allocated to a new thread, and the thread number corresponding to the new thread is written into the thread number item of the URL of the first page; the new thread saves the URL of the first page. The hash value of the parent page and the name of the URL of the first page; Delete the URL of the first page from the task list and write it into the URL list of crawled pages. 5. The task distribution method according to any one of claims 1 to 3, wherein the allocating the first page to the thread where the second page is located specifically comprises: Allocate the first page to the thread where the second page is located, and write the thread number corresponding to the thread where the second page is located in the thread number item of the URL of the first page; The thread where the two pages are located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page; Delete the URL of the first page from the task list and write it into the URL list of crawled pages. 6. A task distribution device, comprising: a parsing unit for parsing the URL of the first page and the hash value of the parent page of the URL of the first page into the task list; a judging unit for judging whether the URL of the first page has been crawled; A determining unit, configured to determine the said first page according to the subdomain name in the URL of the first page and the hash value of the parent page of the URL of the first page when the URL of the first page is not crawled Whether the first page and the second page are adjacent pages, and the first page and the second page are pages allocated at the same IP address; a first allocation unit, configured to allocate the first page to a new thread after determining that the first page and the second page are not adjacent pages; a second allocation unit, configured to allocate the first page to the thread where the second page is located after determining that the first page and the second page are adjacent pages; The execution unit is used to execute the download task on the allocated thread according to the timing control. 7. The task distribution device according to claim 6, wherein the judging unit is specifically configured to judge whether the URL of the first page has been crawled according to the URL list of the crawled page, and if so, then The URL of the first page is removed from the task list. 8. The task distribution device according to claim 6, wherein the determining unit specifically comprises: a judging module for judging whether the subdomain name of the URL of the first page is the same as the subdomain name of the URL of the second page when the URL of the first page is not crawled, and judging the first page Whether the hash value of the parent page of the URL of the page is the same as the hash value of the parent page of the URL of the second page; a first determining module, configured to determine that the first page and the second page are adjacent pages when the subdomain names are the same or the hash values are equal; A second determining module, configured to determine that the first page and the second page are not adjacent pages when the subdomain names are not the same and the hash values are not the same. 9. The task distribution device according to any one of claims 6 to 8, wherein the first distribution unit specifically comprises: The first distribution module is used to assign the first page to a new thread, and write the thread number corresponding to the new thread into the thread number item of the URL of the first page; the new thread saves all The hash value of the parent page of the URL of the first page and the name of the URL of the first page; The first deletion and writing module is used to delete the URL of the first page from the task list and write it into the URL list of the crawled pages. 10. The task distribution device according to any one of claims 6 to 8, wherein the second distribution unit specifically comprises: A second distribution module, configured to allocate the first page to the thread where the second page is located, and write the thread corresponding to the thread where the second page is located in the thread number item of the URL of the first page Thread number; The thread where the second page is located saves the hash value of the parent page of the URL of the first page and the name of the URL of the first page; The second deletion and writing module is configured to delete the URL of the first page from the task list and write it into the URL list of the crawled pages.

Images