CN104516956B - A kind of site information increment crawling method - Google Patents

Abstract

The invention discloses a kind of site information increment crawling method, this method includes：The data of setting length are crawled according to website data presentation order, and data queue is put into according to the presentation order of website data, the data queue end is provided with comparison window, checks the data in comparison window and has crawled the multiplicity of data, when multiplicity reaches preset value, stop data and crawl；Otherwise, said process is repeated, until data in comparison window and the multiplicity for having crawled data reach preset value, stopping data crawling.When the present invention does not crawl for site information in strict accordance with time-sequencing progress increment, in the case of rate is climbed in admissible leakage, reduce and crawl consumption.In workflow, " the setting length that data crawl " and " length of data queue " size are dynamically adapted, algorithm operating efficiency is improved, meets that different leakages climbs rate and crawls loss demand.

Description

A Method of Incremental Crawling of Website Information

technical field

The invention relates to the technical field of network information crawling, in particular to a method for incremental crawling of website information, which is suitable for incremental crawling of website information that is not strictly sorted by time.

Background technique

The global Internet has expanded rapidly since it entered commercial use in the 1990s, and has become an important information infrastructure for promoting economic development and social progress in today's world. Although China's Internet development started later than that of the international Internet, it has also developed rapidly since the beginning of the new century. On the service side, the Internet has penetrated into various fields, especially in information search, communication, business transactions, and mobile wireless applications. On the user side, as of December 2013, my country's Internet users reached 618 million, making it the country with the largest number of users in the world.

With the development of Internet technology and services, the amount of Internet information is huge. In order to facilitate users to obtain interesting content from the huge network information more conveniently, network aggregation services appear. Network aggregation refers to providing users with useful, more targeted information.

Network aggregation first solves the problem of obtaining information from the target website. There is a type of website whose website information is not strictly chronologically sorted. If data is incrementally crawled from these websites, it is difficult to judge which information has been crawled and which is new. If The newly crawled data is verified one by one whether it has been crawled, which will bring a lot of crawling loss. After analyzing the information of this type of website, it is found that the earlier information on the website, that is, the information with high popularity or newer information, is not strictly sorted according to time, while the information at the back of the website, that is, the information with low popularity or older information. Information, relative comparisons are sorted by time series. For example, on a video content website, the information on the homepage under each column often includes newly launched metadata and metadata recommended by the operator or the system, so the information on the entire webpage is not sorted in chronological order. However, the next few pages of the column Metadata are relatively sorted in time series. When incrementally crawling such website information, there needs to be a way to minimize the crawling loss under the allowable missing crawl rate.

Contents of the invention

The purpose of the present invention is to overcome the technical problem of how to reduce the consumption of crawling under the allowable missing crawling rate in the prior art when website information is not strictly sorted according to time for incremental crawling, so as to provide a website information Incremental crawling method.

In order to achieve the above purpose, the present invention provides a method for incremental crawling of website information. The method includes: (a) crawling data of a set length starting from the first data presented on the target website according to the presentation order of the website data; (b) placing the crawled data of a set length in accordance with the presentation order of the website data Enter the data queue, and the end of the data queue is provided with a comparison window; (c) calculate the repetition degree between the data of the set length crawled in the comparison window and the crawled data; (d) stop data crawling according to the calculation result of the repetition degree Or carry out the next data crawl; that is, when the repeatability reaches the preset value, stop data crawling; when the repeatability does not reach the preset value, perform the next data crawl, and then perform step (b)- (d).

Preferably, when the repeatability reaches a preset value and the preset value is less than 1, save the data in the data queue that is not repeated with the crawled data in the database, and then stop the entire data crawling process.

Preferably, when the repeatability reaches a preset value and the preset value is 1, the entire data crawling process is stopped.

Preferably, the crawling of the next data specifically includes: when the repeatability does not reach a preset value, saving the data in the data queue that is not duplicated with the crawled data in the database, and clearing the data queue; According to the presentation order of the website data, continue to crawl the data of the set length from the end position of the last data crawl.

Preferably, the set length of the data crawling is less than or equal to the length of the data queue.

Preferably, when the repeatability does not reach the preset value, during the next data crawling process, the set length of data crawling and the length of the data queue are dynamically adjusted.

The present invention reduces the consumption of crawling under the condition of allowable missing crawling rate when website information is not strictly sorted according to time for incremental crawling. In the workflow, the size of the "set length of data crawling" and "data queue length" can be dynamically adjusted to improve the efficiency of the algorithm and meet different crawling leakage rates and crawling loss requirements.

Description of drawings

Fig. 1 is a flowchart of a website information incremental crawling method according to an embodiment of the present invention;

FIG. 1A is a schematic diagram of processing crawled data when the repeatability in FIG. 1 reaches a preset value;

FIG. 1B is a schematic diagram of the next data crawl when the repeatability in FIG. 1 does not reach the preset value;

Fig. 2 is a schematic diagram of adjusting the set length of data crawling and data queues during incremental crawling of website information according to an embodiment of the present invention.

detailed description

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, the following describes the technical solutions of the present invention through the accompanying drawings and embodiments The technical solution is further described in detail. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

Fig. 1 is a flow chart of the website information incremental crawling method of the present invention.

As shown in Figure 1, the website information incremental crawling method of the embodiment of the present invention includes the following steps:

In step 101, according to the presentation order of the website data, crawl the data of a set length starting from the first data presented on the target website. For example, if a video website has 100 pieces of information, the set length of crawling can be set to 10 pieces. .

In step 102, the crawled data with a set length, that is, the 1-10 website information in step 101 is put into the data queue according to the presentation order of the website data, and a comparison window is set at the end of the data queue. Preferably, the set length of data crawling is less than or equal to the length of the data queue.

In step 103, calculate the data of the set length crawled in the comparison window, that is, the repetition degree of the 1-10 website information in step 101 and the crawled data, or extract several pieces from the 1-10 website information Sampling calculations. Wherein, if the 1-10 pieces of information crawled are defined as the first crawled data today, the crawled data may refer to the video website information crawled yesterday.

In step 104, the data crawling is stopped or the next data crawling is performed according to the calculation result of the repeatability. Among them, the judgment principle is: when the repeatability reaches the preset value, stop data crawling (see step 104a); when the repeatability does not reach the preset value, perform the next data crawling (see step 104b), and then execute Steps 102-104.

FIG. 1A is a schematic diagram of processing crawled data when the repeatability in FIG. 1 reaches a preset value.

As shown in FIG. 1A , when the calculation repeatability in step 104 in FIG. 1 reaches a preset value, step 105 is executed (ie, it is determined whether the preset value is less than 1). If it is less than 1, then execute step 106 (save the data that is not duplicated with the crawled data in the data queue in the database, and then execute step 104a in Figure 1 (that is, stop the entire data crawling process). If the preset value is 1 , execute step 104a in FIG. 1 (that is, stop the entire data crawling process).

FIG. 1B is a schematic diagram of the next data crawl when the repeatability in FIG. 1 does not reach the preset value.

As shown in FIG. 1B , when the calculation repeatability in step 104 in FIG. 1 does not reach the preset value, step 104b is executed (that is, the next data crawling is performed). The crawling of the next data includes: when the repeatability does not reach the preset value, perform step 107 (that is, save the data in the data queue that is not repeated with the crawled data in the database, and clear the data queue); Then execute step 108 (that is, according to the presentation order of the website data, continue to crawl data of a set length from the last data crawl end position, which is consistent with the set length in step 101 of FIG. 1 ). Then execute step 102 in FIG. 1 , repeat the cycle until the repeatability between the data in the comparison window and the crawled data reaches a preset value, and then stop data crawling.

Fig. 2 is a schematic diagram of adjusting the set length of data crawling and data queues during incremental crawling of website information according to an embodiment of the present invention.

As shown in Figure 2, the data crawling length set in Figure 1 can be dynamically adjusted. When the data repetition of the set length of crawling next time does not reach the preset value, step 201 is further executed (that is, it is determined that the next crawling Get whether the data repetition degree of the set length is greater than the data repetition degree of the set length crawled last time), if so, then perform step 202 (that is, adjust the currently set data crawl length, if the data crawl length can be shortened dynamically, reduce the crawling loss), and then perform step 204 (that is, perform the next data crawling again according to the newly adjusted data crawling length). If not, execute step 203 (that is, perform next data crawling according to the originally set data crawling length).

To sum up, the beneficial effect of the present invention is that it can reduce the consumption of crawling under the allowable missing crawling rate when website information is not strictly sorted by time for incremental crawling. The size of "set length of data crawling" and "data queue length" can be dynamically adjusted to improve the efficiency of the algorithm and meet different crawling leakage rates and crawling loss requirements.

It should be understood that the present invention does not limit specific business categories, data categories, and target website categories, and changes made to the above contents also fall within the protection scope of the present invention.

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Protection scope, within the spirit and principles of the present invention, any modification, equivalent replacement, improvement, etc., shall be included in the protection scope of the present invention.

Claims (5)

1. A website information incremental crawling method, is characterized in that, comprises the following steps: (a) According to the presentation order of the website data, start from the first data presented on the target website, and crawl the data of a set length; (b) Putting the crawled data with a set length into the data queue according to the presentation order of the website data, and a comparison window is provided at the end of the data queue; (c) Calculate the repeatability between the data of the set length crawled in the comparison window and the crawled data; (d) Stop data crawling or perform next data crawling according to the repeatability calculation result; That is, when the repeatability reaches the preset value, data crawling is stopped; when the repeatability does not reach the preset value, the next data crawl is performed, and the set length of data crawling and the length of the data queue are dynamically adjusted, and then Perform steps (b)-(d). 2. The website information incremental crawling method according to claim 1, characterized in that, when the repeatability reaches a preset value and the preset value is less than 1, the crawled and uncrawled The data that fetches duplicate data is saved in the database, and then the entire data crawling process is stopped. 3. The website information incremental crawling method according to claim 1, characterized in that, when the repeatability reaches a preset value and the preset value is 1, the entire data crawling process is stopped. 4. The website information incremental crawling method according to claim 1, wherein said crawling of the next data includes: When the repeatability does not reach the preset value, save the data in the data queue that is not duplicated with the crawled data in the database, and clear the data queue; According to the presentation order of the website data, continue to crawl the data of the set length from the end position of the last data crawl. 5. The incremental crawling method for website information according to claim 1, characterized in that the set length of said data crawling is less than or equal to the length of the data queue.