CN111444407A - Automatic extraction method and system for page list information of web crawler - Google Patents

Abstract

The invention relates to a method and system for automatically extracting page list information of a network crawler. The method includes: acquiring a hypertext markup language document of a page to be extracted; determining a set of hypertext markup language objects according to elements in the hypertext markup language document; traversing the set of hypertext markup language objects to determine a document object model; determine the web page structure of the page to be extracted according to the document object model; the web page structure includes list items and list item attributes; determine the extraction mode according to the web page structure of the page to be extracted; use the extraction mode Extract the page list information to be extracted. The invention provides a method and system for automatically extracting page list information of a web crawler, which realizes automatic crawling of web crawler page list information.

Description

A method and system for automatically extracting page list information from a web crawler

technical field

The invention relates to the field of web crawler, in particular to a method and system for automatically extracting page list information of a web crawler.

Background technique

With the rapid development of information technology, it has brought massive and majestic data information. How to quickly, efficiently and accurately extract useful information from massive information puts forward higher requirements for information acquisition technology in the new era. In this context, web crawler technology has been rapidly developed and widely used due to its characteristics of convenient information acquisition, various acquisition methods and semi-automatic information acquisition. However, traditional web crawling technology needs to write crawler scripts according to the characteristics of web pages, and the Internet is full of massive information publishing platforms, systems and websites, and the ways and formats of information displayed on each page are ever-changing. The development cost is high, and when the web page is revised, the crawler program needs to be updated in a targeted manner, and the stability of the crawling is also affected. Moreover, manual intervention is required to achieve automatic page crawling. It can be seen that the prior art cannot realize automatic crawling of web crawler page list information.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method and system for automatically extracting page list information of a web crawler, so as to realize automatic crawling of web crawler page list information.

For achieving the above object, the present invention provides the following scheme:

A method for automatically extracting page list information for a web crawler, comprising:

Obtain the hypertext markup language document of the page to be extracted;

Determine a set of hypertext markup language objects according to elements in the hypertext markup language document; the elements include tags, attributes and text of the hypertext markup language document;

traversing the set of hypertext markup language objects to determine a document object model;

Determine the web page structure of the page to be extracted according to the document object model; the web page structure includes a list item and a list item attribute;

Determine the extraction mode according to the webpage structure of the page to be extracted;

The page list information to be extracted is extracted by using the extraction mode.

Optionally, the obtaining of the hypertext markup language document of the page to be extracted further includes:

Judging whether the page to be extracted is revised, and obtaining a first judgment result;

If the first judgment result indicates that the page to be extracted is revised, the step of obtaining the hypertext markup language document of the page to be extracted is performed;

If the first judgment result indicates that the page to be extracted has not been revised, the page list information to be extracted is directly extracted according to the corresponding extraction mode when the page is not revised.

Optionally, traversing the set of hypertext markup language objects to determine the document object model specifically includes:

The hypertext markup language object set is traversed to determine the relationship between all objects; the relationship between all the objects includes the sibling relationship, the containment relationship, the parent relationship, the child relationship, the height and the level of the hierarchical relationship between the objects. depth;

The document object model is determined according to the relationships among all the objects.

Optionally, the determining the web page structure of the to-be-extracted page according to the document object model specifically includes:

Determine the number of tags of each attribute and the number of tags of all attributes in the to-be-extracted page according to the document object model;

Determine the proportion of the number of times of occurrence of each label of each attribute in the to-be-extracted page to obtain a single proportion set;

Determine the proportion of the combination of tag attributes in the page to be extracted, and obtain a set of combination proportions;

Arrange the single proportion set and the combined proportion set in descending order according to the frequency of occurrence of the tag attribute to obtain a list item; the attribute of the list item is that the proportions of multiple single tag attributes appearing are the same as the combined tag attributes in the combined proportion set. The tag attributes that appear with equal weight;

Determine all document object chains in the to-be-extracted page according to the document object model;

Intercepting each of the document object chains starting with the tag attribute combination corresponding to the list item;

Calculate the frequency of occurrence of the document object chain after each interception;

The frequency of occurrence of each intercepted document object chain is sorted in descending order, and the list item attribute is determined.

Optionally, the interception of each of the document object chains starting from the combination of tag attributes corresponding to the list items further includes:

Judging whether the document object chain contains the tag attribute combination corresponding to the list item, and obtaining a second judgment result;

If the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, then the document object chain is retained, and each tag attribute combination corresponding to the list item is used as a starting point to intercept each Describe the document object chain;

If the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item, the document object chain is eliminated.

A system for automatically extracting page list information for a web crawler, comprising:

A hypertext markup language document acquisition module, used for acquiring the hypertext markup language document of the page to be extracted;

a hypertext markup language object set determining module, configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes and texts of the hypertext markup language document;

a document object model determination module, used for traversing the set of hypertext markup language objects to determine a document object model;

a webpage structure determination module, configured to determine the webpage structure of the to-be-extracted page according to the document object model; the webpage structure includes a list item and a list item attribute;

an extraction mode determination module, configured to determine an extraction mode according to the web page structure of the page to be extracted;

A page list information extraction module, configured to extract the page list information to be extracted by using the extraction mode.

Optionally, also include:

a first judgment module, configured to judge whether the page to be extracted is revised, and obtain a first judgment result;

an execution module, configured to execute the step of obtaining the hypertext markup language document of the page to be extracted if the first judgment result indicates that the page to be extracted is revised;

The extracting module according to the corresponding extraction mode when the version is not revised is configured to directly perform the processing on the page list information to be extracted according to the corresponding extraction mode when the version is not revised if the first judgment result indicates that the page to be extracted has not been revised. extract.

Optionally, the document object model determination module specifically includes:

The unit for determining the relationship between all objects is used to traverse the set of hypertext markup language objects to determine the relationship between all objects; all the relationships between the objects include sibling relationship, containment relationship, and parent relationship between objects , height and depth of child relationship, hierarchical relationship;

A document object model determining unit, configured to determine the document object model according to the relationship among all the objects.

Optionally, the webpage structure determination module specifically includes:

A tag attribute determining unit, configured to determine, according to the document object model, the number of tags for each attribute and the number of tags for all attributes in the to-be-extracted page;

A single proportion set determination unit, used to determine the proportion of the times of occurrence of each tag of each attribute in the page to be extracted, to obtain a single proportion set;

a combination proportion set determination unit, used to determine the proportion of the combination of tag attributes in the page to be extracted, and obtain a combination proportion set;

The list item determination unit is used to sort the single proportion set and the combined proportion set in descending order according to the frequency of occurrence of tag attributes to obtain list items; The tag attributes with equal proportions appearing in the combined tag attributes in the combined proportion set;

a document object chain determining unit, configured to determine all document object chains in the to-be-extracted page according to the document object model;

a document object chain interception unit, configured to intercept each of the document object chains starting with the tag attribute combination corresponding to the list item;

a frequency calculation unit, configured to calculate the frequency of occurrence of the document object chain after each interception;

The list item attribute determining unit is configured to sort the occurrence frequency of each intercepted document object chain in descending order to determine the list item attribute.

Optionally, the webpage structure determination module further includes:

a first judgment unit, configured to judge whether the document object chain contains the tag attribute combination corresponding to the list item, and obtain a second judgment result;

A document object chain retention unit, configured to retain the document object chain if the second judgment result indicates that the document object chain contains a combination of tag attributes corresponding to the list item, and use the tag attribute corresponding to the list item The combination starts to intercept each of the document object chains;

A document object chain elimination unit, configured to eliminate the document object chain if the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

The invention provides a method and system for automatically extracting page list information of a web crawler. By acquiring the hypertext markup language document of the page to be extracted, and determining the set of hypertext markup language objects, the document is determined according to the set of hypertext markup language objects. Object collection, and then determine the web page structure of the page to be extracted through the document object collection, that is, automatically identify the page structure to be extracted, which can solve the problem of the ever-changing ways and formats of information displayed on each page, avoid manual intervention, and realize The web crawler page list information is automatically captured to ensure the stability of the web crawler.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some of the present invention. In the embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative labor.

1 is a schematic flowchart of a method for automatically extracting page list information of a web crawler provided by the present invention;

FIG. 2 is a schematic structural diagram of a system for automatically extracting page list information of a web crawler provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The purpose of the present invention is to provide a method and system for automatically extracting page list information of a web crawler, so as to realize automatic crawling of web crawler page list information.

In order to make the above objects, features and advantages of the present invention more clearly understood, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

1 is a schematic flowchart of a method for automatically extracting page list information of a web crawler provided by the present invention. As shown in FIG. 1 , the method for automatically extracting page list information of a web crawler provided by the present invention includes:

S101: Obtain a hypertext markup language document of a page to be extracted.

Also included before S101:

It is judged whether the page to be extracted is revised, and a first judgment result is obtained.

If the first judgment result indicates that the page to be extracted is revised, the step of acquiring the hypertext markup language document of the page to be extracted is performed.

If the first judgment result indicates that the page to be extracted has not been revised, the page list information to be extracted is directly extracted according to the corresponding extraction mode when the page is not revised.

S102: Determine a set of hypertext markup language objects according to elements in the hypertext markup language document; the elements include tags, attributes and texts of the hypertext markup language document. The elements in the hypertext markup language document are regarded as one object, and the determined set of hypertext markup language objects is HTML, HTML={Obj ₁ , Obj ₂ , Obj ₃ , . . . , Obj _n }.

S103, traverse the set of hypertext markup language objects to determine a document object model.

The hypertext markup language object set is traversed to determine the relationship between all objects; the relationship between all the objects includes the sibling relationship, the containment relationship, the parent relationship, the child relationship, the height and the level of the hierarchical relationship between the objects. depth. The height is the length of the longest path of the leaf, and the depth is the length of the path to its root.

The document object model Dom is determined according to the relationship among all the objects.

S104: Determine a web page structure of the page to be extracted according to the document object model; the web page structure includes a list item and a list item attribute. According to the characteristics of the document object model, the number of all elements and attributes, and the relationship between them can be quickly calculated. After obtaining statistical data and association relationships, analyze them, extract list items, list item attributes, irrelevant information and other contents from the document structure, and eliminate irrelevant information. The web page structure of all pages follows the following rules: ① The content list part of the list page has the same Class or ID attribute, or has the same label element; ② The title of each item of the list item is placed separately and does not repeat, Often it is a tag, or has event attributes; ③ the auxiliary information of the list item, such as time, author, introduction, etc., is in a specific tag; ④ the subject of the list item and the auxiliary information of the list item are always aggregated, included or arranged in parallel .

The number of tags for each attribute and the number of tags for all attributes in the page to be extracted are determined according to the document object model. Calculate the number of all tags and attributes x _i in the page as mi; calculate the total number of all tags and attributes in the page as w,

Determine the proportion of the times of occurrence of each tag of each attribute in the page to be extracted to obtain a single proportion set. That is, the proportion p _i of the times of occurrence of a single tag and attribute in the whole page is calculated to form a single proportion set p, p={p ₁ , p ₂ , p ₃ , p ₄ ,...p _w },

n(_i,j,...)是指多个标签属性组合在页面中出现的数量。Determine the proportion of the tag attribute combination in the page to be extracted, and obtain a combination proportion set k. Sometimes attributes in the page are more than one combination, through the combination of different elements, different structural content is expressed, k={p( _1,2,... ),p( _1,3,... ),p ( _2,3,... ),...,p( _i,j,... )}, where i, j represent different tag attribute combinations, and i, j are all combinations that exist in the page.

n( _i,j,... ) refers to the number of tag attribute combinations that appear in the page.

Arrange the single proportion set and the combined proportion set in descending order according to the occurrence frequency of the tag attribute to obtain a list item; the attribute of the list item is that the proportion of the occurrence of multiple single tag attributes is the same as that of the combined tag attribute in the combined proportion set. Appears as a weighted equal tag attribute. That is, k( _{i ,j,...} )=pi= _pj =....

All document object chains in the to-be-extracted page, that is, DOM chains, are determined according to the document object model. The hierarchical DOM chain is represented by M, and the parallel DOM chain is represented by N.

Each of the document object chains is intercepted starting with the tag attribute combination corresponding to the list item.

The frequency of occurrence of the document object chain after each interception is calculated.

The frequency of occurrence of each intercepted document object chain is sorted in descending order, and the list item attribute is determined.

S105, determining an extraction mode according to the webpage structure of the page to be extracted;

S106: Extract the page list information to be extracted by using the extraction mode.

The step of intercepting each of the document object chains starting from the tag attribute combination corresponding to the list item further includes:

It is judged whether the document object chain contains the tag attribute combination corresponding to the list item, and a second judgment result is obtained.

If the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, the document object chain is retained, and each of the document object chain is intercepted starting from the tag attribute combination corresponding to the list item. Describes the document object chain.

If the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item, the document object chain is eliminated.

In a specific embodiment, HML={html,head,body,h3,div,p,a,span}, traverse the HTML, and record the relationship between all objects (same level, contains, parent level, child level) etc.) and the height and depth of the hierarchical relationship. The height is the length of the longest path of the leaf, and the depth is the length of the path to its root. The result of the traversal determines the document object model, that is, the Dom model:

1) Calculate the number of all tags in the page to be 25 (because the internal tags in the head do not participate in data extraction, so the useless tags in the head are removed and do not participate in the calculation), and the number of attributes is 11;

2) Calculate the total number of all tags and attributes in the page as 25+11=36;

3) Calculate the proportion p of the number of times a single tag and attribute appear on the entire page; as shown in Table 1:

Table 1

tags/attributes p tags/attributes p html 1/36 a 5/36 head 1/36 span 10/36 body 1/36 main 5/36 div 1/36 list 5/36 p 5/36 content 1/36

In the above table, html, head, body, div, p, a, span are tags, and main, list, and content are attributes.

4) Calculate the proportion of the combination of tag attributes in the page. Sometimes attributes in the page are more than one combination, and different structural content is expressed through the combination of different elements. The set of combined tag attributes is denoted as k. In this example, the p tag has two class attributes, namely main and list;

k={p _{main, list} }

5) Arrange the sets p and k in descending order according to the frequency of occurrence of tag attributes, and obtain the same ratio of occurrence of single-element set and combined element set, and the ratio of multiple single-element occurrences is equal to the ratio of combined tag attributes in the combined element set The label properties of , found by comparison:

p _(mian,list) =p _main =p _list

Then the tag P whose attributes are main and list are list items, and the tags html, head, body, div and attribute content are irrelevant tags and irrelevant attributes.

②Determining algorithm of list item attributes

Through analysis, it is found that in this example, the list item and the list item attributes are in an inclusive relationship, which is a hierarchical M, and the list item attributes are a parallel type N. The process of determining the list item attributes is as follows:

1) List all DOM chains in the page, wherein the hierarchical DOM chain is represented by M, and the parallel DOM chain is represented by N. In this example, there is only one level of inclusion, that is, p includes a and span.

M={P→a|span|span}

2) Intercept the above DOM chain starting with the list item label attribute (combination), directly discard the DOM chain without the list item label attribute, and calculate the frequency of occurrence of each DOM chain, and construct a set R as a result. In this example, there is only one chain {P→a|span|span}, and the chain appears 5 times, then:

3) Arrange the DOM chains in descending order according to the frequency, denoted as R', and the attribute with a higher frequency is the attribute of the list item to be selected. Then according to the web page rules obtained above, it can be determined that span is a list item attribute.

Result: The list item is P, and its attributes are main and list; the content of the list item is a tag, and the list item attribute is the span tag.

FIG. 2 is a schematic structural diagram of a system for automatically extracting page list information of a web crawler provided by the present invention. As shown in FIG. 2 , a system for automatically extracting page list information of a web crawler provided by the present invention includes: hypertext markup Language document acquisition module 201 , hypertext markup language object set determination module 202 , document object model determination module 203 , web page structure determination module 204 , extraction mode determination module 205 and page list information extraction module 206 .

The hypertext markup language document obtaining module 201 is configured to obtain the hypertext markup language document of the page to be extracted.

The hypertext markup language object set determining module 202 is configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes and texts of the hypertext markup language document.

The document object model determining module 203 is configured to traverse the set of hypertext markup language objects to determine the document object model.

The webpage structure determining module 204 is configured to determine the webpage structure of the to-be-extracted page according to the document object model; the webpage structure includes a list item and a list item attribute.

The extraction mode determination module 205 is configured to determine the extraction mode according to the webpage structure of the page to be extracted.

The page list information extraction module 206 is configured to extract the page list information to be extracted by using the extraction mode.

The system for automatically extracting page list information of a web crawler provided by the present invention further comprises: a first judgment module, an execution module, and an extraction module according to the corresponding extraction mode when the version is not revised.

The first judgment module is used for judging whether the page to be extracted is revised to obtain a first judgment result.

The execution module is configured to execute the step of acquiring the hypertext markup language document of the page to be extracted if the first judgment result indicates that the page to be extracted is revised.

The extracting module according to the extraction mode corresponding to the unmodified version is configured to directly extract the list information of the to-be-extracted page according to the corresponding extraction mode when the unmodified version is not modified if the first judgment result indicates that the page to be extracted has not been modified. .

The document object model determination module 203 specifically includes: a relationship determination unit among all objects and a document object model determination unit.

The relationship determination unit between all objects is used to traverse the set of hypertext markup language objects to determine the relationship between all objects; all the relationships between objects include sibling relationship, containment relationship, parent relationship, The height and depth of child relationships, hierarchical relationships.

The document object model determining unit is configured to determine the document object model according to the relationship among all the objects.

The webpage structure determination module 204 specifically includes: a tag attribute determination unit, a single weight set determination unit, a combined weight set determination unit, a list item determination unit, a document object chain determination unit, a document object chain interception unit, a frequency calculation unit, and a list item. Property determines the unit.

The tag attribute determining unit is configured to determine, according to the document object model, the number of tags for each attribute and the number of tags for all attributes in the page to be extracted.

The unit for determining a single proportion set is configured to determine the proportion of the occurrence times of each tag of each attribute in the page to be extracted to obtain a single proportion set.

The combination proportion set determination unit is configured to determine the proportion of the combination of tag attributes in the to-be-extracted page to obtain a combination proportion set.

The list item determination unit is used to sort the single proportion set and the combined proportion set in descending order according to the frequency of occurrence of the tag attribute to obtain a list item; Tag attributes with equal weights appearing in the combined tag attributes in the weight set.

The document object chain determining unit is configured to determine all document object chains in the to-be-extracted page according to the document object model.

The document object chain interception unit is configured to intercept each of the document object chains starting with the tag attribute combination corresponding to the list item.

The frequency calculation unit is used for calculating the frequency of occurrence of the document object chain after each interception.

The list item attribute determining unit is configured to sort the occurrence frequency of each intercepted document object chain in descending order to determine the list item attribute.

The webpage structure determination module 204 further includes: a first judgment unit, a document object chain retention unit and a document object chain elimination unit.

The first judging unit is configured to judge whether the document object chain contains the tag attribute combination corresponding to the list item, and obtain a second judgment result.

The document object chain retention unit is configured to retain the document object chain if the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, and use the tag attribute combination corresponding to the list item Each of the document object chains is intercepted for the start.

The document object chain elimination unit is configured to eliminate the document object chain if the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

In this paper, specific examples are used to illustrate the principles and implementations of the present invention. The descriptions of the above embodiments are only used to help understand the methods and core ideas of the present invention; meanwhile, for those skilled in the art, according to the present invention There will be changes in the specific implementation and application scope. In conclusion, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. a kind of page list information automatic extraction method of web crawler, is characterized in that, comprises: Obtain the hypertext markup language document of the page to be extracted; Determine a set of hypertext markup language objects according to elements in the hypertext markup language document; the elements include tags, attributes and text of the hypertext markup language document; traversing the set of hypertext markup language objects to determine a document object model; Determine the web page structure of the page to be extracted according to the document object model; the web page structure includes a list item and a list item attribute; Determine the extraction mode according to the webpage structure of the page to be extracted; The page list information to be extracted is extracted by using the extraction mode. 2. The method for automatically extracting page list information of a web crawler according to claim 1, wherein the acquisition of the hypertext markup language document of the page to be extracted further comprises before: Judging whether the page to be extracted is revised, and obtaining a first judgment result; If the first judgment result indicates that the page to be extracted is revised, the step of obtaining the hypertext markup language document of the page to be extracted is performed; If the first judgment result indicates that the page to be extracted has not been revised, the page list information to be extracted is directly extracted according to the corresponding extraction mode when the page is not revised. 3. The method for automatically extracting page list information of a web crawler according to claim 1, wherein the traversing of the hypertext markup language object collection to determine a document object model specifically comprises: The hypertext markup language object set is traversed to determine the relationship between all objects; the relationship between all the objects includes the sibling relationship, the containment relationship, the parent relationship, the child relationship, the height and the level of the hierarchical relationship between the objects. depth; The document object model is determined according to the relationships among all the objects. 4. The method for automatically extracting page list information of a web crawler according to claim 1, wherein the determining the web page structure of the page to be extracted according to the document object model specifically comprises: Determine the number of tags of each attribute and the number of tags of all attributes in the to-be-extracted page according to the document object model; Determine the proportion of the number of times of occurrence of each label of each attribute in the to-be-extracted page to obtain a single proportion set; Determine the proportion of the combination of tag attributes in the page to be extracted, and obtain a set of combination proportions; Arrange the single proportion set and the combined proportion set in descending order according to the frequency of occurrence of the tag attribute to obtain a list item; the attribute of the list item is that the proportions of multiple single tag attributes appearing are the same as the combined tag attributes in the combined proportion set. The tag attributes that appear with equal weight; Determine all document object chains in the to-be-extracted page according to the document object model; Intercepting each of the document object chains starting with the tag attribute combination corresponding to the list item; Calculate the frequency of occurrence of the document object chain after each interception; The frequency of occurrence of each intercepted document object chain is sorted in descending order, and the list item attribute is determined. 5. The method for automatically extracting page list information of a web crawler according to claim 4, wherein the described document object chain is intercepted starting from the label attribute combination corresponding to the list item, before the Also includes: Judging whether the document object chain contains the tag attribute combination corresponding to the list item, and obtaining a second judgment result; If the second judgment result indicates that the document object chain contains the tag attribute combination corresponding to the list item, then the document object chain is retained, and each tag attribute combination corresponding to the list item is used as a starting point to intercept each Describe the document object chain; If the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item, the document object chain is eliminated. 6. A system for automatically extracting page list information of a web crawler, characterized in that, comprising: A hypertext markup language document acquisition module, used for acquiring the hypertext markup language document of the page to be extracted; a hypertext markup language object set determining module, configured to determine a hypertext markup language object set according to elements in the hypertext markup language document; the elements include tags, attributes and texts of the hypertext markup language document; a document object model determination module, used for traversing the set of hypertext markup language objects to determine a document object model; a webpage structure determination module, configured to determine the webpage structure of the to-be-extracted page according to the document object model; the webpage structure includes a list item and a list item attribute; an extraction mode determination module, configured to determine an extraction mode according to the web page structure of the page to be extracted; A page list information extraction module, configured to extract the page list information to be extracted by using the extraction mode. 7. The page list information automatic extraction system of a kind of web crawler according to claim 6, is characterized in that, also comprises: a first judgment module, configured to judge whether the page to be extracted is revised, and obtain a first judgment result; an execution module, configured to execute the step of obtaining the hypertext markup language document of the page to be extracted if the first judgment result indicates that the page to be extracted is revised; The extracting module according to the corresponding extraction mode when the version is not revised is configured to directly perform the processing on the page list information to be extracted according to the corresponding extraction mode when the version is not revised if the first judgment result indicates that the page to be extracted has not been revised. extract. 8. The system for automatically extracting page list information of a web crawler according to claim 6, wherein the document object model determination module specifically comprises: The unit for determining the relationship between all objects is used to traverse the set of hypertext markup language objects to determine the relationship between all objects; all the relationships between the objects include sibling relationship, containment relationship, and parent relationship between objects , height and depth of child relationship, hierarchical relationship; A document object model determining unit, configured to determine the document object model according to the relationship among all the objects. 9. The system for automatically extracting page list information of a web crawler according to claim 6, wherein the web page structure determination module specifically comprises: A tag attribute determining unit, configured to determine, according to the document object model, the number of tags for each attribute and the number of tags for all attributes in the to-be-extracted page; A single proportion set determination unit, used to determine the proportion of the times of occurrence of each tag of each attribute in the page to be extracted, to obtain a single proportion set; a combination proportion set determination unit, used to determine the proportion of the combination of tag attributes in the page to be extracted, and obtain a combination proportion set; The list item determination unit is used to sort the single proportion set and the combined proportion set in descending order according to the frequency of occurrence of tag attributes to obtain list items; The tag attributes with equal proportions appearing in the combined tag attributes in the combined proportion set; a document object chain determining unit, configured to determine all document object chains in the to-be-extracted page according to the document object model; a document object chain interception unit, configured to intercept each of the document object chains starting with the tag attribute combination corresponding to the list item; a frequency calculation unit, configured to calculate the frequency of occurrence of the document object chain after each interception; The list item attribute determination unit is configured to sort the occurrence frequency of each intercepted document object chain in descending order to determine the list item attribute. 10. The system for automatically extracting page list information of a web crawler according to claim 9, wherein the web page structure determination module further comprises: a first judgment unit, configured to judge whether the document object chain contains the tag attribute combination corresponding to the list item, and obtain a second judgment result; A document object chain retention unit, configured to retain the document object chain if the second judgment result indicates that the document object chain contains a combination of tag attributes corresponding to the list item, and use the tag attribute corresponding to the list item The combination starts to intercept each of the document object chains; A document object chain elimination unit, configured to eliminate the document object chain if the second judgment result indicates that the document object chain does not contain the tag attribute combination corresponding to the list item.

Images