JP7434867B2 - Methods, devices and storage media for extracting information from web pages - Google Patents

Description

TECHNICAL FIELD This disclosure relates to natural language processing, and specifically to information extraction based on multiple web pages.

Collecting and extracting information from the Internet is an important means of building a knowledge base. For example, product information can be extracted from an e-commerce company's web pages to build a product knowledge base. Conventional methods are mainly classified into two types.

The first method is to manually create a template or have a teacher create a template for pages with a similar structure (for example, the product list page of an e-commerce company's website has a similar structure). It is also possible to learn structural templates of product information included in web pages using a non-supervised method, and use the structural templates obtained by these learning methods to analyze other similar web pages. As shown in FIG. 1A, product information on books and shoes may be extracted by learning the structure information of the mobile phone page.

In the case of pages with a single structure (dissimilar), the second method dynamically analyzes the structure of the web page and searches the web page for related information using a list of keywords, as shown in Figure 1B. You may also specify the position in and extract the value.

The following presents a brief summary of the invention in order to provide a basic understanding of aspects of the invention. This brief summary is not an exhaustive summary of the present invention, does not intentionally specify the main points or important parts of the present invention, and is not intended to intentionally limit the scope of the present invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

The present invention provides a method, apparatus, and storage medium for extracting information from web pages.

In one aspect of the invention, a method for extracting information from a web page includes the steps of: generating a tree structure for each page containing the domain name of the web page in the web page and all extended web pages thereof; determining a navigation bar node in the tree structure; determining a leaf node covered by the navigation bar node that matches one or more keywords; and determining information on a page corresponding to the matching leaf node. A method is provided, comprising: extracting.

Another aspect of the present invention provides an apparatus for extracting information from a web page, the apparatus generating a tree structure for each page containing a domain name of the web page in the web page and all extended web pages thereof. a generation means; a navigation bar node determination means for determining a navigation bar node in the tree structure; a match node determination means for determining a leaf node covered by the navigation bar node that matches one or more keywords; and an information extraction means for extracting information in a page corresponding to a matching leaf node.

Other aspects of the invention further provide corresponding computer program code, computer readable storage media, and computer program products.

According to the method and apparatus for extracting information from web pages according to the present invention, necessary information is extracted from a plurality of web pages distributed under the same domain name based on the URL (uniform resource locator) of the home page. be able to.

The above and other advantages of the present invention will become clearer through the following detailed description of preferred embodiments of the present invention with reference to the drawings.

In order to provide an understanding of these and other advantages and features of the present disclosure, specific embodiments of the present disclosure will now be described in detail with reference to the drawings. The drawings and the following detailed description are included in and constitute a part of this specification. Elements having the same function and structure are designated by the same reference numerals. Note that these drawings are merely for illustrating typical examples of the present disclosure, and do not limit the scope of the present disclosure.
FIG. 2 is a diagram illustrating an example of a web page with a similar structure. FIG. 3 is a diagram illustrating an example of information extraction of a web page having a single structure. FIG. 3 is a diagram illustrating an example of information extraction of multiple pages. FIG. 2 is a diagram showing an example of the overall flow of the method of the present invention. 1 is a flowchart illustrating a method for extracting information from a web page according to an embodiment of the present invention. It is a diagram showing an example of an HTML structure and a Dom tree structure corresponding to a navigation bar node. FIG. 3 is a diagram showing an example of information extraction. 1 is a block diagram illustrating an example of an apparatus for extracting information from a web page according to an embodiment of the present invention. FIG. 1 is a block diagram illustrating an exemplary configuration of a general-purpose personal computer capable of implementing a method and/or apparatus according to an embodiment of the present invention. FIG.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. For convenience of explanation, not all features of an actual implementation are described in the specification. Note that during actual implementation, the specific embodiment may be changed in order to realize the specific goals of the developer, for example, the embodiment may be changed depending on the system and business constraints. good. In addition, the development work is very complex and time-consuming, but for those skilled in the art to whom this disclosure is concerned, this development work is just an example work.

It should be noted that, in order to make the present invention clear, only the structural features of the apparatus and/or process steps closely related to the present invention are shown in the drawings, and details unrelated to the present invention are omitted.

As mentioned above, collecting and extracting information from the Internet is an important means of building a knowledge base. Although the prior art shown in FIGS. 1A and 1B can meet some requirements, it still has limitations.

As shown in Figure 2A, http://owtware.com is the URL of the company's homepage, and the company's information such as products, partner companies, and contacts are distributed on different pages, and the main information location of the three pages is The parts that do not have a similar structure.

If only the URL of the home page is known, conventional methods cannot extract such information distributed across multiple pages. On the other hand, the URL of a home page is usually easily obtainable. Therefore, a method for extracting other information by extending the URL information of a home page remains a problem to be solved.

To solve the problems existing in the prior art, the present invention (1) automatically expands other pages containing related information when only the home page URL is known; and (2) extracts the main page from each related page. A multi-page based information extraction method is provided that can obtain locations containing information and (3) perform separate information extraction for pages with different attribute types.

FIG. 2B is a diagram showing an example of the overall flow of the method of the present invention. As shown in FIG. 2B, the method according to the present invention mainly includes the following three parts.

(1) Obtain a set of multiple pages by expanding the homepage.

(2) Perform statistical classification on a set of web pages using a statistical method to obtain navigation bar nodes, and then use a keyword dictionary to classify leaf nodes included in the navigation bar nodes. , and retrieve the page to be extracted based on the matching node information.

(3) Extraction is performed using different analyzers depending on the information type of the page to be extracted.

Hereinafter, a method for extracting information from a web page according to an embodiment of the present invention will be described in detail with reference to FIGS. 3, 4A, and 4B.

FIG. 3 is a flowchart illustrating a method for extracting information from a web page according to an embodiment of the present invention.

First, in step 301, a tree structure is generated for each page including the domain name of the web page in the web page and all its extended web pages. Specifically, in this embodiment, taking the URL shown in FIG. 2A as an example, the company homepage URL is u _root = http://www. outsideware. com/, and the information to be extracted is other attributes of the company, such as products, contacts, etc.

First, use a crawler to crawl the HTML page _root corresponding to u _root , analyze the page, and set all URLs contained therein u = [u ₀ , u ₁ , u ₂ ,..., u _n ]. Considering that the URLs contained in a page may or may not be related to the company, such as advertisements or external links, a set of some URLs u' = [u ' ₀ , u' ₁ , u' ₂ , ..., u' _n ], where u' _i contains domain(u _root ) and domain(URL) extracts the URL top-level domain name. For example, domain(u _root )=www. outsideware. com. In this way, all URLs with the same domain name, for example http://www.owtware.com/index.php/zh/products/, can be kept.

Preferably, p _i may be further expanded to take into account that the page p _i corresponding to u' _i may include other URL information. For each pi, expand URLs and corresponding pages using similar rules, and merge identical URLs and pages after each expansion. The expansion process may be repeated n times. Normally, n may be set to 2 so that a certain number of pages can be obtained and the number of pages is not too large. As a result, a set of pages with the same domain name p = [<p ₀ , u ₀ >, <p ₁ , u ₁ >, <p ₂ , u ₂ >, ..., <p _n , _un >] where p _i represents a web page and u _i represents a URL corresponding to the web page.

Next, in step 302, navigation bar nodes in the tree structure are determined. Specifically, in this embodiment, navigation bar nodes are obtained from the set p. As mentioned above, the purpose is to obtain pages containing the company information, such as products, contacts, etc., from the set p. Links in navigation bar nodes typically allow you to retrieve pages corresponding to this information. There are three main reasons to choose navigation bar nodes as information anchors.

(1) Information is accurate. The page to which the link contained in the navigation bar node points can be considered an introduction to the company. For example, a page corresponding to "Products and Services" introduces the company's products, and "Contact Information" links to a page with information such as the company's address and phone number. Links that appear in other parts of the web page do not necessarily explain information about the company, but may be information such as introductions or advertisements for other companies.

(2) Information is comprehensive. The navigation bar node basically contains all the information related to the company, and when you get the navigation bar node, you can get all the pages containing related information, which is very useful for subsequent information extraction.

(3) Relatively easy to obtain. Although different web pages may have different structures, the style of navigation bar nodes is largely the same. This commonality allows navigation bar nodes to be accurately located in the web structure.

Below, a method for determining a navigation bar node will be exemplified.

Due to the above three characteristics, frequently appearing nodes may be obtained by counting the nodes in each page p _i (p _i ∈p). Since these nodes include navigation bar nodes, the navigation bar nodes may be obtained by rearranging these frequently appearing nodes based on feature values. The specific method is as follows.

As shown in FIG. 4A, for each page p _i in the set p, p _i is first converted into a Dom tree structure.

For each leaf node node _i in the Dom tree, a path pattern path _i of node _i is obtained, and path _i is composed of the text corresponding to the leaf node and the path to the n-th ancestor node. Practical experience shows that for most pages, n may be an integer value greater than or equal to 5. For example, for the navigation bar node "Contact", if n=5, path _i = "ul_li_ul_li_a_contact" can be obtained.

Next, the document frequency df _i of each path _i , that is, the number of times path _i appears in different documents, is calculated. A path frequency dictionary node_pattern_dictionary {<path ₁ , df ₁ >, ..., <path _n , df _n >} may be obtained by statistics, where df _i >t, t is a threshold set as follows. It is.

In order to reduce the influence of the number of pages |p| on the final result, the threshold value t is set in stages.

After obtaining the path frequency dictionary, a second scan is performed on the Dom tree structure corresponding to each p _i in the set p, and this time, for each non-leaf node node _i , all non-NULLs covered by it are scanned. Assuming that the set of leaf nodes is c = [c ₀ , c ₁ , c ₂ , ..., c _n ], for each c _i , if path _i (c _i ) exists in the path frequency dictionary node_pattern_dictionary, then the node Record the information of _i . Finally, a candidate dictionary candidate_pattern_dictionary {<path ₁ ,[df ₁ ,cn ₁ ]>,...,<path _n ,[df _n ,cn _n ]>} may be obtained, where path _i is a non-leaf It represents the path information from node _i to the ancestor node, df _i represents the document frequency, and cn _i represents the number of all non-NULL leaf nodes covered by node _i . Unlike path _i of leaf nodes, path _i of non-leaf nodes does not contain textual information. As shown at 3 in FIG. 4A, the corresponding path from the "contact" to the ul node is ul_li_ul_div_div, and n=5.

Finally, sort the candidate dictionary candidate_pattern_dictionary according to the value of (cn*df/|p|), obtain the route corresponding to the maximum value as a navigation bar node route template, and navigate the ancestor node in the route corresponding to the maximum value. It may also be determined as a bar node. The template may be used to locate navigation bar nodes for a given HTML page under the company's home page.

Note that determining a navigation bar node using the above statistical method is merely an example of a method for determining a navigation bar node. The present invention is not limited thereto, and other suitable methods may be used to determine navigation bar nodes.

Next, in step 303, leaf nodes covered by the navigation bar node that match one or more keywords are determined. Specifically, in this embodiment, after a navigation bar node is obtained in step 302, for each non-NULL leaf node covered by the navigation bar node, the dictionary keyword_dict is used to match the text corresponding to the leaf node. I do. The dictionary keyword_dict includes predetermined keywords such as "product introduction" and "contact information", for example. If a leaf node matches a keyword, the "href" attribute may be retrieved from the corresponding HTML element, and the attribute value is the URL of the corresponding web page. For example, the HTML element corresponding to the "Contact" node in FIG. 4A includes the following link:

href=http://www.owtware.com/index.php/zh/about/contact-us/
Therefore, a set of web pages containing related information from the set p' = [<p' ₀ , u' ₀ , t' ₀ >, <p' ₁ , u' ₁ , t' ₁ >, <p' ₂ , u' ₂ , t' ₂ >, ..., <p' _n , u' _n , t' _n >], where p' _i and u' _i are the above defined p _i and u _i , and t' _i represents the type corresponding to the page, such as product, person, contact information, etc. This allows different analyzers to be selected for extraction depending on different types of pages.

For each p' _i , we first need to preprocess the HTML page. The purpose of preprocessing is to first extract the main information on the page. This process is common and independent of the web page type t'. The extracted results may be used as input when performing extraction later. As shown in (1) of Figure 4B, the original HTML page contains a lot of content, but only the parts shown in solid lines are necessary, including navigation bar nodes, side lists, and labels. It is necessary to remove all other parts including elements such as Footer, otherwise the extraction will be susceptible to noise data.

Considering the route frequency dictionary node_pattern_dictionary and the candidate dictionary candidate_pattern_dictionary generated in step 302, the following method may be used to determine leaf nodes that match one or more keywords covered by a navigation bar node.

For non-leaf node node _i in set p'i, assuming that the set of all non-NULL leaf nodes covered by it is c=[c ₀ , c ₁ , c ₂ ,..., c _n ], then If the following three conditions are met simultaneously, node _i may be determined to be a target content node containing leaf nodes that match one or more keywords.

Here, c _i is a non-NULL leaf node covered by node _i , c _j is a non-NULL leaf node covered by node _j , i≠j, and text_len(*) corresponds to a leaf node. Represents the length of the text. In other words, the total length of the text of all non-NULL leaf nodes covered by node _i is greater than the total length of the text of all non-NULL leaf nodes covered by another node node _j .

When a node _i that simultaneously satisfies the above three conditions is determined, this means that a leaf node that matches a predetermined keyword is determined.

Finally, in step 304, information on the page corresponding to the matching leaf node is extracted. Specifically, in this embodiment, after the node node _i that satisfies the above three conditions is determined, the information included in the leaf nodes covered by the node may be extracted.

Preferably, each leaf node may be made into an independent attribute extraction space, and each node <div class="panel-grid-cell"...> may be made into an independent attribute extraction space, as shown in (2) and (3) in FIG. 4B. Let it be an attribute space. This allows determining the boundaries of the attribute values, ie each value is only a value from section {{...}}. For example, when extracting person information, the information included in section {{…}} may be considered to represent the same person, and the information in different {{…}} may be considered to represent different people. , extraction errors can be avoided.

Preferably, after the extraction range is determined, a different analyzer, such as an entity recognizer (NER), a proper noun recognizer, or a numerical value recognizer, is selected depending on the type t' _i of p' _i to extract the specific information. may be extracted. (3) in FIG. 4B shows an example of the results of the proper noun recognizer.

Although the above describes extracting related information based on a company homepage, the present invention is not limited to this, and may be extended to extracting any information from any web page as necessary. good.

The above method may be fully implemented by a computer-executable program, or may be partially or completely implemented using hardware and/or firmware. When implemented in hardware and/or firmware, or when a computer executable program is loaded onto a hardware device capable of executing the program, an apparatus for extracting information from a web page as described below is implemented. In the following, the detailed contents mentioned above will be omitted and an overview of these devices will be explained. Note that these devices can execute the above-mentioned method, but the above-mentioned method is not limited to one that employs or is executed by a component of the device described later.

FIG. 5 is a block diagram illustrating an example of an apparatus 500 for extracting information from a web page according to an embodiment of the present invention. The device 500 includes a tree structure generation section 501, a navigation bar node determination section 502, a match node determination section 503, and an information extraction section 504. The tree structure generation unit 501 generates a tree structure for each page including the domain name of the web page in the web page and all its extended web pages. The navigation bar node determining unit 502 determines the navigation bar node in the tree structure. The match node determining unit 503 determines leaf nodes covered by the navigation bar node that match one or more keywords. The information extraction unit 504 extracts information on the page corresponding to the matching leaf node.

The apparatus 500 for extracting information from a web page shown in FIG. 5 corresponds to the method shown in FIG. Accordingly, the details of the apparatus 500 for extracting information from a web page have already been described in detail in the description of the method for extracting information from a web page in FIG. 3, and the description thereof will be omitted here.

The above processes and devices may be implemented by software and/or firmware. When implemented by software and/or firmware, the software for implementing the above method is configured from a storage medium or a network to a computer having a dedicated hardware configuration (e.g., general purpose personal computer 600 shown in FIG. 6). Programs may be installed, and the computer can perform various functions when various programs are installed.

FIG. 6 is a block diagram illustrating an exemplary configuration of a general-purpose personal computer that can implement the method and/or apparatus according to the embodiments of the present invention. In FIG. 6, a central processing unit (CPU) 601 executes various processes using programs stored in a read-only memory (ROM) 602 or programs loaded into a random access memory (RAM) 603 from a storage unit 608. do. The RAM 603 stores data necessary for the CPU 601 to execute various processes as necessary. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output interface 605 is also connected to bus 604.

Input unit 606 (including a keyboard, mouse, etc.), output unit 607 (including a display, such as a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage unit 608 (including, for example, a hard disk), communication A unit 609 (including, for example, a network interface card, such as a LAN card, modem, etc.) is connected to the input/output interface 605 . The communication unit 609 executes communication processing via a network, for example, the Internet. Optionally, driver 610 may be connected to input/output interface 605. The removable medium 611 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., and is set up in the driver 610 as necessary, and the computer program read from it is stored in the storage unit as necessary. 608 is installed.

When implementing the above processing using software, a program constituting the software is installed via a network, such as the Internet, or a storage medium, such as the removable medium 611.

Note that these storage media are not limited to the removable medium 611 shown in FIG. 6 that stores a program and provides the program to the user separately from the device. Removable media 611 may include, for example, magnetic disks (including floppy disks), optical disks (including optical disks - read only memory (CD-ROM), and digital versatile disks (DVD)), and magneto-optical disks (miniature disks). (MD) (registered trademark)) and semiconductor memory. Alternatively, the storage medium may be the ROM 602, a hard disk included in the storage unit 608, etc., which stores the programs and is provided to the user together with the device containing them.

The invention further provides a computer program product in which a corresponding computer program code, machine readable instruction code is stored. The instruction code, when read and executed by the device, can perform the method according to the embodiment of the invention described above.

Accordingly, the invention further includes a storage medium having recorded thereon a program product including machine readable instruction codes. The storage medium includes, but is not limited to, a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

Further, regarding the embodiments including the above-mentioned examples, the following additional notes are further disclosed.
(Additional note 1)
A method for extracting information from a web page, the method comprising:
generating a tree structure for each page containing the domain name of the web page in the web page and all its extended web pages;
determining a navigation bar node in the tree structure;
determining leaf nodes covered by the navigation bar node that match one or more keywords;
extracting information in a page corresponding to a matching leaf node.
(Additional note 2)
The method of clause 1, wherein the navigation bar node is determined using a statistical method.
(Additional note 3)
The step of determining a navigation bar node in the tree structure includes:
determining non-leaf nodes that include only leaf nodes whose number of appearances in the tree structure is greater than a predetermined threshold;
3. The method of claim 2, comprising reordering the non-leaf nodes to determine the navigation bar node.
(Additional note 4)
Determining whether the number of occurrences of a leaf node is greater than a predetermined threshold includes:
3. The method according to claim 3, comprising determining whether the number of occurrences of text and route information of the leaf node in the tree structure is greater than the predetermined threshold.
(Appendix 5)
The method according to appendix 4, wherein the route information is a route from the leaf node to its nth ancestor node, where n is a positive integer.
(Appendix 6)
The method according to appendix 5, wherein n is 5 or more.
(Appendix 7)
sorting the non-leaf nodes to determine the navigation bar node;
calculating a feature value of the non-leaf node, the feature value being determined by the number of leaf nodes covered by the non-leaf node and the number of times;
3. The method according to claim 3, comprising determining a non-leaf node having a maximum feature value among the non-leaf nodes as the navigation bar node.
(Appendix 8)
8. The method of claim 7, wherein the feature value is a ratio of the product of the number of leaf nodes covered by the non-leaf nodes and the number of times to the total number of pages containing the domain name of the web page.
(Appendix 9)
The step of extracting information on the page corresponding to the matching leaf node is as follows:
determining a target node included in a page corresponding to the matching leaf node;
9. A method according to any one of appendices 1 to 8, comprising the step of respectively extracting the text of each leaf node covered by the target node.
(Appendix 10)
The target node is
The number of occurrences of text and route information of each leaf node included in the target node in the tree structure is equal to or less than the predetermined threshold;
The target node is not a non-leaf node among non-leaf nodes that include only leaf nodes whose number of occurrences in the tree structure is greater than a predetermined threshold, and the text of all leaf nodes included in the target node is 9. The method of clause 9, wherein the total length is determined by being greater than the total length of text of other non-leaf nodes in the tree structure.
(Appendix 11)
Respectively extracting the text of each leaf node covered by the target node comprises:
The method according to appendix 9, comprising selecting different analyzers to perform the extraction depending on the type of page corresponding to the target node.
(Appendix 12)
The method according to appendix 11, wherein each leaf node of the target node is an independent attribute extraction space.
(Appendix 13)
12. The method according to appendix 11, wherein the analyzer is an entity recognizer, a proper noun recognizer, or a numerical value recognizer.
(Appendix 14)
9. The method according to any one of appendices 1 to 8, wherein navigation bar nodes in the web page and all extended web pages thereof are determined using the route information of the determined navigation bar nodes.
(Additional note 15)
9. The method according to any one of appendices 1 to 8, wherein a page containing the domain name of the web page in the web page and all its extension web pages is determined by extracting a URL top-level domain name.
(Appendix 16)
9. The method according to any one of appendices 1 to 8, wherein the tree structure is an HTML document object model (DOM).
(Appendix 17)
9. The method according to any one of appendices 1 to 8, wherein the keyword is a predetermined keyword.
(Appendix 18)
9. The method according to any one of appendices 1 to 8, wherein the expanded web page is expanded n times to obtain a page that includes the domain name of the web page, where n is an integer of 2 or more.
(Appendix 19)
A device for extracting information from a web page, the device comprising:
Tree structure generating means for generating a tree structure for each page including the domain name of the web page in the web page and all extended web pages thereof;
navigation bar node determining means for determining a navigation bar node in the tree structure;
match node determining means for determining a leaf node covered by the navigation bar node that matches one or more keywords;
an information extraction means for extracting information in a page corresponding to a matching leaf node.
(Additional note 20)
A computer-readable storage medium storing a program, the program being executed by a processor;
generating a tree structure for each page containing the domain name of the web page in the web page and all its extended web pages;
determining a navigation bar node in the tree structure;
determining leaf nodes covered by the navigation bar node that match one or more keywords;
and extracting information in a page corresponding to a matching leaf node.

Note that the terms "comprising," "having," or any other variations are not limited to exclusive inclusion, and a process, method, object, or device that includes a series of elements is not limited to just including those elements. and includes other elements not explicitly listed or elements inherent in the process, method, object, or apparatus. Also, unless there are further limitations, the term "comprising an" more restricted element does not exclude the presence of other identical elements in a process, method, article, or apparatus that includes the element.

Although preferred embodiments of the present invention have been described above with reference to the drawings, the embodiments and examples described above are illustrative and not restrictive. Those skilled in the art may make various modifications, improvements, and equivalent changes to the present invention within the spirit and scope of the claims. These modifications, improvements, or equivalent changes are included within the protection scope of the present invention.

Claims (10)

A method for extracting information from a web page, the method comprising:
generating a tree structure for each page containing the domain name of the web page in the web page and all its extended web pages;
determining a navigation bar node in the tree structure;
determining leaf nodes covered by the navigation bar node that match one or more keywords;
A method for performing the steps of: extracting information in a page corresponding to a matching leaf node. The step of determining a navigation bar node in the tree structure includes:
determining non-leaf nodes that include only leaf nodes whose number of appearances in the tree structure is greater than a predetermined threshold;
2. The method of claim 1, comprising reordering the non-leaf nodes to determine the navigation bar node. Determining whether the number of occurrences of a leaf node is greater than a predetermined threshold includes:
3. The method of claim 2, comprising determining whether the number of occurrences of text and route information of the leaf node in the tree structure is greater than the predetermined threshold. 4. The method of claim 3, wherein the path information is a path from the leaf node to its nth ancestor node, where n is a positive integer. 5. The method according to claim 4, wherein n is 5 or more. sorting the non-leaf nodes to determine the navigation bar node;
calculating a feature value of the non-leaf node, the feature value being determined by the number of leaf nodes covered by the non-leaf node and the number of times;
3. The method of claim 2, comprising: determining the non-leaf node having the largest feature value among the non-leaf nodes as the navigation bar node. The step of extracting information on the page corresponding to the matching leaf node is as follows:
determining a target node included in a page corresponding to the matching leaf node;
7. A method according to any preceding claim, comprising the step of respectively extracting the text of each leaf node covered by the target node. The target node is
The number of occurrences of text and route information of each leaf node included in the target node in the tree structure is equal to or less than a predetermined threshold;
The target node is not a non-leaf node among non-leaf nodes that include only leaf nodes whose number of occurrences in the tree structure is greater than a predetermined threshold, and the text of all leaf nodes included in the target node is 8. The method of claim 7, wherein the total length is determined by being greater than the total length of text of other non-leaf nodes in the tree structure. A device for extracting information from a web page, the device comprising:
Tree structure generating means for generating a tree structure for each page including the domain name of the web page in the web page and all extended web pages thereof;
navigation bar node determining means for determining a navigation bar node in the tree structure;
match node determining means for determining a leaf node covered by the navigation bar node that matches one or more keywords;
an information extraction means for extracting information in a page corresponding to a matching leaf node. A computer-readable storage medium storing a program, the program being executed by a processor;
generating a tree structure for each page containing the domain name of the web page in the web page and all its extended web pages;
determining a navigation bar node in the tree structure;
determining leaf nodes covered by the navigation bar node that match one or more keywords;
and extracting information in a page corresponding to a matching leaf node.

Images