CN103970749B - Block importance computational methods and system in a kind of webpage - Google Patents

Abstract

The invention discloses a block importance calculation method in a web page, comprising the following steps: providing a first web page, the first web page includes a plurality of area blocks; analyzing the first web page to obtain a plurality of specific blocks with different importance levels Regions; perform semantic analysis on multiple specific regions and multiple regional blocks respectively; classify multiple regional blocks according to the semantic similarity between each specific region and each specific region; according to the importance level of each specific region The importance levels of the multiple area blocks in the corresponding classification results are obtained. According to the embodiment of the present invention, it has the advantage of high and accurate calculation of the block importance level. The invention also provides a block importance calculation system in the web page.

Description

Method and system for calculating block importance in web pages

technical field

The invention relates to the technical field of the Internet, in particular to a method and system for calculating block importance in a web page.

Background technique

Internet webpages can be divided into several areas according to the reader's vision or the expression needs of the webpage designer. These areas have different functions in expressing the main content of the page, and the degree of visual attention of readers is also different. For example, the upper part of the webpage is usually used internally by the website, and it plays a role in guiding users to understand the overall structure of the website, and makes little contribution to expressing the main content of the webpage; The reader's main reading area. Therefore, for search engine retrieval, it is necessary to calculate the contribution of each area of the webpage to express the main content of the webpage, that is, the importance value of the block, which plays an important role in guiding the webpage into blocks and retrieval matching.

It is generally believed that: the area that expresses the main content of the page has the highest importance; the area that has nothing to do with the main content has the lowest importance; the area that expresses content related to the main content has intermediate importance.

Ruihua Song et al. invented a method to calculate block importance. This approach assumes that areas of the same theme within a page can be combined into independent blocks. The calculation process is: first divide the web page into blocks, then input the feature value of each block into the algorithm for calculating the importance, and then obtain the importance value of each block. The training goal is to minimize the square of the difference between the algorithm's result and the block importance value marked by the user. This method mainly uses the spatial location features and content features of blocks of a single page. Spatial position features refer to the absolute position of the block in the entire web page or its relative position relative to the entire page, and content features refer to page content such as pictures, links, texts, and user comment submission areas contained in a single page block.

Shian-Hua et al. proposed a method for identifying news article blocks based on table label classification. This method first regards the table tag as a block segmentation method to obtain blocks; then calculates the features of each block, confirms the situation of these features on all other pages in the site, and calculates the information entropy of each feature; The average value of feature information entropy is the information entropy of the block. When the information entropy of the block is less than the threshold, it is determined that the block is an article block, otherwise it is not an article block. This method calculates pages with multiple pages of information, which must come from the same site.

Lan Yi and others believe that the copyright, advertisement and other areas of the page have nothing to do with expressing the main content of the page, and are web page noise. Furthermore, based on the hypothesis that the page noise area of the same site contains similar content and display forms, a method to eliminate the noise of the web page is proposed. This method also assumes that the pages of the same site come from the same type of web page template. Based on this understanding, this method defines a data structure called Site Style Tree (SST), which calculates the similar presentation forms and contents of pages in the site. For each node on the SST, calculate the number of its child nodes and their distribution on all pages. If the number of sub-nodes of a node is more, and the distribution of categories is more, the score of the node is greater. When the score is less than the threshold, the node is judged as a noise node, otherwise it is a meaningful node.

In summary, the prior art has the following disadvantages:

1: Only the absolute position relationship of the block in the page is used, and the relationship between the block and the specific area of the page is not used.

2: Most of them only use a single page of information.

3: The multi-page information used assumes that the multi-page information must come from the same site and does not contain the relationship information with a specific area.

4: The scope of application is narrow and the problem solving is limited.

Contents of the invention

The purpose of the present invention is to solve at least one of the above-mentioned technical drawbacks.

Therefore, the purpose of the present invention is to propose a block importance calculation method in a web page, which has the advantages of high precision and accuracy in block importance level calculation.

Another object of the present invention is to propose a block importance calculation system in web pages.

In order to achieve the above object, the embodiment of the first aspect of the present invention discloses a block importance calculation method in a web page, comprising the following steps: providing a first web page, the first web page includes a plurality of area blocks; A webpage is parsed to obtain multiple specific areas with different importance levels; semantic analysis is performed on the multiple specific areas and the multiple area blocks respectively; according to the semantics between each area block and each specific area classify the plurality of area blocks by similarity; and obtain the importance level of the plurality of area blocks in the corresponding classification result according to the importance level of each specific area.

According to the block importance calculation method in the web page of the embodiment of the present invention, the web page is analyzed to obtain a plurality of specific areas with different importance levels, and the multiple area blocks in the web page are semantically related to the specific area. Blocks are classified, and the importance of each area block in the web page is obtained according to the importance of the specific area corresponding to the classification result. The embodiment of the present invention obtains the importance of the area block through the content relationship between the specific area and the area block, etc., The method can significantly improve the recall rate and accuracy rate of the region block classification, thus having the advantages of high precision and accuracy in block importance calculation.

In addition, the block importance calculation method in the webpage according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

In some examples, the method further includes: modifying the importance levels of the multiple specific areas and the multiple area blocks according to web pages of the same cluster related to the first web page.

In some examples, the step of modifying the importance levels of the plurality of specific regions and the plurality of region blocks according to the same cluster webpages related to the first webpage further includes: obtaining the same clusters related to the first webpage Cluster web pages; respectively obtain multiple specific areas and multiple classification results of each web page in the same cluster of web pages; calculate each specific area of each web page in the same cluster of web pages and each area block in each classification result The distribution information in the corresponding webpage and the relationship information between the area block and the corresponding specific area in each classification result; calculate all the statistical information of the distribution information; according to the statistical information of the distribution information and the The relationship information is used to modify the corresponding specific area in the first webpage and the area blocks in each classification result; and obtain the corresponding classification result according to the importance of each specific area of the first webpage after modification The importance of the multiple region blocks described in .

In some examples, the same-cluster webpage is a webpage having a similar DOM tree structure to the first webpage.

In some examples, the segment information includes: location coordinates, occupied area, and word distribution information.

In some examples, the multiple specific areas include: a web page path guide area, a title area of web page content, and a web page copyright statement area.

In some examples, the importance level of the webpage copyright statement area is lower than that of the webpage path guidance area, and the importance level of the webpage path guidance area is lower than that of the title area of the webpage content.

In some examples, after obtaining the importance levels of the multiple area blocks, the method further includes: evaluating the content in the multiple area blocks according to the importance levels of the multiple area blocks.

In some examples, it also includes: performing content monitoring on the area block with the highest importance level.

The embodiment of the second aspect of the present invention discloses a block importance calculation system in a webpage, including: an acquisition module, used to acquire a first webpage, and the first webpage includes a plurality of area blocks; a parsing module, used to analyze The first webpage is parsed to obtain multiple specific areas with different importance levels; an analysis module is used to perform semantic analysis on the multiple specific areas and the multiple area blocks; a classification module is used to analyze each The semantic similarity between each area block and each specific area is used to classify the plurality of area blocks; and a calculation module is used to obtain the corresponding classification results according to the importance level of each specific area. The importance level of each block.

According to the block importance calculation system in the web page of the embodiment of the present invention, the web page is analyzed to obtain a plurality of specific areas with different importance levels, and the multiple area blocks in the web page are semantically related to the specific area. Blocks are classified, and the importance of each area block in the web page is obtained according to the importance of the specific area corresponding to the classification result. The embodiment of the present invention obtains the importance of the area block through the content relationship between the specific area and the area block, etc., It can significantly improve the recall rate and accuracy rate of regional block classification, thus having the advantages of high precision and accuracy in block importance calculation

In addition, the block importance calculation system in the webpage according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

In some examples, the acquiring module is further configured to: acquire webpages in the same cluster as the first webpage.

In some examples, it further includes: an amending module, configured to amend the importance levels of the plurality of specific regions and the plurality of region blocks according to webpages of the same cluster related to the first webpage.

In some examples, the correction module is configured to calculate each specific area of each web page in the same cluster of web pages after obtaining multiple specific areas and multiple classification results of each web page in the same cluster and the distribution information of each area block in the corresponding web page in each classification result and the relationship information between the area block in each classification result and the corresponding specific area, and calculate the statistical information of all the distribution information, And modify the corresponding specific area in the first webpage and the area block in each classification result according to the statistical information of the distribution information and the relationship information.

In some examples, the same-cluster webpage is a webpage having a similar DOM tree structure to the first webpage.

In some examples, the segment information includes: location coordinates, occupied area, and word distribution information.

In some examples, the multiple specific areas include: a web page path guide area, a title area of web page content, and a web page copyright statement area.

In some examples, the importance level of the webpage copyright statement area is lower than that of the webpage path guidance area, and the importance level of the webpage path guidance area is lower than that of the title area of the webpage content.

In some examples, it further includes: a judging module, configured to judge content in the multiple area blocks according to the importance levels of the multiple area blocks.

In some examples, it also includes: a monitoring module, configured to monitor the content of the area block with the highest importance level.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is the flow chart of the block importance calculation method in the webpage according to one embodiment of the present invention;

2A-2C are schematic diagrams of multiple specific areas of a block importance calculation method in a web page according to an embodiment of the present invention;

3 is a flowchart of a method for calculating block importance in a web page according to another embodiment of the present invention;

Fig. 4A and Fig. 4B are schematic diagrams of specific area blocks before modification of the block importance calculation method in web pages according to an embodiment of the present invention;

5A and 5B are schematic diagrams of revised region blocks of a block importance calculation method in a web page according to an embodiment of the present invention; and

Fig. 6 is a structural diagram of a block importance calculation system in a webpage according to an embodiment of the present invention.

detailed description

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

In describing the present invention, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", The orientation or positional relationship indicated by "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than Nothing indicating or implying that a referenced device or element must have a particular orientation, be constructed, and operate in a particular orientation should therefore not be construed as limiting the invention.

In the description of the present invention, it should be noted that unless otherwise specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, or it can be two The internal communication of each element may be directly connected or indirectly connected through an intermediary. Those skilled in the art can understand the specific meanings of the above terms according to specific situations.

A method and system for calculating block importance in a web page according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

Fig. 1 is a flowchart of a method for calculating block importance in a web page according to an embodiment of the present invention. As shown in Figure 1, the block importance calculation method in the webpage according to one embodiment of the present invention comprises the following steps:

Step S101: Provide a first webpage, the first webpage includes a plurality of area blocks, wherein the first webpage refers to the webpage whose block importance is to be calculated at present, that is, most common webpages can be used at present.

In one embodiment of the present invention, the first webpage can be a webpage in any website, and the area block refers to the smallest granularity block in the webpage, and the smallest granularity block is an independent and indivisible block, such as a certain title or the A certain link, etc., and the importance of the content in each smallest granularity block is consistent. In the embodiment of the present invention, calculating the importance of a block in a webpage refers to predicting the importance of a smallest granularity block in a webpage.

Step S102: Parse the first webpage to obtain multiple specific regions with different importance levels. In an embodiment of the present invention, the multiple specific areas include but not limited to the web page path guide area, the title area of the web page content, and the web page copyright statement area. Further, the importance level of the webpage copyright statement area is lower than that of the webpage path guidance area, and the importance level of the webpage path guidance area is lower than that of the title area of the webpage content.

Wherein, a specific region refers to a region that expresses a specific and definite meaning. These specific areas exist in almost all pages, and their meanings and positions are relatively fixed. For example, the area representing the path relationship from the current page to the home page of the website (a specific area) can be called the mypos area; the area where the title of the webpage is located (a specific area) can be called the realtitle area; the area indicating the page copyright and privacy statement Area (a specific area), can be called copyright area. In an example of the present invention, a specific area of a web page can be identified through a web page parsing method. For example, taking the web page: http://finance.sina.com.cn/consume/puguangtai/20121220/031014059223.shtml as an example, specific area, wherein, Figure 2A shows the mypos area, Figure 2B shows the realtitle area, and Figure 2C shows the copyright area.

Step S103: Semantic analysis is performed on multiple specific areas and multiple area blocks respectively.

Specifically, performing semantic analysis on multiple specific regions and multiple regional blocks is to analyze the meaning of the content expressed by multiple specific regions and multiple regional blocks, for example, take the web page: http://finance.sina. com.cn/consume/puguangtai/20121220/031014059223.shtml as an example. After performing semantic recognition on multiple specific regions, as shown in Figures 2A, 2B and 2C, the specific regions of the page are described respectively: mypos, realtitle and copyright regions. Semantic analysis of the three shows that the main semantics of the mypos area is: "KFC McDonald's Raw Chicken Instant Door Sina Financial Life Text", the main semantics of the realtitle area is: "KFC antibiotic concealment", and the main semantics of the copyright area is: "Sina company copyright". In addition, performing semantic analysis on multiple regional blocks is similar to the above-mentioned method for performing semantic recognition on multiple specific regions, which will not be described here.

Step S104: Classify multiple region blocks according to the semantic similarity between each region block and each specific region.

Specifically, after obtaining the semantics of multiple area blocks and multiple specific areas in the first webpage, if the main semantics represented by the multiple area blocks have the highest similarity with the main semantics represented by the realtitle area, that is, the semantics are the closest, Then it can be considered that this area block expresses the main content of the page and is the most important area, that is, the block has the highest importance; if it has the highest semantic similarity with the mypos area, it can be considered that the content expressed by this area block is related to the main content of the page , which supplements information to the main content of the page, but is not the most specific content expressed by the page. It is a secondary area, and the importance of the block is lower than that of the area block with the highest similarity to the main semantics represented by the realtitle area; If the similarity with the semantics represented by the copyright area is the highest, it can be considered that the content expressed by the block in this area is not directly related to the main content of the page, and is an irrelevant area, and the importance of the block is lower than that with the highest similarity with the semantics represented by the mypos area. Importance of block. Based on the above analysis, it can be known that multiple area blocks can be classified, for example, for web pages:

According to http://finance.sina.com.cn/consume/puguangtai/20121220/031014059223.shtml, multiple area blocks can be divided into three categories. The second category consists of regions with high similarity to the semantics represented by the region blocks in the second and third categories, and the second category is composed of regions with a high similarity to the semantics represented by the mypos region and with the first and third categories The semantic similarity represented by the region blocks in the category is composed of regions with low similarity, and the third category is composed of regions with a high similarity to the semantics represented by the copyright region and represented by the region blocks in the first category and the second category Region blocks composed of region blocks with low semantic similarity.

Step S105: According to the importance level of each specific area, the importance levels of multiple area blocks in the corresponding classification result are obtained. Wherein, the importance levels of the area blocks in each classification result are the same, and the importance levels of the area blocks in different classification results are different. For example: for the above-mentioned area blocks of the first type (a classification result), its importance level is the highest, the importance level of the area blocks of the second type is lower than that of the area blocks of the first type, and the area blocks of the third type The importance level of the blocks is lower than that of the area blocks of the second type.

According to the block importance calculation method in the web page of the embodiment of the present invention, the web page is analyzed to obtain a plurality of specific areas with different importance levels, and the multiple area blocks in the web page are semantically related to the specific area. Blocks are classified, and the importance of each area block in the web page is obtained according to the importance of the specific area corresponding to the classification result. The embodiment of the present invention obtains the importance of the area block through the content relationship between the specific area and the area block, etc., The method can significantly improve the recall rate and accuracy rate of the region block classification, thus having the advantages of high precision and accuracy in block importance calculation.

In order to further improve the calculation accuracy of the importance of area blocks in a webpage, the method for calculating the importance of blocks in a webpage in a further embodiment of the present invention further includes: calculating the plurality of specific areas according to the same cluster of webpages related to the first webpage and the importance levels of the plurality of area blocks are corrected.

Specifically, as shown in FIG. 3 , the step of modifying the importance levels of the multiple specific areas and the multiple area blocks according to the web pages of the same cluster related to the first web page includes:

Step S301: Obtain the same cluster webpages related to the first webpage. In an embodiment of the present invention, the same-cluster webpage is a webpage having a similar DOM tree structure to the first webpage. In this example, the DOM tree is the abbreviation of HTML Document Object Model (Document Object Model). HTML DOM refers to the document object model applicable to HTML/XHTML. Specifically, the DOM tree refers to parsing the HTML source code of the web page into a "tree" structure , so that the program can traverse and access any content in the web page.

Step S302: Obtain multiple specific regions and multiple classification results of each webpage in the same cluster of webpages. In this example, the method for obtaining multiple specific areas and multiple classification results of each web page in the same cluster is the same as the method for obtaining multiple specific areas and multiple classification results in the first web page in the above embodiment.

Step S303: Calculate the distribution information of each specific area of each web page in the same cluster of web pages and each area block in each classification result in the corresponding web page, and the relationship between each area block in each classification result and the corresponding specific area relationship information. Among them, the segment information includes: the coordinates of the location (that is, the coordinates of each specific area and each area block in the corresponding webpage), the occupied area, and word distribution information, etc. Specifically, the segment information refers to the text in the area block Length, text term semantic distribution, number of links, number of pictures, picture area and other information.

Step S304: Calculate statistical information of all distribution information. For example, calculate statistical parameters such as the mean variance of all subsection information. The wider the distribution of subsection information, the richer the content expressed by the area block, and it is also the part that changes frequently on each page, which may be the highest level of importance. area block.

Step S305: Correct the corresponding specific area in the first webpage and the area block in each classification result according to the statistical information of the distribution information and the relationship information and the relationship information. Among them, the relationship information refers to: the distribution information of the relationship between the area block and the specific area, for example: in a web page, the mean value and variance of the positional relationship between the area block and the specific area, the mean value and variance of the page area, the mean value and variance of the text length, Text area, text signature, link repetition ratio, link number mean and variance, picture number mean and variance, picture area mean and variance, picture area accounted for block area mean and variance, user comment area mean and variance Variance, information such as the mean and variance of the area of the user's comment area to the area of the block. For example: if the space and content relationship between the area block and the realtitle area are relatively close, it means that the probability of the high importance of the area block will increase; if the content relationship between the area block and the copyright area or mypos area is relatively close, it means that the block is important The probability of high sex will decrease. Therefore, for the first webpage and the webpage of the same cluster, assuming that there are 5 webpages of the same cluster as the first webpage, for the area block 1 in the first webpage, in the first webpage, if according to step S101 to step The step of S105 obtains that the area block 1 is divided into the classification results corresponding to the realtitle area, and for the five web pages of the same cluster, the area blocks corresponding to the area block 1 are classified into the classification corresponding to the mypos area in the respective web pages In the result, according to the probability distribution of the importance level of the area block, it can be seen that the area block 1 should be classified into the classification results corresponding to the mypos area, therefore, it is necessary to modify the area block 1 in the first webpage, that is, the area block 1 is adjusted from the classification result corresponding to the realtitle area to the classification result corresponding to the mypos area.

As a specific example, count the multi-page information in the same xpath area of all web pages in the same cluster of web pages, where xpath is the path from the root node of the DOM tree to the current node. In this example, the multi-page information refers to the above-mentioned Refers to the segment information and relationship information. Still taking http://finance.sina.com.cn/consume/puguangtai/20121220/031014059223.shtml as an example, if only the specific area shown in Figure 2A and 2B is calculated on a single page (the first page), from the above Analysis shows that errors and even errors may occur. Since the DOM tree structure of pages in the same cluster, such as http://finance.sina.com.cn/consume/20121225/085714106842.shtml is very similar to it, by calculating the specific Area, and then to fix possible problems in a specific area of a single page (the first page). Taking the calculation of the realtitle area in the first web page as an example, the specific steps are as follows:

Step 1: Find the same cluster of webpages of the first webpage according to the url of the webpage, for example, the url of the first webpage: http://finance.sina.com.cn/consume/puguangtai/20121220/031014059223.shtml

The url of a web page is:

http://finance.sina.com.cn/consume/20121225/085714106842.shtml

Since the url of the first webpage is similar to the url of the above-mentioned certain webpage, it can be considered that its DOM tree structure is also similar.

Step 2: Calculate the realtitle areas corresponding to the above two urls respectively. The results obtained respectively may be shown in Figure 4A and Figure 4B, both of which have errors. Among them, the realtitle area shown in Figure 4A is incomplete, while the realtitle area shown in Figure 4B Unnecessary information such as text appearing on the lower side of FIG. 4B and the like are included.

Step 3: Calculate statistical information such as the coordinates of the above two realtitle areas, the mean area, word distribution, etc., and correct the realtitle area shown in Figure 4A according to the above information, and the corrected results are shown in Figure 2B.

Step 4: Repeat the above steps 2 and 3 to obtain the correction results of other specific areas of the first web page.

Step 5: Assuming that the correction result of a certain area block or a certain area of the first web page is shown in Figure 5A, calculate the number of words in the above-mentioned area block or area in the respective pages that are the same as those in the realtitle area and the proportion of words in the realtitle area The ratio of the number of words, etc. As shown in FIG. 5B , the corrected related content area, that is, the area of information such as topics related to the main content of the webpage is shown in FIG. 5B .

Step 6: Calculate the average value of the ratio described in step 5 to obtain the page information of the co-occurrence ratio of words in the realtitle area and the above-mentioned area block or area. Therefore, the area block can be corrected according to the page information of the co-occurrence ratio.

Step S306: According to the corrected importance of each specific area of the first web page, the importance of multiple area blocks in the corresponding classification result is obtained.

According to the block importance calculation method in the web page of the embodiment of the present invention, the area block and the specific area in the first web page are corrected by using the relationship between the area block and the specific area in the web page of the same cluster, and the error in the first web page can also be corrected. Correct the area blocks and specific areas, so as to more accurately obtain the classification results of specific areas, area blocks, and area blocks, and further improve the recall rate and accuracy of area block classification, thereby ensuring the accuracy of block importance calculations in web pages. Make the block importance calculation results more credible.

In an embodiment of the present invention, after obtaining the importance levels of the multiple area blocks, the method further includes: evaluating the contents of the multiple area blocks according to the importance levels of the multiple area blocks. For example: search engines need to score the relationship of web page links, such as pagerank. The present invention can identify the main content area of the page, the relevant content area and the irrelevant content area, that is, the area corresponding to the area blocks in the multiple classification results. Generally, links located in irrelevant content areas are less important; links located in main content areas and related content areas are more important. Therefore, in practical applications, for example, search engines score links in different areas of a page. As shown in Figure 2C, it is located in an irrelevant area, and Figure 5B is located in a related content area. The relationship score between the link shown in Figure 2A and this page is lower than the relationship score between the link shown in Figure 5B and this page.

In an embodiment of the present invention, it further includes: performing content monitoring on the area block with the highest importance level. In a specific application, there are cheating behaviors on many pages on the Internet, for example, the main content of the page is embedded with irrelevant advertisement content. Embodiments of the present invention can find important areas in the page by monitoring the content of the area block with the highest importance level, and thus can guide the search engine to analyze the internal content of the area, such as term semantic distribution, sentence topic correlation, and then Determine whether there is cheating.

The block importance calculation method in the webpage according to the embodiment of the present invention has the advantage of high and accurate block importance level calculation precision.

Fig. 6 is a structural diagram of a block importance calculation system in a webpage according to an embodiment of the present invention. As shown in FIG. 6 , the block importance calculation system 600 in a webpage according to an embodiment of the present invention includes: an acquisition module 610 , an analysis module 620 , an analysis module 630 , a classification module 640 and a calculation module 650 .

Wherein, the acquisition module 610 is used to acquire the first webpage, the first webpage includes a plurality of area blocks, and the plurality of specific areas include: the webpage path guidance area, the title area of the webpage content and the webpage copyright statement area, further, the webpage copyright statement area The importance level of the webpage path guidance area is lower than that of the webpage path guidance area, and the importance level of the webpage path guidance area is lower than that of the title area of the webpage content. The parsing module 620 is used for parsing the first webpage to obtain multiple specific regions with different importance levels. The analysis module 630 is used for performing semantic analysis on multiple specific areas and multiple area blocks. The classifying module 640 is used for classifying a plurality of area blocks according to the semantic similarity between each area block and each specific area. The calculation module 650 is used to obtain the importance levels of multiple area blocks in the corresponding classification result according to the importance level of each specific area.

According to the block importance calculation system in the web page of the embodiment of the present invention, the web page is analyzed to obtain a plurality of specific areas with different importance levels, and the multiple area blocks in the web page are semantically related to the specific area. Blocks are classified, and the importance of each area block in the web page is obtained according to the importance of the specific area corresponding to the classification result. The embodiment of the present invention obtains the importance of the area block through the content relationship between the specific area and the area block, etc., The method can significantly improve the recall rate and accuracy rate of the region block classification, thus having the advantages of high precision and accuracy in block importance calculation.

In an embodiment of the present invention, the acquiring module 610 is further configured to: acquire webpages in the same cluster related to the first webpage, where the webpages in the same cluster are webpages having a similar DOM tree structure to the first webpage. Further, the block importance calculation system 600 in a web page further includes: a correction module 660, which is used to assign importance levels to the multiple specific areas and the multiple area blocks according to the same cluster web pages related to the first web page Make corrections. Specifically, the correction module 660 is used to calculate each specific region of each webpage in the same cluster and each classification result of each webpage in the same cluster after obtaining multiple specific regions and multiple classification results of each webpage in the same cluster. The distribution information of each area block in the corresponding webpage and the relationship information between the area blocks in each classification result and the corresponding specific area, and calculate the statistical information of all the distribution information, and according to the statistical information and relationship of the distribution information The information modifies the corresponding specific area in the first web page and the area block in each classification result. In this example, the segment information includes: location coordinates (coordinates of a specific area or area block in the corresponding web page), occupied area (area occupied by a specific area or area block in the corresponding web page), and word distribution information.

According to the block importance calculation system in the web page of the embodiment of the present invention, the area block and the specific area in the first web page are corrected by using the relationship between the area block and the specific area in the web page of the same cluster, and the error in the first web page can also be corrected. Correct the area blocks and specific areas, so as to more accurately obtain the classification results of specific areas, area blocks, and area blocks, and further improve the recall rate and accuracy of area block classification, thereby ensuring the accuracy of block importance calculations in web pages. Make the block importance calculation results more credible.

In one embodiment of the present invention, the block importance calculation system 600 in the web page further includes: a judging module 670, which is configured to judge content in multiple regional blocks according to the importance levels of the multiple regional blocks. For example: search engines need to score the relationship of web page links, such as pagerank. The present invention can identify the main content area of the page, the relevant content area and the irrelevant content area, that is, the area corresponding to the area blocks in the multiple classification results. Generally, links located in irrelevant content areas are less important; links located in main content areas and related content areas are more important. Therefore, in practical applications, for example, search engines score links in different areas of a page. As shown in Figure 2C, it is located in an irrelevant area, while Figure 5B is located in a related content area. The relationship score between the link shown in Figure 2A and this page is lower than that shown in Figure 5B.

In an embodiment of the present invention, the block importance calculation system 600 in the webpage further includes: a monitoring module 680, which is used to monitor the content of the area block with the highest importance level. In a specific application, there are cheating behaviors on many pages on the Internet, for example, the main content of the page is embedded with irrelevant advertisement content. Embodiments of the present invention can find important areas in the page by monitoring the content of the area block with the highest importance level, and thus can guide the search engine to analyze the internal content of the area, such as term semantic distribution, sentence topic correlation, and then Determine whether there is cheating.

The block importance calculation system in the web page according to the embodiment of the present invention has the advantages of high precision and accuracy in block importance level calculation.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (17)

1. the block importance computational methods in a kind of webpage, it is characterised in that comprise the following steps：

The first webpage is provided, first webpage includes multiple region units；

First webpage is parsed to obtain multiple specific regions with different importance rates；

Semantic analysis is carried out to the multiple specific region and the multiple region unit respectively；

The multiple region unit is classified according to the semantic similarity between each region unit and each specific region；

The important of multiple region units described in corresponding classification results is obtained according to the importance rate of each specific region Property grade；And

According to the same cluster webpage related to the first webpage to importance of the multiple specific region and the multiple region unit etc. Level is modified.

2. according to the method described in claim 1, it is characterised in that the basis same cluster webpage related to the first webpage is to institute The step of importance rate for stating multiple specific regions and the multiple region unit is modified further comprises：

Obtain the same cluster webpage related to the first webpage；

Multiple specific regions of each webpage and multiple classification results in the same cluster webpage are obtained respectively；

Each region unit is calculated in the same cluster webpage in each specific region of each webpage and each classification results in correspondence Webpage in distributed intelligence and region unit in each classification results and corresponding specific region between relation information；

Calculate the statistical information of all distributed intelligences；

According to the statistical information of the distributed intelligence and the relation information to corresponding specific region in first webpage and Region unit in each classification results is modified；And

Institute in corresponding classification results is obtained according to the importance of each specific region of revised first webpage State the importance of multiple region units.

3. method according to claim 2, it is characterised in that wherein, the same cluster webpage is to have with first webpage There is the webpage of similar DOM tree structure.

4. according to the method in claim 2 or 3, it is characterised in that the distributed intelligence includes：Place coordinate, shared face Product and word distributed intelligence.

5. according to the method described in claim 1, it is characterised in that the multiple specific region includes：

Webpage Route guiding region, the Title area of web page contents and webpage copyright statement region.

6. method according to claim 5, it is characterised in that the importance rate in the webpage copyright statement region is less than The webpage Route guiding region, the importance rate in the webpage Route guiding region is less than the header area of the web page contents Domain.

7. according to the method described in claim 1, it is characterised in that obtain the multiple region unit importance rate it Afterwards, in addition to：

The content in the multiple region unit is judged according to the importance rate of the multiple region unit.

8. according to the method described in claim 1, it is characterised in that also include：Importance rate highest region unit is carried out Content is monitored.

9. the block importance computing system in a kind of webpage, it is characterised in that including：

Acquisition module, for obtaining the first webpage, first webpage includes multiple region units；

Parsing module, for being parsed to first webpage to obtain multiple given zones with different importance rates Domain；

Analysis module, for carrying out semantic analysis to the multiple specific region and the multiple region unit；

Sort module, for according to the semantic similarity between each region unit and each specific region by the multiple region unit Classified；

Computing module, for obtaining multiple described in corresponding classification results according to the importance rate of each specific region The importance rate of region unit；And

Correcting module, for the basis same cluster webpage related to the first webpage to the multiple specific region and the multiple region The importance rate of block is modified.

10. system according to claim 9, it is characterised in that the acquisition module is additionally operable to：

Obtain the same cluster webpage related to first webpage.

11. system according to claim 9, it is characterised in that the correcting module is used for：Obtain the same cluster webpage In each webpage multiple specific regions and multiple classification results after, calculate each spy of each webpage in the same cluster webpage Determine in region and each classification results in distributed intelligence and each classification results of each region unit in corresponding webpage Relation information between region unit and corresponding specific region, and calculate the statistical information of all distributed intelligences, and root According to the statistical information and the relation information of the distributed intelligence to corresponding specific region in first webpage and each point Region unit in class result is modified.

12. system according to claim 11, it is characterised in that wherein, the same cluster webpage is and first webpage Webpage with similar DOM tree structure.

13. the system according to claim 11 or 12, it is characterised in that the distributed intelligence includes：It is place coordinate, shared Area and word distributed intelligence.

14. system according to claim 9, it is characterised in that the multiple specific region includes：Webpage Route guiding area Domain, the Title area of web page contents and webpage copyright statement region.

15. system according to claim 14, it is characterised in that the importance rate in the webpage copyright statement region is low In the webpage Route guiding region, the importance rate in the webpage Route guiding region is less than the title of the web page contents Region.

16. system according to claim 9, it is characterised in that also include：

Module is judged, the content in the multiple region unit is commented for the importance rate according to the multiple region unit Sentence.

17. system according to claim 9, it is characterised in that also include：

Monitoring modular, for carrying out content monitoring to importance rate highest region unit.