JP2009199302A - Program, device, and method for analyzing document - Google Patents

Abstract

An object of the present invention is to improve the accuracy of document category classification by determining main parts of a document in a layout structure.
A category feature learning unit 2 extracts a word from a learning target document and calculates a feature amount for each category of the word. The layout feature learning unit 3 calculates the importance for each partial structure of the layout based on the feature amount for each category of words. By looking at the magnitude of importance, it is possible to determine the importance of each partial structure in the document. The category probability determining unit 4 uses the importance for each partial structure of the layout to identify a location important for the category classification from the document, and then determines the category using the feature amount and the importance.
[Selection] Figure 3

Description

  The present invention relates to a program, an apparatus, and a method for analyzing document contents and layout structure of a structured document.

  There is known a technique for automatically classifying a document category by counting the appearance frequency of words when analyzing the contents of the document.

  As one usage form of such a document category classification method, for example, it may be used to determine whether the content of a web page is classified into a category inappropriate for minors. This makes it possible to perform a filtering process that restricts browsing of web pages. The web page here refers to a document published on the Internet. This document includes various information embedded in the document in addition to the layout information and text information by HTML.

  When automatically classifying web page categories based on predetermined criteria, the layout structure is important in addition to the contents of the pages. Therefore, Patent Document 1 discloses a method of performing classification in consideration of appearance characteristics in addition to document content characteristics.

JP 2000-268040 A

  In the conventional method, for each category, a characteristic layout structure of the category is determined. As a result, even if the correspondence between a specific layout structure and a category can be grasped, it has not been possible to determine where in the document the content that is the subject of the document exists.

  For example, even if a certain web page has the usual layout structure that a web page of an electronic bulletin board has, whether it is a web page dealing with sports topics or a web page dealing with game topics It is difficult to judge.

The invention according to claim 1 is an analysis program for a document having a layout structure, in which an element extraction process for extracting a plurality of constituent elements from a learning target document having a layout structure and the configuration extracted from the learning target document For each element, based on a category feature learning process for calculating a feature amount in the category of the learning target document, a plurality of elements included in the layout structure of the learning target document based on the learning target document and the feature amount A document analysis program characterized by causing a computer to execute layout feature learning processing for calculating the importance of each partial structure.
The invention according to claim 2 is the document analysis program according to claim 1, in addition to the feature amount calculated by the category feature learning process, and the importance degree calculated by the layout feature learning process. A category determination process for determining a category of a determination target document having a layout structure based on the layout structure of the learning target document is executed by a computer.
The invention according to claim 3 is the document analysis program according to claim 2, wherein the category determination processing outputs a probability that the determination target document belongs to each of a plurality of categories.
According to a fourth aspect of the present invention, in the document analysis program according to the first aspect, in addition, a predetermined keyword is searched from a plurality of search target documents having a layout structure, the layout structure of the learning target document, A document search process for calculating a degree of association between each of the search target documents and the keyword based on the importance calculated by the layout feature learning process is executed by a computer.
The invention according to claim 5 is the document analysis program according to any one of claims 1 to 3, wherein the document is image data.
The invention according to claim 6 is the document analysis program according to claim 5, wherein the component extracted from the image data by the element extraction means is a pixel.
The invention according to claim 7 is the document analysis program according to any one of claims 1 to 4, wherein the document is a document described in HTML.
The invention according to claim 8 is characterized in that, in the document analysis program according to any one of claims 1 to 4 or 7, the component is a word.
The invention according to claim 9 is the document analysis program according to any one of claims 1 to 4 or 7, wherein the component is a word and a co-occurrence relationship of the word.
The invention according to claim 10 is the document analysis program according to claim 8 or 9, wherein the word is extracted by morphological analysis.
According to an eleventh aspect of the present invention, in the document analysis program according to any one of the first to tenth aspects, the feature amount is calculated by a TF-IDF method.
The invention according to a twelfth aspect is the document analysis program according to any one of the first to eleventh aspects, wherein the layout feature learning processing is performed by using the layout structure from a plurality of the learning target documents having similar layout structures. A new document in which the common parts are extracted is created, and the importance of each partial structure in the layout structure of the new document is calculated.
The invention according to claim 13 is the document analysis program according to any one of claims 1 to 12, wherein the layout structure has a tree structure.
According to a fourteenth aspect of the present invention, in the document analysis program according to the thirteenth aspect, when the layout feature learning process calculates the importance of the partial structure positioned higher in the tree structure, the portion positioned higher A partial structure located at a lower level is excluded from the structure.
According to a fifteenth aspect of the present invention, in the document analysis program according to the thirteenth aspect, when the layout feature learning process calculates the importance of the partial structure positioned higher in the tree structure, the portion positioned lower The structure is included in a partial structure located at a higher level.
The invention according to claim 16 is an apparatus for analyzing a document having a layout structure, wherein an element extraction unit extracts a plurality of components from a learning target document having a layout structure, and the configuration extracted from the learning target document For each of the elements, a category feature learning unit that calculates a feature amount in the category of the learning target document, and a plurality of elements included in the layout structure of the learning target document based on the learning target document and the feature amount. A document analysis apparatus comprising: a layout feature learning unit that calculates importance of each partial structure with respect to the partial structure.
The invention according to claim 17 is the document analysis apparatus according to claim 16, in addition to the feature amount calculated by the category feature learning unit, and the importance level calculated by the layout feature learning unit. A category determination unit that determines a category of a determination target document having a layout structure based on the layout structure of the learning target document.
The invention according to claim 18 is the document analysis apparatus according to claim 16, further comprising: searching for a predetermined keyword from a plurality of search target documents having a layout structure, the layout structure of the learning target document, A document search unit that calculates the degree of association between each of the search target documents and the keyword based on the importance calculated by the layout feature learning unit is provided.
The invention according to claim 19 is a method for analyzing a document having a layout structure, wherein an element extracting step of extracting a plurality of components from a learning target document having a layout structure, and the configuration extracted from the learning target document For each element, a category feature learning step for calculating a feature amount in the category of the learning target document, and a plurality of elements included in a layout structure of the learning target document based on the learning target document and the feature amount. A layout analysis learning step for calculating the importance of each partial structure for the partial structure.
The invention according to claim 20 is the document analysis method according to claim 19, in addition to the feature amount calculated by the category feature learning step, and the importance degree calculated by the layout feature learning step. And a category determination step of determining a category of a determination target document having a layout structure based on the layout structure of the learning target document.
The invention according to claim 21 is the document analysis method according to claim 19, further comprising: searching for a predetermined keyword from a plurality of search target documents having a layout structure, and the layout structure of the learning target document; A document search step of calculating a degree of association between each of the search target documents and the keyword based on the importance calculated by the layout feature learning step is provided.

  According to the present invention, the layout structure and the category are not immediately associated, but information on whether or not the layout structure is important for category determination can be associated with the layout structure. That is, according to the present invention, the location corresponding to the subject of the document can be specified for the document having the layout structure, so that the category classification accuracy is improved.

-First embodiment-
A document classification apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of the document classification device according to the present embodiment. The document classification apparatus 1 shown in the figure inputs a document described in HTML (hereinafter referred to as an HTML document). For the category to which the input HTML document belongs, category determination processing is executed according to the procedure described later. The result of the category determination process is output as the probability that the input HTML document belongs for each of a plurality of categories.

  When performing document category classification using the document classification device 1, it is necessary to perform learning in advance. Learning in the document classification device 1 is divided into two steps, one of which is category feature learning and the other is layout feature learning. When learning is performed, a plurality of documents whose categories are known in advance are prepared, and after first performing category feature learning, layout feature learning is performed.

  The document classification device 1 includes a category feature learning unit 2, a layout feature learning unit 3, a category probability determination unit 4, and a data storage unit 5. The category feature learning unit 2 includes a learning target document input unit 21, an element extraction unit 22, and a category feature calculation unit 23, and performs category feature learning.

  The layout feature learning unit 3 includes a learning target document input unit 31, a layout analysis unit 32, an element extraction unit 33, and a layout feature analysis unit 34, and performs layout feature learning. The category probability determination unit 4 includes a determination target document input unit 41, a category determination unit 42, an element extraction unit 43, and a category score calculation unit 44, and performs an HTML document category determination process.

  The data storage unit 5 includes a category feature table 51, a layout feature table 52, a removal tag table 53, a removal attribute table 54, a similar tag table 55, and a category table 56. The category feature table 51 records the result of category feature learning. The layout feature table 52 records the result of layout feature learning. The three tables, the removal tag table 53, the removal attribute table 54, and the similar tag table 55, record data used for normalization processing (detailed later) of the HTML document. In the category table 56, a list of categories used for category feature learning, layout feature learning, and category determination processing is recorded.

  Next, an HTML document that is a target of category determination processing by the document classification device 1 will be described with reference to the drawings. FIG. 2 is a diagram illustrating an example of an HTML document. An HTML document is composed of HTML elements from a start tag to an end tag. For example, in the HTML document 61 of FIG. 2A, the description (62) starting with “<P>” and ending with “</ P>” is one HTML element.

  HTML elements form a tree structure, and another HTML element may be located below a certain HTML element. For example, the description (64) that starts with “<EM>” and ends with “</ EM>” included in the HTML element 62 is one HTML element positioned below the HTML element 62. FIG. 2B is a diagram representing the HTML elements constituting the HTML document 61 of FIG. 2A in the form of a tree structure. An HTML element 64 is positioned below the HTML element 62.

  In the following description, the remaining part of an HTML element obtained by removing the start tag and the end tag of the HTML element itself is referred to as the content of the HTML element. For example, in FIG. 2A, the description surrounded by a dotted line 63 is the content of the HTML element 62. It should be noted that the content of the HTML element does not have to be included as the content of the HTML element. In this case, the content of the HTML element 62 is a description obtained by removing the HTML element 64 from the description surrounded by the dotted line 63.

  In the present embodiment, an HTML tag is removed from the content of an HTML element, and an element used for category determination is divided for each word. That is, in this embodiment, one element is one word. A known morphological analysis technique is used for the division into words.

  In addition to one element being one word, two adjacent words may be treated as one element. In this case, from the two words, three elements are extracted, which are two adjacent words as separate elements and two words combined. Further, two words that are not adjacent to each other may be used as an element, or three or more words may be used as an element. By using these methods, it is possible to perform category determination in consideration of co-occurrence between words.

  Next, procedures for learning and category determination using the document classification device 1 will be described with reference to the drawings. FIG. 3 is a diagram showing a procedure for using the document classification device 1. First, category feature learning indicated by 65 is performed. An HTML document whose category is already known is prepared as a learning target. When the learning target document and the category to which the learning target document belongs are given to the category feature learning unit 2, the learning result is recorded in the category feature table 51. The above procedure is repeated for a plurality of documents to be learned.

  Next to the category feature learning, layout feature learning indicated by 66 is performed. For the layout feature learning, the same learning target document as that used in the category feature learning is used. Similar to the category feature learning, when the learning target document and the category to which the learning target document belongs are given to the layout feature learning unit 3, the learning result is recorded in the layout feature table 52. The above procedure is repeated for a plurality of documents to be learned. The layout feature learning uses the contents of the category feature table 51, the removal tag table 53, the removal attribute table 54, and the similar tag table 55.

  When the category feature learning and the layout feature learning are completed, the category determination indicated by 67 can be performed. When an HTML document to be subjected to category determination is given to the category probability determination unit 4, a category determination result 68 is obtained. The category determination result 68 indicates the probability that the category determination target document belongs to each category. The category determination result 68 shown in FIG. 3 is an example of the category determination result.

  Next, processing contents of the category feature learning unit 2 will be described with reference to the drawings. FIG. 4 is a block diagram showing the category feature learning unit 2 in detail. The learning target document input unit 21 removes the HTML tag from the learning target document and outputs it to the element extraction unit 22. The element extraction unit 22 decomposes the input into elements and outputs all the obtained elements to the category feature calculation unit 23. The category feature calculation unit 23 updates the contents of the category feature table 51 based on the given element and the category to which the learning target document belongs.

  FIG. 5 is a diagram showing the data structure of the category feature table 51. The category feature table 51 includes an element 69, a category 70, an appearance frequency 71, and a feature amount 72. One record of the category feature table 51 records the appearance frequency of an element in a certain category and how much the element represents the feature of the category. A smaller feature amount 72 means a general element used in documents of any category, and a larger feature amount 72 means a characteristic element that is likely to appear in a certain category. The feature amount 72 is calculated by using a known TF-IDF (Term Frequency-Inverse Document Frequency) method.

  FIG. 6 is a flowchart for explaining the processing contents of category feature learning. First, in step S1, a learning target document and a category to which the document belongs are received. In step S2, all HTML tags are removed from the learning target document. In step S3, the document from which the HTML tag has been removed is decomposed into elements, and all elements are extracted. In subsequent steps, processing is performed on the element extracted in step S3.

  In step S4, one unselected element is selected. In step S <b> 5, it is determined whether or not a record including the selected element in the “element” column exists in the category feature table 51. If the record exists in the category feature table 51, an affirmative determination is made in step S5, and the process proceeds to step S7. On the other hand, if a negative determination is made in step S5, the process proceeds to step S6. In step S 6, a record storing the selected element in the “element” column is added to the category feature table 51. At this time, records are added for all categories. That is, the selected element is stored in the “element” column, and records storing each category are added to the “category” column for the total number of categories. The “appearance frequency” of the record to be added is 0, and the “feature” is 1.

  In step S7, the record including the selected element in the “element” column and including the category to which the learning target document belongs in the “category” column is searched from the category feature table 51, and the “occurrence frequency” of the found record is set to 1. Is added. In step S8, a record including the selected element in the “element” column is searched from the category feature table 51, and the “feature” of the found record is updated according to a predetermined method. In step S9, it is determined whether all the elements extracted in step S3 have been selected. If there is an unselected element, a negative determination is made in step S9, and the process returns to step S4. On the other hand, if a positive determination is made in step S9, the category feature learning is terminated.

  Next, processing contents of the layout feature learning unit 3 will be described with reference to the drawings. FIG. 7 is a block diagram showing the layout feature learning unit 3 in detail. The learning target document input unit 31 performs normalization processing (described in detail later) on the learning target HTML document, and outputs the result to the layout analysis unit 32. The layout analysis unit 32 repeatedly outputs the content of the HTML element to the element extraction unit 33 for each HTML element constituting the input HTML document. As a result, the importance level of each HTML element is obtained from the layout feature analysis unit 34.

  The element extraction unit 33 decomposes the content of the input HTML element into elements, and outputs all the obtained elements to the layout feature analysis unit 34. The layout feature analysis unit 34 acquires the feature amount of each element from the category feature table 51 based on the given element and the category to which the learning target document belongs. Then, the importance level of the HTML element output from the layout analysis unit 32 is calculated from the obtained feature amounts of each element, and is output to the layout analysis unit 32. The layout analysis unit 32 that has received the importance continues to output the content of the HTML element for another HTML element and acquires the importance. The layout analysis unit 32 repeats the above procedure, so that the importance is finally calculated for all HTML elements included in the learning target document. The layout analysis unit 32 updates the layout feature table 52 based on the normalized HTML document and the importance of each HTML element included in the HTML document.

  FIG. 8 is a diagram showing the data structure of the layout feature table 52. A plurality of tables shown in FIG. 8 are recorded in the layout feature table 52. One table is created based on one HTML document. Of the data stored in the table, the HTML element 73 and the attribute value 74 reflect the original HTML document as it is. The identifier 75 is a unique identifier assigned by the layout analysis unit 32. The importance degree 76 is a numerical value indicating how important the HTML element is for category determination, and the larger the value, the greater the influence on the category determination.

  FIG. 9 is a flowchart showing a processing procedure for layout feature learning. First, in step S10, a learning target document and a category to which the document belongs are received. In step S11, normalization processing (described in detail later) is performed on the learning target document. In step S12, an identifier that is unique within the learning target document and 1 that is an initial value of importance are given to each HTML element constituting the learning target document.

  In step S13, one unselected HTML element is selected from the document to be learned. In step S14, the content of the HTML element selected in step S13 is taken out into elements. In step S15, one unselected element is selected from the elements extracted in step S14. In step S16, the category feature table 51 is searched for a record that includes the selected element in the “element” column and includes the category to which the learning target document belongs in the “category” column. In step S17, the “feature” is extracted from the record found in step S16, and the importance of the HTML element selected in step S13 is multiplied.

  In step S18, it is determined whether all the elements extracted in step S14 have been selected. If all the elements have been selected, an affirmative determination is made in step S18, and the process proceeds to step S19. On the other hand, if a negative determination is made in step S18, the process returns to step S15. In step S19, it is determined whether or not all HTML elements constituting the learning target document have been selected. If an unselected HTML element exists, a negative determination is made in step S19, and the process returns to step S13. On the other hand, if an affirmative determination is made in step S19, the layout feature learning ends.

  Next, the HTML document normalization process executed in step S11 of FIG. 9 will be described with reference to the drawings. FIG. 10 is a flowchart showing the normalization processing procedure of the HTML document. In step S21, an unknown HTML tag is deleted from the HTML document. In step S22, the missing end tag and attribute, and the omitted end tag and attribute are complemented. In step S23, the HTML tag included in the removal tag table 53 is deleted. In step S24, the HTML tags to be identified are replaced with the same HTML tags according to the similar tag table 55. In step S25, the attribute included in the removal attribute table 54 is deleted. In step S26, blank characters and line feed characters are removed and added. Specifically, first, redundant white space characters and line feed characters are deleted. Next, a line feed character is added so that the HTML tag is at the beginning of the line. Similarly, a line feed character is added so that the content of the HTML element starts from the beginning of the line.

  Next, processing contents of the category probability determination unit 4 will be described with reference to the drawings. FIG. 11 is a block diagram showing the category probability determination unit 4 in detail. The determination target document input unit 41 searches the table included in the layout feature table 52 for a table having an HTML element structure most similar to the HTML document to be determined and outputs the table to the category score calculation unit 44. . In addition, the determination target HTML document is normalized as described above and output to the category determination unit 42. The category determination unit 42 manages category scores. The category score is a numerical value assigned to each category, and the initial value is 0. The higher the category score, the higher the probability that the determination target document belongs. The category determination unit 42 repeatedly outputs the content of the HTML element to the element extraction unit 43 for each HTML element constituting the input HTML document. As a result, category score update information is received from the category score calculation unit 4.

  The element extraction unit 43 decomposes the content of the input HTML element into elements, and outputs all the obtained elements to the category score calculation unit 44. The category score calculation unit 44 outputs the category score to be added to each category to the category determination unit 42 based on the given elements, the table given from the determination target document input unit 41, and the category feature table 51. To do. Upon receiving the category score, the category determination unit 2 continues to output the content of the HTML element for another HTML element, and acquires the category score. The category determination unit 42 repeats the above procedure, and finally, category scores in all categories are calculated. The category determination unit 42 calculates the probability that the determination target document belongs to each category from the category score, and outputs it as a determination result.

  FIG. 12 is a flowchart showing a processing procedure for category determination. First, in step S27, an initial category score of 0 is given to all categories. In step S28, normalization processing is performed on the category determination target document according to the procedure described in FIG. In step S29, a table including the structure of the HTML element most similar to the category determination target document is acquired from the tables recorded in the layout feature table 52. A specific method for measuring the similarity of HTML elements is that a HTML document obtained by deleting an HTML tag from a category determination target document and an HTML document based on each table included in the layout feature table 52 are known documents. The comparison is performed in line units by a comparison algorithm (for example, an algorithm for obtaining the longest common partial character string using dynamic programming). As a result of comparison, a table having the highest identity is acquired. For the determination of identity, the ratio of different parts may be used, or the edit distance may be used. In the following description, the table acquired by the above procedure is called a comparison target table.

  In step S30, one unselected HTML element is selected from the determination target HTML document. In step S31, the content of the HTML element selected in step S30 is decomposed into elements and taken out. In step S32, one unselected element is selected from the elements extracted in step S31. In step S <b> 33, it is determined whether or not a record including the selected element in the “element” column exists in the category feature table 51. If the corresponding record exists in the category feature table 51, an affirmative determination is made in step S33, and the process proceeds to step S34. On the other hand, if a negative determination is made in step S33, the process proceeds to step S38. In step S34, it is determined whether or not the HTML element selected in step S30 exists in the comparison target table. When the same HTML element exists in the comparison target table, an affirmative determination is made in step S34, and the process proceeds to step S35. On the other hand, if a negative determination is made in step S34, the process proceeds to step S37.

  In step S35, the importance is acquired from the same HTML element as the HTML element selected in step S30, which exists in the comparison target table. In step S36, the following processing is performed for each category. First, a record including the element selected in step S32 in the “element” column and including the category to be processed in the “category” column is searched from the category feature table 51 to obtain “feature”. Next, the importance acquired in step S35 is multiplied by the above-mentioned “feature amount”, and the result is added to the category score of the category to be processed.

  In step S37, the following processing is performed for each category. First, a record including the element selected in step S32 in the “element” column and including the category to be processed in the “category” column is searched from the category feature table 51 to obtain “feature”. Next, the above “feature amount” is added to the category score of the category to be processed. In step S37, unlike step S36, the multiplication of importance is not performed.

  In step S38, it is determined whether all the elements extracted in step S31 have been selected. When all the elements have been selected, an affirmative determination is made in step S38, and the process proceeds to step S39. On the other hand, if a negative determination is made in step S38, the process returns to step S32. In step S39, it is determined whether all the HTML elements existing in the determination target HTML document have been selected. If there is an unselected HTML element, a negative determination is made in step S39, and the process returns to step S30. On the other hand, if an affirmative determination is made in step S39, the category determination process ends.

  A category determination result 68 (FIG. 3) is obtained by dividing the category score obtained by the above category determination process by the total of the category scores.

According to the document classification method according to the first embodiment described above, the following operational effects can be obtained.
(1) The location important for category determination is specified by the layout feature. As a result, even when handling an HTML document having a universal layout structure, the accuracy of category determination is improved because importance is given to a portion where a difference for each category is likely to appear.
(2) A specific HTML tag or attribute can be deleted in advance by the removal tag table and the removal attribute table. As a result, HTML tags that do not contribute to the features for each category are not included in the layout features, so that the accuracy of category determination is improved.
(3) Elements in HTML elements that do not exist in the comparison target table are not ignored but are simply added to the category score. As a result, category determination can be performed in consideration of characteristic elements that appear in places that are not common parts of the layout structure, so that the accuracy of category determination is improved.

  In the first embodiment described above, the determination target document is an HTML document, and so-called text data is a processing target. In the second embodiment described in detail below, it is assumed that the document to be determined is an image.

-Second embodiment-
In the present embodiment, the learning target document and the determination target document are image data. For example, in the case of an image of a person category, a skin color exists near the center of the image, and in the case of an image of a sunset category, there are many red colors at the top of the image. Is learned as a category feature. Next, layout characteristics are learned. For example, in the image of the sunset category, when the image is divided into upper and lower parts, layout features are learned such that the upper part of the image is important.

  The category determination of a specific image is performed from the category features and layout features learned by the above processing. For example, when the determination target image is divided vertically, it is determined that the upper part of the image is important from the learning result described above, and the comparison result at the upper part of the image is more important than the lower part of the image.

  According to the document classification method according to the second embodiment described above, in addition to the effects obtained by the document classification method according to the first embodiment, the following effects can be obtained. The category determination target document is image data, and the learning results of the category features and layout features are used. Accordingly, category determination can be performed on a document other than document data using the layout feature.

  In the above-described embodiment, automatic document classification is performed using layout analysis. It is also possible to apply the layout analysis method of the present invention to a document search method. In the third embodiment described in detail below, a document is searched from the result of layout analysis.

--Third embodiment--
A document search apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 13 is a block diagram showing the overall configuration of the document search apparatus according to the present embodiment. When a keyword is input, the document search apparatus 101 shown in the figure executes search processing based on the keyword for the document group included in the search target document DB 157. Specifically, documents in which keywords are used are specified, and those documents are ordered based on the strength of the relationship with the keywords. The search results are output as a plurality of ordered documents.

  When the document search apparatus 101 is used, it is necessary to perform learning in advance as in the document classification apparatus 1 in the first embodiment. Details of the learning are the same as those of the document classification device 1 and are therefore omitted.

  The document search apparatus 101 includes a category feature learning unit 102, a layout feature learning unit 103, a document search unit 104, and a data storage unit 105. The category feature learning unit 102 and the layout feature learning unit 103 are the same as the document classification device 1 in the first embodiment.

  The data storage unit 105 includes a search target document DB 157 in addition to the data storage unit 5 in the first embodiment. The search target document DB 157 stores a plurality of documents to be searched.

  The document search unit 104 includes a keyword input unit 141 and a score calculation unit 142. Upon receiving the keyword to be searched, the keyword input unit 141 first extracts a document group including the keyword from the search target document DB 157. Next, each extracted document is transmitted to the score calculation unit 142 together with the keyword.

  When the score calculation unit 142 receives the keyword and the document to be processed, as in the determination target document input unit 41 in the first embodiment, the score calculation unit 142 creates a table having an HTML element structure most similar to the document to be processed. , From the layout feature table 152. Next, the appearance location of the keyword in the document to be processed is checked. For each keyword, the importance corresponding to the appearance location in the layout feature table 152 is added to the score, and the final score is returned to the keyword input unit 141.

  The keyword input unit 141 that has executed the above processing for all the documents included in the document group arranges the document group in the order of points, and outputs it as a search result. As described above, a search result based on the layout structure is obtained.

  According to the document search method according to the third embodiment described above, the following operational effects can be obtained. For a document group including keywords, the keywords are weighted based on the layout structure of the document, not simply by the number of occurrences of the keywords. As a result, a document in which a keyword is used in the subject part of the document is ranked higher in the search result, so that the search accuracy is improved.

The following modifications are also within the scope of the present invention.
(1) In the layout feature table, one table may be recorded for a plurality of learning target document groups having similar structures instead of recording one table for each learning target document. . In this case, only HTML elements common to a plurality of documents are recorded in the table. For extraction of common HTML elements, the same technique as that for determining the similarity of HTML documents can be used. Further, the importance of HTML elements may be averaged among a plurality of HTML documents.

(2) Document elements other than words and co-occurrence words may be used. For example, two adjacent characters may be used as one element without performing morphological analysis, and the distance between words may be used as one element. An image embedded in a document may be used as one element.

(3) The feature quantity recorded in the category feature table may use a technique other than the TF-IDF method.

(4) The document to be subjected to category determination may be in a format other than an HTML document as long as it has some structure. For example, it may be a document created by a so-called word processor or an image. When an image is a determination target, the element may be a pixel position or color, or may be handled after being converted into a markup language such as a Scalable Vector Format (SVG) format.

(5) In a document having a sentence, a pattern appearing in the sentence may be used for layout feature learning. For example, when a sentence such as “Introduction” or “Introduction” appears, it can be regarded as an independent part in the layout structure.

(6) The category of the document used for layout feature learning may not be known. In this case, as a category to which the learning target document belongs, a category in which the learning target document is classified based on a well-known automatic document classification technique and category feature learning results is used.

(7) The layout analysis of the present invention can be used in addition to category determination and keyword search. For example, in document summarization processing, a portion with a low importance can be largely omitted, and a portion with a high importance can be left.

  As long as the characteristics of the present invention are not impaired, the present invention is not limited to the above-described embodiments, and other forms conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. .

It is a block diagram which shows the whole structure of the document classification | category apparatus in 1st Embodiment. It is a figure which shows the example of the HTML document made into a process target by 1st Embodiment. It is a figure which shows the procedure which utilizes the document classification device in 1st Embodiment. It is a block diagram for demonstrating the processing content of a category feature learning part. It is a figure which shows the data structure of a category feature table. It is a flowchart which shows the process sequence of category feature learning. It is a block diagram for demonstrating the processing content of a layout feature learning part. It is a figure which shows the data structure of a layout feature table. It is a flowchart which shows the processing procedure of layout feature learning. It is a flowchart which shows the process sequence which normalizes an HTML document. It is a block diagram for demonstrating the processing content of a category probability determination part. It is a flowchart which shows the process sequence of category determination. It is a block diagram which shows the whole structure of the document search device in 3rd Embodiment.

Explanation of symbols

1 Document Classification Device 2 Category Feature Learning Unit 3 Layout Feature Learning Unit 4 Category Probability Determination Unit 5 Data Storage Unit 51 Category Feature Table 52 Layout Feature Table

Claims (21)

An analysis program for a document having a layout structure,
An element extraction process for extracting a plurality of components from a learning target document having a layout structure;
For each of the components extracted from the learning target document, a category feature learning process for calculating a feature amount in the category of the learning target document;
Layout feature learning processing for calculating the importance of each partial structure for a plurality of partial structures included in the layout structure of the learning target document based on the learning target document and the feature amount;
A document analysis program characterized in that it is executed on a computer. The document analysis program according to claim 1, further comprising:
Based on the feature amount calculated by the category feature learning process, the importance calculated by the layout feature learning process, and the layout structure of the learning target document, a category of a determination target document having a layout structure is determined. The category determination process to determine
A document analysis program characterized by being executed by a computer. The document analysis program according to claim 2,
The category determination process outputs a probability that the determination target document belongs to each of a plurality of categories. The document analysis program according to claim 1, further comprising:
A predetermined keyword is searched from a plurality of search target documents having a layout structure, and each of the search target documents is determined based on the layout structure of the learning target document and the importance calculated by the layout feature learning process. Document search processing for calculating the degree of association with the keyword,
A document analysis program characterized by being executed by a computer. In the document analysis program as described in any one of Claims 1-3,
A document analysis program characterized in that the document is image data. The document analysis program according to claim 5,
A document analysis program characterized in that a component extracted from the image data by the element extraction means is a pixel. In the document analysis program as described in any one of Claims 1-4,
A document analysis program characterized in that the document is a document described by HTML (HyperText Markup Language). In the document analysis program according to any one of claims 1 to 4 or 7,
A document analysis program, wherein the component is a word. In the document analysis program according to any one of claims 1 to 4 or 7,
The document analysis program, wherein the component is a word and a co-occurrence relationship of the word. The document analysis program according to claim 8 or 9,
A document analysis program characterized in that the word is extracted by morphological analysis. In the document analysis program as described in any one of Claims 1-10,
The document analysis program characterized in that the feature amount is calculated by a TF-IDF (Term Frequency-Inverse Document Frequency) method. In the document analysis program as described in any one of Claims 1-11,
The layout feature learning process creates a new document by extracting a common part of the layout structure from a plurality of documents to be learned having a similar layout structure, and for each partial structure in the layout structure of the new document A document analysis program characterized by calculating the importance of. In the document analysis program as described in any one of Claims 1-12,
The document analysis program, wherein the layout structure includes a tree structure. The document analysis program according to claim 13,
The layout feature learning process excludes a partial structure positioned at a lower level from a partial structure positioned at a higher level when calculating the importance of the partial structure positioned at a higher level in a tree structure. program. The document analysis program according to claim 13,
The layout feature learning process includes a partial structure positioned at a lower level in a partial structure positioned at a higher level when the importance of a partial structure positioned at a higher level in a tree structure is calculated. . An apparatus for analyzing a document having a layout structure,
An element extraction unit that extracts a plurality of components from a learning target document having a layout structure;
A category feature learning unit that calculates a feature amount in a category of the learning target document for each of the components extracted from the learning target document;
A layout feature learning unit that calculates importance for each partial structure for a plurality of partial structures included in the layout structure of the learning target document based on the learning target document and the feature amount;
A document analysis apparatus comprising: The document analysis apparatus according to claim 16, further comprising:
Based on the feature amount calculated by the category feature learning unit, the importance calculated by the layout feature learning unit, and the layout structure of the learning target document, a category of a determination target document having a layout structure is determined. A document analysis apparatus comprising a category determination unit for determining. The document analysis apparatus according to claim 16, further comprising:
A predetermined keyword is searched from a plurality of search target documents having a layout structure, and each of the search target documents is determined based on the layout structure of the learning target document and the importance calculated by the layout feature learning unit. A document analysis apparatus comprising a document search unit for calculating a degree of association with the keyword. A method for analyzing a document having a layout structure,
An element extraction step for extracting a plurality of components from a learning target document having a layout structure;
A category feature learning step for calculating a feature amount in the category of the learning target document for each of the components extracted from the learning target document;
A layout feature learning step of calculating importance for each partial structure for a plurality of partial structures included in the layout structure of the learning target document based on the learning target document and the feature amount;
A document analysis method comprising: The document analysis method according to claim 19, further comprising:
Based on the feature amount calculated in the category feature learning step, the importance calculated in the layout feature learning step, and the layout structure of the learning target document, a category of a determination target document having a layout structure is determined. A document analysis method comprising a category determination step of determining. The document analysis method according to claim 19, further comprising:
A predetermined keyword is searched from a plurality of search target documents having a layout structure, and based on the layout structure of the learning target document and the importance calculated by the layout feature learning step, A document analysis method comprising a document search step of calculating a degree of association with the keyword.

Images