JP2007122403A - Automatic extraction device, extraction method and extraction program for document title and related information - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a title extraction device for automatically and accurately extracting a document title.
A document title extraction device includes a title candidate sentence extraction unit that extracts a plurality of title candidate sentences from a text document input from a document input unit, and a feature amount of each of the extracted plurality of title candidate sentences. A feature amount extraction unit 34 for extracting a document title, a title determination unit 36 for determining a document title from a plurality of title candidate sentences based on the extracted feature amount, and an output unit 38 for outputting an extraction result. The feature amount includes similarity information that is a function of similarity between at least a title candidate sentence and a plurality of sentences in the document.
[Selection] Figure 2

Description

  The present invention relates to a document title extraction apparatus that automatically extracts a document title from a document read by a scanner or the like.

  An apparatus for reading a general document of a paper original by an optical scanner or the like and extracting a document title from digitized image data is being put into practical use. For example, Patent Document 1 relates to a title extraction device that easily extracts a title portion from a document image obtained by converting a document into image data. According to this, a circumscribed area that is connected to black pixels in a document image is disclosed. The rectangle area to be extracted is extracted as a character rectangle, and a plurality of adjacent character rectangles are integrated to extract a rectangle area circumscribing those character rectangles as a character string rectangle, and then the underline attribute of each character string rectangle , Calculate the point of title likeness based on attributes such as framed attributes, ruled line attributes, etc., and the position of the character string rectangle in the document image and the mutual positional relationship, and the character string rectangle that acquired the high point as the title rectangle To extract.

  Patent Document 2 discloses a natural language analysis means for identifying a character code in a character string rectangle cut out from a document image and analyzing the certainty of character code identification and the likelihood of a natural language title. The present invention relates to a title extracting apparatus that extracts titles by methods such as statistical information on endings, centering, underline, specific font, and character rectangle size.

  Non-Patent Document 1 discloses a technology that makes it possible to extract the address, city name, URL, and time of a technical paper using a regular expression pattern, and extracts a portion that is not extracted as a paper start portion as an author and title.

  Non-Patent Document 2 uses sentence features such as linguistic features (number of words, line position, word to non-word ratio, initial capital letter to lowercase letter ratio, number ratio) for the sentence start part. , A technique for determining a title using SVM is disclosed.

JP-A-9-134406 JP 2000-148788 A E Berkowitz, M Elkhadiri, T Sahouri and M Abraham. 2004. Intelligent Content Based Title and Author Name Extraction from Formatted Documents.Proceedings Fifteenth Midwest Artificial Intelligence and Cognitive Science Conference.age 119-124. Hui Han, C Giles, E Manavoglu, Hongyuan Zha, Zhenyue Zhang and E Fox. 2003. Automatic Document Metadata Extraction using Support Vector Machines. ACM / IEEE Joint Conference on Digital Libraries. Page 37-48.

  However, since the title extraction apparatus of Patent Document 1 performs title extraction using a layout characteristic of a line area for an atypical document, there is a problem that the extraction rate is not sufficient. In Patent Document 2, the title is determined using some title attributes. However, there are many short character string rectangles having title attributes for a document having a plurality of short character string rectangles. There is a problem that it is easy to judge.

  In addition, since the technology disclosed in Non-Patent Document 1 and Non-Patent Document 2 depends on the structure of the document, it is difficult to apply to a document other than a technical paper, and when the start information of the document is small, There is a problem that a correct title cannot be extracted.

  The present invention solves the above-described conventional problems, does not necessarily depend on the layout and content range of a document, fully utilizes linguistic knowledge, the length of a candidate title sentence, the similarity between a candidate sentence and another sentence Information such as ranking, author, organization name, title keyword, distance between candidate sentence and author, prohibited keyword, zip code, symbol, etc. are used as feature quantities of the title candidate sentence, and this feature quantity is classified by a classification device (for example, SVM). A title extraction device that can extract the document title and related information with high accuracy using a flexible determination method by making the best use of the unique attributes of the title by determining whether or not it is a title and extraction It is an object to provide a method and an extraction program.

  A document title extraction apparatus according to the present invention includes a title candidate sentence extraction unit that extracts a plurality of title candidate sentences from a document, a feature amount extraction unit that extracts a feature quantity of each of the extracted title candidate sentences, and an extraction A determination unit that determines a document title from a plurality of title candidate sentences based on the feature amount that has been output, and an output unit that outputs a determination result. The feature amount includes at least a title candidate sentence and a plurality of sentences in the document. Similarity information, which is a function of the degree of similarity, is included.

  Preferably, the similarity information includes rank information indicating a degree of similarity between the title candidate sentence and a plurality of sentences in the document. The similarity information is calculated using the vector information of the sub character string selected from the title candidate sentence and the vector information of the sub character string selected from the sentence in the document. The vector information is calculated based on the appearance frequency of N (N is a natural number of 2 or more) gram selected from the title candidate sentence and the appearance frequency of N gram selected from the sentence in the document. By using such similarity information, it is possible to extract and determine a document title with high accuracy using linguistic information without using morphological analysis.

  Further, when the vector information is calculated based on the appearance frequency of the N-gram, if the predetermined use-prohibited N-gram is included, the vector information is corrected. By excluding a character string that cannot be a title or a character string that is unlikely to be a title, the determination and extraction accuracy of the document title can be improved.

  The similarity information may be calculated based on the edit distance between the title candidate sentence and the sentence in the document, or may be calculated based on the length of the maximum common character string between the title candidate sentence and the sentence in the document. It may be.

  Further, when the title candidate sentence includes a predetermined title keyword, the feature amount includes title keyword information indicating the position and appearance frequency of the keyword, or is determined in the title candidate sentence. When a use-prohibited title keyword is included, use-prohibited title keyword information indicating the position and appearance frequency of the use-prohibited title keyword may be included. Thereby, various features can be included in the feature amount of the title candidate sentence, and the determination accuracy of the title can be improved.

  The determination means for determining the document title extracts an optimal title candidate sentence based on the feature amount of the title candidate sentence. Preferably, the feature amount is classified by SVM (support vector machine) and the determination is performed. The output means includes a display device such as a display, for example, and outputs the determined title sentence and related information. Related information includes authors and organization names.

  The document title extracting device further includes an input unit for inputting an image document, and a text document extracting unit for extracting a text document from the input image document. The title candidate sentence extracting unit is a title candidate from the extracted text document. A sentence may be extracted. The input means for inputting the image document includes a scanner that optically reads the image document, and extracts text data by OCR or the like from the image document data read by the scanner. Preferably, the title candidate sentence extraction unit extracts title candidate sentences in a certain candidate target range from the top of the text document. This is because a document that can be a title is often included in the top part.

  The feature amount may include layout information obtained from the input image document. By using these pieces of information, the document title determination accuracy is improved.

  A method for extracting titles from a document according to the present invention includes a step of extracting a plurality of title candidate sentences from a document, and a feature amount including similarity information between the title candidate sentences and a plurality of sentences in the document. A step of extracting a sentence; a step of determining a document title from a plurality of title candidate sentences based on the extracted feature amount; and a step of outputting a determination result. Furthermore, the program for extracting a title from a document according to the present invention includes a step of extracting a plurality of title candidate sentences from the document, and a feature amount including similarity information between the title candidate sentence and the plurality of sentences in the document. Extracting candidate sentences, determining a document title from a plurality of title candidate sentences based on the extracted feature quantities, and outputting a determination result.

  According to the document title extracting apparatus of the present invention, each feature amount of the title candidate sentence is extracted, and the feature amount is a function of similarity between the title candidate sentence and a plurality of sentences in the document. It is possible to extract document titles and related attributes with high accuracy using a flexible judgment method that fully utilizes language knowledge, without necessarily depending on document layout, image information and content range. It is. Preferably, the extraction method using SVM does not affect the imperfection of discrimination rule application and the misrecognition of OCR, so the title of the scanned text document (text document on which OCR has been performed) and related Ideal for automatic extraction of information. By using SVM, it is possible to improve the extraction performance (extraction range expansion, extraction accuracy) of the system by learning.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, exemplary embodiments of the invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a document title extraction apparatus according to an embodiment of the present invention. The title extraction device 10 includes an input device 12, a display device 14, a main storage device 16, a storage device 18, a central processing unit (CPU) 20, and a bus 22 for connecting them.

  The input device 12 includes a keyboard for inputting information by key operation, an optical reading device (scanner) for optically reading a document or the like written on a manuscript, an input interface for inputting data from an external device, an external memory, or the like. Including. The display device 14 includes a display that displays a title extracted from the document and related information thereof. The main storage device 16 includes a ROM or a RAM, and stores a title candidate sentence from a document, a feature amount of the title candidate sentence, a program for determining a document title, arithmetically processed data, and the like. To do. The storage device 18 includes, for example, a large-capacity storage device such as a hard disk, and stores image document data optically read by a scanner, various dictionary databases used for extracting feature amounts, and the like. A CPU (Central Processing Unit) 20 controls each unit in accordance with a program stored in the main storage device 16.

  FIG. 2 is a block diagram functionally showing the document text extraction apparatus. The document input unit 30 inputs a text sentence of a document. The text sentence may be, for example, text data received via an input interface or text data extracted by an OCR (character recognition device) from image document data optically read by a scanner. Of course, it may be obtained by other methods.

  The candidate title sentence extraction unit 32 extracts candidate titles that can be candidate titles from the input text sentence. The title candidate sentence extraction unit sets a predetermined range from the beginning of the input text document as a candidate target range, and sets a portion separated from the text sentence included in the candidate target range by a specific symbol and a line feed symbol as a title candidate sentence. .

  FIG. 3 shows an operation flow by the title candidate sentence extraction unit 32. The title candidate sentence extraction unit 32 sets the α% portion and the candidate target range from the top of the input document (step S101). α is an integer, for example, 50. Next, the title candidate sentence extraction unit separates the text sentence included in the candidate target range by a symbol (;.?! = ~ @ # $% ^ & * _ | \ N;.?! ...) and a line feed symbol. This part is set as a title candidate sentence (step S102). Finally, the separated title candidate sentences are stored as a set in a storage device or the like (step S103).

  FIG. 4 shows an example in which Japanese is an input document. FIG. 6A shows an input document of a text sentence read by a scanner or the like, and FIG. 6B shows an example in which a 50% portion from the top of the input document is extracted as a candidate target range. (C) shows a set of title candidate sentences separated from the candidate target range by symbols and line feeds.

  Returning to FIG. 2 again, the extracted title candidate sentence is supplied to the feature amount extraction unit 34. The feature amount extraction unit 34 extracts a feature amount for determining the similarity between the title candidate sentences from all the title candidate sentences. As shown in FIG. 5, the feature amount includes candidate sentence length 40, similarity rank information 41, author information 42, organization name information 43, title keyword information 44, author position information 45, title prohibited keyword 46, It is composed of nine elements such as a zip code 47 and a number of symbols 48.

  FIG. 6 is a diagram for explaining a method of calculating information of each unit constituting the feature amount. The “candidate sentence length” 40 is the length of the title candidate sentence, and its unit is Byte. For example, it is represented by the value of the length (Byte) / 150 (constant) of the candidate sentence.

  The “similarity rank information” 41 calculates the similarity between the title candidate sentence and other sentences in the document, and sets the highest similarity as the similarity of the title candidate sentence. All title candidates are sorted in descending order of similarity, and M is assigned from rank 1 to M (M is the number of title candidate sentences). Similarity rank information = 1 / represented by similarity rank.

The similarity can be obtained by the following method.
Method 1: A method of obtaining the similarity between sentences (or the distance between sentences) using the VSM vector feature amount of the title candidate sentence. As the VSM vector feature amount, TF (TF / IDF) of a word, or a function value of TF and IDF can be used. Further, the character string may be divided into N-grams, and N-grams of TF (TF / IDF) or function values of TF and IDF may be used. Furthermore, a method for calculating similarity and distance between vectors that are publicly available may be used.

Method 2: A method of obtaining a distance between sentences by an edit distance between character strings.
Method 3: A method of obtaining similarity between sentences by the length of the maximum common character string between two character strings.
Method 4: Any other published method.
In this embodiment, as will be described later, the character string is divided into two grams, and the similarity between the two-gram vector feature amounts is calculated.

  In the “author information” 42, when the author is included in the title candidate sentence, the flag is set to “1”, and the others are “0”. For example, a publicly-known name extraction technique or a person name extraction technique can be used. FIG. 7 is a surname dictionary showing the surnames and the number of Japanese surnames listed in a telephone directory. If the title candidate sentence and the surname dictionary of FIG. 7 are compared and hit, the flag may be set to “1”, and if not hit, “0” may be set. Similarly to the surname dictionary, a Japanese name dictionary is prepared, the title candidate sentence is compared with the name dictionary, and the flag may be set to “1” if hit and “0” if not hit. Alternatively, the flag may be set to “1” only when both the last name and the name are hit. Whether or not to hit may be a partial match such as a forward match and a backward match as well as a case where the character strings completely match.

  In the “organization name information” 43, when the title candidate sentence includes the organization name information, the flag is set to “1”, and the others are “0”. For example, the organization name dictionary in which the organization name is registered in advance and the title candidate sentence are compared. If the organization name is hit, the flag is “1”, and if it is not hit, “0” is set. Whether or not to hit may be a partial match such as a forward match and a backward match as well as a case where the character strings completely match.

  The “title keyword information” 44 is information indicating whether or not a title candidate sentence includes a predetermined title keyword, and is the sum of the position of the title keyword and the frequency of appearance of the title keyword. For example, the title keyword is registered as a title keyword dictionary. The “author position information” 45 is assigned the title candidate sentence numbers from 1 in ascending order (ascending order) before and after (position) appearing in the document. Suppose that an author appears for the first time in the i-th title candidate sentence. “Author author position information” = 1 for the candidate title sentences numbered 1 to i + 3, and “author position information” = 0 for the other candidate sentences.

  “Title forbidden keyword” 46 is information indicating whether or not the title candidate sentence includes a predetermined title forbidden keyword, and the position of the title forbidden keyword and the frequency of appearance of the title forbidden keyword are totaled. Is. For the title-prohibited keyword, a character string that is not used in the title or a character string that is unlikely to be used is registered in the dictionary in advance, and it is checked whether or not it corresponds to that.

  The “zip code” 47 is a postal code consisting of six consecutive digits. The flag is set to “1” when the postal code is included in the title candidate sentence, and “0” in other cases. The “number of symbols” 48 is the number of “,”, “.”, “;” Included in the title candidate sentence.

  In addition, as shown in FIG. 5, the candidate sentence similarity feature amount is an example of nine elements, but is not limited thereto. The title extraction may include at least the second “similarity rank information”, and “similarity rank information” and other information may be appropriately combined. For example, “similarity rank information” and the fifth dimension “title keyword information” may be used as the feature quantity, or “similarity rank information” and the seventh dimension “title prohibited keyword information” may be used as the feature quantity. Of course, other language information may be added. For example, address information or the like may be added. Furthermore, if it is an image document read from a scanner, document layout information (such as the positional relationship of candidate sentences) and image information (such as character size, color, and character type) can be obtained. It is also possible to add as quantity information.

  Returning to FIG. 2 again, the feature amounts of all the title candidate sentences extracted by the feature amount extraction unit 34 are supplied to the title determination unit 36. The title determination unit 36 includes a determination classification unit configured by learning. The classification unit may use any published classification technique. For example, a SVM (Support Vector Machine) classification technique can be used as a specific example. The SVM engine is described in, for example, the paper “Text Classification by Support Vector Machine”, 1998, Natural Language Processing, 128-24.

  When the title is extracted by the title determination unit 36, the extraction result is supplied to the extraction result output unit 38. The extraction result output unit 38 causes the display device 14 to display the extracted title. At the same time, related information such as authors may be displayed.

  Next, a method for calculating the similarity between feature quantities will be described. First, all continuous two-character strings (2 grams) are extracted from the left to the right of the title candidate sentence. For example, if the title candidate sentence is “intellectual property right”, a character string of 2 grams is cut out such as “intellectual”, “target property”, “property”, and “industry right”. The 2-gram vector feature amount of the title candidate sentence is represented by A = (β1, β2,... ΒN). A 2-gram vector feature amount of another sentence in the document is represented by B = (β′1, β′2,... Β′N). The following formula calculates all similarities sim (A, B) between the title candidate sentence and other sentences in the document.

  FIG. 8 shows a flow when calculating a 2-gram vector feature. All consecutive two-character strings (2 grams) are extracted from the left to the right of the title candidate sentence (step S201). Next, the appearance frequency # (x) of all 2 grams is obtained (step S202). Next, the forbidden 2 gram dictionary 50 in which 2 grams are registered as prohibition of use is referred to. If the forbidden 2 gram is included, the dimension number of the vector feature amount is corrected (step S203). Finally, vector feature amounts A and B are generated using the corrected 2-gram frequency # ′ (x) (step S204).

FIG. 9 shows a method of calculating the 2-gram frequency # ′ (x).
MI (x, y): mutual information of 2 grams x, y;
# (X) is the number of times 2 grams X appears in this document;
N is the number of times 2 grams of all appear;
# (X, y): number of times X and Y co-occur in this document;

  As described above, the document title extraction apparatus according to the present embodiment extracts the feature amount for similarity from the title candidate sentence, and extracts and determines the document title based on the feature amount. By combining the standards, it is possible to extract the document title and related information with high accuracy. Since the title and related information are extracted completely based on the content of the document, a highly versatile document title that does not necessarily depend on the document layout, image information, and content range can be extracted. The keyword extraction range does not depend on the area because the keyword information of the paper, the outline content, and the related information of the specialized area are not necessarily required. Furthermore, without using morphological analysis, a two-character sub-character string selected from the title candidate sentence is extracted, and the mutual information between all the two-character strings existing in the sentence is used as the vector of the sentence, By making the Cos value between vectors the similarity between sentences, there is a feature that slight misrecognition of OCR hardly affects the determination of the title, which is suitable for extracting the title of the scanned image document. Information such as length of title candidate sentence, ranking of similarity, author, organization name, title keyword, distance between candidate sentence and author, keyword forbidden keyword, zip code, symbol, etc. is used as a feature amount of the sentence, and classification device (for example, It is possible to extract a title with high accuracy by determining whether or not it is a title using SVM).

  FIG. 10 is a block diagram showing a second embodiment of the document title extracting apparatus of the present invention. In the second embodiment, the document input unit 30 shown in FIG. 2 is modified. The image document input unit 60 inputs an image document and outputs the input image document data to the layout and image information extraction unit 62. For example, a scanner or the like can be used for input. The layout and image information extraction unit 62 extracts layout information and image information from the image document data. The layout information includes, for example, information such as the positional relationship of title candidate sentences, and the image information includes information such as character size, color, and font. For extracting the layout information and the image information, a known technique can be used. For example, the layout information and the image information are disclosed in JP-A-9-134406 and JP-A-2000-148788.

  Next, the text information extraction unit 64 extracts text information from the image information by, for example, OCR. As the OCR, a known technique or a commercially available OCR can be used. The extracted text information is supplied to the title candidate sentence extraction unit 32. In the second embodiment, when the feature amount extraction unit 34 extracts the feature amount of the title candidate sentence, the layout information and image information obtained by the layout and image information extraction unit 62 can be included.

  In the second embodiment, an image document can be read by a scanner or the like, and a document title can be automatically extracted from the read image document. At the same time, by adding the layout information included in the image document data to the feature amount of the title candidate sentence, it is possible to further improve the document title determination accuracy.

  Next, an example when the document title extracting apparatus of the present invention is implemented for a Chinese document will be described. As for a Chinese document, as shown in FIG. 2, a text document is input by the document input unit 30, and a title candidate sentence is extracted from the document by a title candidate sentence extraction unit 32. As will be described below, the feature amount extraction unit 34 is optimized in terms of author names, organization names, etc., in Chinese feature amounts.

  FIG. 11 shows a Chinese name dictionary and a name dictionary. The method here is limited to Chinese names. The author extraction method is composed of Chinese name identification and name identification, and the following criteria can be used.

Since there are few Chinese names with more than 4 characters, if the title candidate sentence has 4 or more characters, it is determined that the name is not a person name.
Since there are very few two-letter surnames in Chinese names, it is determined whether the first two characters of the character string of the title candidate sentence are two-letter surnames. If it is a two character surname, it can be determined that the candidate character string is a surname.
Calculate person name judgment value. First, a list table of characters appearing in Chinese surnames having an appearance frequency (referred to as a surname dictionary) and a list of characters appearing in names (referred to as a name dictionary) are prepared. The first character dictionary and the first character dictionary are sorted in the order of appearance frequency. Further, the surname character dictionary is divided into three grooves A, B, and C.
A group: If the cumulative appearance frequency is included in up to 95% of the total of the appearance frequencies scanned from the head in the surname character dictionary, the group of scanned characters is set as the A group.
B-groove: If the cumulative appearance frequency is included in up to 99% of the total of appearance frequencies scanned from the head in the surname character dictionary, the group of scanned characters is set as the B group.
C groove: A set of all characters in the surname character dictionary is defined as a C groove. That is, if it corresponds to the remaining 1%, it becomes a C group other than A and B.

Similarly, the name character dictionary is divided into three grooves D, E, and F. The judgment values for the last name and the name are represented by M and N, respectively.
If the leading part of the candidate character string is a surname included in the A set, M = SA;
If the leading part of the candidate character string is a surname included in the B set, M = SB;
If the leading part of the candidate character string is a surname included in the C set, M = SC;
If the first part of the candidate character string is not a surname included in the C set, M = 0;
If the last part of the candidate character string is a name character included in the D set, N = SD;
If the last part of the candidate character string is a name character included in the E set, N = SE;
If the last part of the candidate character string is a name character included in the F set, N = SF;
If the last part of the candidate character string is not a name character included in the F set, N = 0;

  If M + N> threshold, the candidate character string is determined to be a person name. SA, SB, SC, SD, SE, and SF are constants. There is a relationship of SA> SB> SC, SD> SE> SF.

  Next, an organization name extraction method will be described. Here, the following criteria can be used by using a publicly-known name extraction technique or a person name extraction technique.

Length measurement. If the length of the input character string is 4 characters or less, it is determined that the title candidate is not an organization name, and the process ends.
Check if the text of the title candidate sentence includes the organization name. FIG. 12 is a diagram showing an example of a Chinese organization name dictionary. If the title candidate includes the organization name of the dictionary, the judgment value + A is set.
It is determined whether the character string of the title candidate sentence is the general name of the organization. If it is a generic name, the determination value is + B. This is also done by collating with the organization name dictionary.
It is checked whether the character string of the title candidate sentence includes the organization name keyword. If it is included, the judgment value is + C. This collates whether or not an organization name keyword (for example, “university”) is included in a specific position (for example, the end of a sentence) of a character string.
As described above, if the determination value> the threshold value is satisfied, the character string of the title candidate sentence is determined as the organization name. A, B, and C are constants.

  FIG. 13 shows a Chinese 2-gram title keyword dictionary and a 2-gram title prohibited keyword dictionary. When extracting the title keyword information 44 and the title prohibition keyword information 46 (see FIG. 5) of the feature amount of the title candidate sentence, the two-gram title keyword dictionary and the two-gram title prohibition keyword dictionary are referred to. For example, “main disease” shown at the beginning of a title-prohibited keyword means “this disease” in Japanese, and it is determined that such a character string is not used in the document title.

  Next, an example in which a title candidate sentence is extracted from a Chinese sample document will be described. FIG. 14A shows a title candidate sentence extracted from a Chinese sample document (the meaning of Japanese is described in parentheses). FIG. 14B shows the feature quantities for these title candidate sentences. In the figure, the number before “:” is the dimension number of the feature quantity (the nine elements of the feature quantity shown in FIG. 5, where 1 is the length of the candidate sentence, 2 is the rank information of similarity, 3 Indicates author information, 4 indicates organization name information, 5 indicates title keyword information, 6 indicates author position information, 7 indicates no-title keyword information, 8 indicates a zip code, and 9 indicates the number of symbols). The numbers indicate the corresponding dimension values. The document after “#” is the corresponding title candidate sentence.

  For example, in the second title candidate sentence from the top, the ranking of the similarity degree in the second dimension is “1”, that is, the degree of similarity by the 2-gram vector with the other title candidate sentences in the document is the highest. The title keyword information of the fifth dimension is “1”, indicating that the title keyword is included.

  The feature amount of the title candidate sentence obtained in this way is classified by SVM. FIG. 15 shows the results of classifying the feature values of FIG. 14B by SVM, and the data surrounded by the broken line in the first column represents the distance to the primary classification boundary surface. As can be seen from the above results, the one closest to the primary classification boundary surface is “Initial exploration of construction work / quantity work management” (initial examination of track work management in track maintenance area), and this candidate sentence is used as the document title. Extracted.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific embodiments according to the present invention, and various modifications can be made within the scope of the gist of the present invention described in the claims. Deformation / change is possible.

  The document title extraction apparatus according to the present invention can be used for extracting document titles in various languages as a document information extraction method using language knowledge. Furthermore, it is possible to digitize a paper document in real time as if it were a copy, and to automatically index the paper document without depending on the layout, image information, and content range of the paper document.

It is a figure which shows the hardware constitutions which perform the document title extraction apparatus which concerns on the Example of this invention. It is a figure which shows the functional block of the document title extraction apparatus which concerns on a present Example. It is a figure explaining the operation | movement flow of a title candidate sentence extraction part. It is a figure which shows the example which extracted the title candidate sentence from the Japanese input document. It is a figure explaining the feature-value extracted by the candidate sentence similarity feature-value extraction part. It is a figure explaining how to obtain a feature amount. It is a figure which shows an example of a Japanese surname dictionary. It is a figure which shows the calculation flow of 2 gram vector feature-value. It is a figure explaining the calculation method of 2 gram frequency # '(x). It is a figure which shows the functional block of the document title extraction apparatus which concerns on 2nd Example of this invention. It is a figure which shows the Chinese surname and the name dictionary. It is a figure which shows the dictionary of the organization name of China. It is a figure which shows the Chinese 2-gram title keyword dictionary and the 2-gram title prohibition keyword dictionary. FIG. 14A shows a title candidate sentence of the Chinese sample document, and FIG. 14B shows a feature amount of the title candidate sentence. It is a figure which shows the result of having classified the feature-value of the title candidate sentence shown in FIG. 14 by SVM.

Explanation of symbols

10: Document title extraction device 12: Input device 14: Display device 16: Main storage device 18: Storage device 20: CPU
30: Document input unit 32: Title candidate sentence extraction unit 34: Feature amount extraction unit 36: Title determination unit 38: Extraction result output unit 60 Image document input unit 62: Layout and image information extraction unit 64: Text document extraction unit

Claims (25)

Title candidate sentence extraction means for extracting a plurality of title candidate sentences from a document;
Feature quantity extracting means for extracting the feature quantity of each of the extracted plurality of title candidate sentences;
Determination means for determining a document title from a plurality of title candidate sentences based on the extracted feature amount;
Output means for outputting the determination result,
The document title extracting apparatus, wherein the feature amount includes at least similarity information that is a function of similarity between a title candidate sentence and a plurality of sentences in the document. The document title extraction apparatus according to claim 1, wherein the similarity information includes rank information indicating a degree of similarity between a title candidate sentence and a plurality of sentences in the document. The said similarity information is calculated using the vector information of the sub character string selected from the title candidate sentence, and the vector information of the sub character string selected from the sentence in the document. Document title extraction device. The vector information is calculated based on an appearance frequency of N (N is a natural number greater than or equal to 2) grams selected from title candidate sentences and an appearance frequency of N grams selected from sentences in the document. 3. The document title extraction device according to any one of 3. 5. The document title extraction according to claim 4, wherein when vector information is calculated based on the appearance frequency of the N-gram, the vector information is corrected when a predetermined use-prohibited N-gram is included. apparatus. The document title extraction apparatus according to claim 1, wherein the similarity information is calculated based on an edit distance between a title candidate sentence and a sentence in the document. The document title extraction apparatus according to claim 1, wherein the similarity information is calculated based on a length of a title candidate sentence and a maximum common character string of sentences in the document. 2. The document title extracting apparatus according to claim 1, wherein when the title candidate sentence includes a predetermined title keyword, the feature amount includes title keyword information indicating a position and an appearance frequency of the keyword. When the title candidate sentence includes a predetermined use prohibition title keyword, the feature amount includes use prohibition title keyword information indicating a position and an appearance frequency of the use prohibition title keyword. Item 2. The document title extraction device according to Item 1. The document title extraction device according to claim 1, wherein the determination unit classifies the feature amount of each title candidate sentence by SVM and extracts an optimal title candidate sentence from the classification result. The document title extraction apparatus according to claim 1, wherein the output unit outputs the determined title sentence and related information. The document title extraction apparatus further includes an input unit for inputting an image document, and a text document extraction unit for extracting a text document from the input image document. The title candidate sentence extraction unit is a title candidate from the extracted text document. The document title extraction apparatus according to claim 1, wherein a sentence is extracted. 13. The document title extracting apparatus according to claim 12, wherein the title candidate sentence extracting unit extracts a title candidate sentence in a certain candidate target range from the beginning of the text document. The document title extracting apparatus according to claim 13, further comprising means for extracting layout information from the image document, wherein the feature amount includes the extracted layout information. A method for extracting a title from a document,
Extracting a plurality of title candidate sentences from the document;
Extracting a feature amount including similarity information, which is a function of similarity between a title candidate sentence and a plurality of sentences in a document, for all title candidate sentences;
Determining a document title from a plurality of title candidate sentences based on the extracted feature values;
A document title extracting method including a step of outputting a determination result. The document title extraction method according to claim 15, wherein the similarity information includes rank information indicating a degree of similarity between a title candidate sentence and a plurality of sentences in the document. The similarity information is calculated using vector information of a sub character string selected from a title candidate sentence and vector information of a sub character string selected from a sentence in a document, according to claim 15 or 16. Document title extraction method. 16. The vector information is calculated based on an appearance frequency of N (N is a natural number greater than or equal to 2) grams selected from title candidate sentences and an appearance frequency of N grams selected from sentences in a document. The document title extraction method according to any one of 17. 19. The document title extraction according to claim 18, wherein when vector information is calculated based on the appearance frequency of the N-gram, the vector information is corrected when a predetermined N-gram for use prohibition is included. Method. The document title extraction method according to claim 15, wherein the similarity information is calculated from an edit distance between a title candidate sentence and a sentence in the document. The document title extraction method according to claim 15, wherein the similarity information is calculated based on a length of a title candidate sentence and a maximum common character string of sentences in the document. The document title extraction method according to claim 15, wherein the feature amount includes title keyword information indicating a position and an appearance frequency of a keyword when a predetermined title keyword is included in the title candidate sentence. When the title candidate sentence includes a predetermined use prohibition title keyword, the feature amount includes use prohibition title keyword information indicating a position and an appearance frequency of the use prohibition title keyword. Item 15. The document title extraction method according to Item 15. A program for extracting a title from a document,
Extracting a plurality of title candidate sentences from the document;
Extracting a feature amount including similarity information, which is a function of similarity between a title candidate sentence and a plurality of sentences in a document, for all title candidate sentences;
Determining a document title from a plurality of title candidate sentences based on the extracted feature values;
A document title extraction program including a step of outputting a determination result. The similarity information is calculated using an appearance frequency of N (N is a natural number greater than or equal to 2) grams selected from title candidate sentences and an appearance frequency of N grams selected from sentences in a document. 24. Document title extraction program according to 24.

Images