JP2012104051A - Document index creating device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a document index which corresponds to described contents of a document file as much as possible.SOLUTION: In the network system S which includes a document database 10, a search engine 20 for searching the document database 10 using a predetermined search algorithm, a terminal device 30 used by a user, and a data processing device 40 for creating a document index for document search, the data processing device 40 divides document data of one document file into a plurality of document paragraphs, discriminates the paragraphs into a first unit sentence (paragraph) including an attention word A and a second unit sentence (paragraph) including no attention work A, evaluates a degree of relevance between the first unit sentence and the second unit sentence, obtains a paragraph weight value of the second unit sentence, and sets the weight value of the attention word A by adding the paragraph weight value to a basis weight value based on the appearance frequency of the attention word A.

Description

  The present invention relates to a document index creation apparatus used when a document search process is performed on a document database that stores a plurality of document files.

  For example, on a global communication network such as the Internet or a local communication network such as an in-house network in a company, a search process is executed by inputting a query from a terminal to a document database storing a large number of document files. A search system for extracting a desired document file is widely used. In such a search system, a search index created by analyzing a query input from a terminal, and a document index created in advance for each of the stored document files, using a predetermined search algorithm A technique is adopted in which document files with high similarity are extracted as hit documents by collation.

  The document index is usually composed of words. In order to improve the search accuracy, the specific word is given high weighting, and a document file including such a specific word is devised so that it is extracted preferentially (as a high ranking ranking) in the search. As one of the weighting methods, there is a method of assigning a high weight to a word that appears frequently in a document file. In addition, a so-called page in which, as the number of links (bookmarks) from other document files increases, it is regarded as a document file having a higher importance level, and all words included in such document files are given higher weights. A rank method (see Patent Document 1) is also known.

USP 6,285,999 gazette

  However, in the method of simply giving a high weight value to words that appear frequently in a document file, there is a fear that a document index that does not match the original contents of the document file may be created. For example, when a certain “word A” appears only once in the description in the document file X, if only the viewpoint of appearance frequency is considered, it appears only once, so “word A” A low weight value will be given. However, in the description of the document file X, although “Word A” is not used repeatedly, there are many cases in which “Word A” includes very closely related contents. It can be said that such a document file X is highly likely to be a useful document file for a user who sets “word A” as a keyword. However, since the document file X is given a low weight value for “word A”, the document file X does not appear at the top of the ranking of the search results, and the user overlooks this.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a document index creation device that can create a document index that matches the description contents of a document file as much as possible.

  A document index creation device according to an aspect of the present invention that achieves the above object includes a document database that stores a plurality of document files, and an extraction unit that extracts one document file to be created as a document index from the document database. A document separation unit that divides the document data of the one document file into a plurality of unit sentences based on an identifier indicating a sentence break included in the document data, and performs document analysis processing on the document data. Document analysis means for extracting a word group, one attention word from the word group is specified, the plurality of unit sentences, a first unit sentence in which the attention word appears, and a first word in which the attention word does not appear Selecting means for selecting two unit sentences; evaluation means for evaluating a sentence relevance between the first unit sentences and the second unit sentences; and the document analyzing means. Creating means for creating a document index by giving a weight value to each of the extracted word groups, the creating means using the appearance frequency of the attention word and the evaluation value of the sentence relevance An apparatus for setting the weight value for a word of interest (claim 1).

  According to this configuration, the word weight value is not set simply by the appearance frequency of the attention word in one document file. That is, the sentence relevance between the first unit sentence including the attention word and the second unit sentence not including the attention word is evaluated, and the document index is created by adding the evaluation value to the appearance frequency of the attention word. The high degree of sentence relevance means that even if the attention word is not directly used in the second unit sentence, there is a high possibility that the second unit sentence includes a description related to the attention word. Can be estimated. In this case, even if the appearance frequency of the attention word is low, on the basis of the high possibility that the related description exists in the second unit sentence not including the attention word, by giving a relatively high weight value to the attention word, A document index corresponding to the actual description content can be created for one document file.

  In the above-described configuration, the evaluation unit calculates the sentence relevance level based on the similarity between the appearance frequency of the word group included in the first unit sentence and the appearance frequency of the word group included in the second unit sentence. It is desirable to evaluate (Claim 2).

  According to this configuration, it is possible to easily determine the sentence relevance between the first unit sentence and the second unit sentence.

  In this case, the evaluation unit obtains the word density of the attention word based on the ratio of the appearance frequency of the attention word to the appearance frequency of all words included in the first unit sentence as the evaluation value of the sentence relevance level, The creation means multiplies the sentence relevance by the word density to obtain a weight value of the second unit sentence, and sets the weight value of the second unit sentence to a basic weight value based on the appearance frequency of the attention word. It is desirable to add and set a weight value for the attention word.

  According to this configuration, the higher the word density of the attention word in the first unit sentence, the higher the weight value of the second unit sentence. This is based on the presumption that the more frequently the attention word appears in the first unit sentence, the more likely the description related to the attention word is made in the second unit sentence. Therefore, it is possible to further create a document index according to the description content of the document file.

  In the above configuration, it is desirable that the sentence separating unit divides the sentence into paragraphs based on an identifier indicating a line break and / or an identifier indicating the presence of a blank line.

  According to this configuration, it is possible to divide a sentence in units of paragraphs, which are normal divisions of sentences. Therefore, it is possible to accurately compare unit sentences.

  Further, it is desirable to further include normalizing means for normalizing a weight value for the attention word based on the number of characters constituting the document data of the one document file.

  According to this configuration, the weight value can be set by correcting the influence of the document data.

  According to the present invention, it is possible to provide a document index creation device that creates a document index corresponding to the description content of a document file in a document search system for a document database. As a result, it is possible to improve the search accuracy and improve the efficiency of various processing operations using the document search system.

It is a block diagram which shows roughly the hardware constitutions of the network system with which the document index creation apparatus concerning this invention is applied. It is a typical flowchart which shows the outline | summary of the search system using the said network system. It is a figure which shows the general example of a document search. It is a schematic diagram which shows the concept of word weight provision in embodiment of this invention. It is a functional block diagram which shows the function structure of a network system. It is a figure which shows the technique of word weight provision in embodiment of this invention, Comprising: (a) is a schematic diagram of a document file, (b) is a figure of the table format for demonstrating the calculation process of a paragraph weight. It is a flowchart which shows the creation process of a document index. It is a flowchart which shows the creation process of a document index.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram schematically showing a hardware configuration of a network system S to which a document index creation apparatus according to the present invention is applied. The network system S includes a document database 10 that can be used on a communication network, a search engine 20 that performs a search process on the document database 10 using a predetermined search algorithm, a terminal device 30 that a user uses, A data processing apparatus 40 that mainly performs processing for creating a document index for document search is connected to be able to perform data communication via the Internet IN or the local network LN.

  The document database 10 is a database in which a large number of document files to be searched are stored. When the document database 10 is connected to the Internet IN, the document database 10 is a set of a large number of websites each having a domain name. As such a document database 10, for example, a database accessible through a portal site such as Yahoo (registered trademark) or Google (registered trademark) can be exemplified. When the document database 10 is connected to a local net LN built inside a company or the like, the document database 10 is a database that stores shared document files in the company.

  The search engine 20 is equipped with a search algorithm having predetermined parameters, and performs a search process for a document file corresponding to a given query with respect to the document database 10. That is, the search engine 20 analyzes a query to create a search index, and extracts a keyword or attribute (metadata) for each search target document file stored in the document database 10. Read the document index. Further, the search engine 20 performs a collation process between the document index and the search index using the search algorithm, and extracts a document file having a high degree of matching (matching a query).

  The terminal device 30 is a communication terminal 31, 32, 33, 34... Such as a personal computer, a mobile phone, or a personal digital assistant owned by many users. The terminal device 30 can access the search engine 20 and the document database 10 via the Internet IN or the local net LN. For example, the terminal 31 has a keyboard 311 for inputting keywords and sentences (queries) as search processing conditions, a display 312 for displaying a search screen, a list of hit documents, document contents, and the like, and a display on the display 312. A mouse 313 for inputting a command (such as a document selection instruction or a selection end instruction) to the screen is provided.

  Each user gives a query to the search engine 20 via the keyboard 311 of the respective communication terminal 31, 32, 33, 34..., And obtains a list of document files hit by the search processing using the query. . Further, the user can select a desired document file from the list and display the contents of the document file on his / her display 312.

  The data processing device 40 performs document analysis processing for each search target document file stored in the document database 10, extracts keywords and attributes, and creates a search document index. The document index includes a word extracted from each document file and a weight value assigned to each word. The weight value represents the importance of each word in each document file. When a word having a high value as a weight value in the document file becomes a search query, the document file is searched for a hit ranking. Will be ranked higher. Normally, the weight value is set based on the appearance frequency of a word. However, in the present embodiment, the relation between a paragraph that includes a word (keyword) that can be a search query and a paragraph that does not include the word is considered. It is characterized in that the weight value is set. This will be described in detail later.

  FIG. 2 is a schematic flowchart showing an outline of the document file search process executed by the network system S. When a query is given from the terminal device 30 as processing on the search source side, the search engine 20 performs document analysis processing including, for example, morphological analysis on the query document (step S1). Next, a search index is created based on the words obtained by the document analysis process (step S2). This search index mainly consists of a group of keywords that are closely related to the question event. Then, using the search index, a document search process is performed on the document database 10 including a large amount of data (step S3).

  As processing on the search destination side, the data processing device 40 periodically performs document analysis for extracting metadata (step S01) and creation of a document index (step S02). The present embodiment is characterized by this document index creation step. The results obtained in the search process in step S3 are output as a list (ranking display) after ranking according to the degree of match in the search algorithm to the terminal device 30 of the user who has input the query (step S4). .

  FIG. 3 is a schematic diagram showing an example of a conventional general document search. When the keyword “A” is given as a search query from the terminal device 30, the search processing is performed on the document database 10 by the search engine. Then, a document file including the word A is extracted as a hit document, and a document file having a particularly high appearance frequency of the word A is displayed in a higher ranking on the terminal device 30. Here, the document file X in which the word A appears five times is shown.

  However, there are many cases where the original content of a document file cannot be measured only by the appearance frequency of words. For example, although “word A” appears five times in the description in the document file X, the description related to “word A” is actually shallow as a whole document, or conversely, “word A” is 1 Even a document file that appears only once may contain content that is very closely related to “word A” as a whole document. In such a case, the document file useful for the user is obviously the latter, but the appearance frequency of “word A” is low, so the latter document file is not ranked high, and the user overlooks as a result. Can happen. Therefore, simply giving a high weight value to words that occur frequently in a document file may create a document index that does not match the original content of the document file. Can do.

  FIG. 4 is a schematic diagram showing the concept of weighting a word in the present embodiment. In this embodiment, instead of simply relying on the appearance frequency of the word A (word of interest), a paragraph including the word A (first unit sentence) and a paragraph in which the word A does not appear adjacent to the paragraph (second) Evaluate the text relevance to the unit text. Then, the paragraph weight value corresponding to the sentence relevance level is added to the original weight value of word A (which depends on the word appearance frequency).

  It can be said that the paragraph including the keyword (for example, word A) given by the user from the terminal device 30 is a paragraph including the description desired by the user. On the other hand, it cannot be said that the description of a paragraph that does not include the word A is completely unrelated to the topic expected by the user. Rather, in the paragraph adjacent to the paragraph including the word A, it can be said that there is a high possibility that a description related to the topic related to the word A exists even if the word A is not used. In this case, the information value of the document file is high for the user. Therefore, if a paragraph that does not include word A but has a high degree of text relevance exists close to the preceding paragraph that includes word A, a corresponding weight value is added to word A, and the document index Let's create. Thereby, when the word A is given as a search query, the document file is ranked higher, and the probability that the user overlooks the document file can be suppressed.

  FIG. 5 is a functional block diagram showing a functional configuration of the network system S. As shown in FIG. The document database 10 includes a plurality of document servers 11, 12, 13... (Web server or in-house local server) in which various document files are stored. The search engine 20 functionally includes a document index storage unit 21, a search index creation unit 22, a search processing unit 23, and a ranking display processing unit 24. As described above, the terminal device 30 includes the query input unit 301, the display unit 302, and the operation unit 303. The data processing device 40 (document index creation device) includes a document extraction unit 41, a document analysis unit 42 (document analysis unit), a document index creation unit 43 (creation unit), and a storage unit 40M.

  The document index storage unit 21 stores a document index for searching for document files included in the document database 10. The document index is created by the data processing device 40 at a predetermined creation timing (for example, once a day), and the stored content is updated.

  The search index creation unit 22 performs document analysis on a query (keyword or question document) given from the terminal device 30 and creates a search index. For example, when the query is a question document, a search index is created by extracting the words as independent words in the document and weighting them by referring to the appearance frequency of these words.

  The search processing unit 23 performs a document file search process on the document database 10 using a search algorithm. Specifically, the search index and the document index are collated, and a document file having a high similarity to the search index (query) is extracted. Here, a preset search algorithm is used for this search process. Examples of the search algorithm include a cosine scale, Dice coefficient, Jaccard coefficient, T score, mutual information amount, and Simpson coefficient.

  The ranking display processing unit 24 creates a list in which a plurality of document files hit by the search processing by the search processing unit 23 are ranked in descending order of similarity (matching degree) to the query. This list can be browsed by the terminal device 30 and is actually displayed on the display unit 302 after the search process is completed.

  The query input unit 301 of the terminal device 30 is a part that receives an input of a query such as a keyword for search processing from a user, and is, for example, the keyboard 311 shown in FIG.

  The display unit 302 is, for example, the display 312 shown in FIG. 1, and includes a browsing screen (query input screen) of the search engine 20, a list of hit document files and their excerpts, and a document file selected by the user from the list. Display the contents.

  The operation unit 303 is, for example, the mouse 313 illustrated in FIG. 1 and gives an operation instruction to the selection unit, the link unit, the task bar, and the like on the image displayed on the display unit 302. The user gives a selection instruction to select one of the hit document file lists displayed on the display unit 302 to display the contents and a selection end instruction to stop the display via the operation unit 303.

  The data processing device 40 extracts a document file from the document database 10 in order to create a document index for searching for a document file included in the document database 10, analyzes a sentence described in the document file, and analyzes a word And set a weight value for each word.

  The document extraction unit 41 performs a process of extracting a plurality of document files from the document database 10 at predetermined intervals as a trigger for creating a document index.

  The document analysis unit 42 performs document analysis processing such as normalization processing, document structure analysis processing, synonym processing on the document data included in each document file extracted by the document extraction unit 41, and converts the document data into word units. Divide into The normalization process is a process for deleting characters and symbols unnecessary for analysis from the analysis target document and unifying full-width and half-width characters so that the document structure analysis can be performed normally. In the document structure analysis process, the classification target documents after the normalization process are each subjected to a morphological analysis to divide the document into words, and a parsing process to perform dependency between words (nouns and verbs). And the like. The synonym process is a process in which terms having different expressions are treated as a single word by utilizing a thesaurus (synonym dictionary) that absorbs fluctuations of synonyms and notations. By such document analysis processing, independent words are derived.

  The document index creation unit 43 creates a basic index (an index to which no weight is given yet) as a basis for the document index for each document file, based on the words extracted by the document analysis unit 42. Then, the words listed in the basic index are weighted using the word appearance frequency and the paragraph weight value described with reference to FIG. 4 to complete the document index.

  The storage unit 40M is a memory used as a work area for temporarily storing various work data and the like in the data processing device 40. The storage unit 40M temporarily stores, for example, a word list for each document file extracted by the document analysis unit 42, sentence relevance data calculated by the document index creation unit 43, word density data, paragraph weight data, word frequency data, and the like. To remember. Various set values and arithmetic expressions are also stored in the storage unit 40M.

  The document index creation unit 43 is functionally configured to execute the above processing. The document separation unit 44 (sentence separation unit), the selection unit 45 (selection unit), the relevance calculation unit 46 (evaluation unit), and the word density calculation unit 47, a paragraph weight calculation unit 48, and a weight setting unit 49.

  The sentence separation unit 44 performs a process of dividing the document data of one document file into a plurality of unit sentences based on an identifier indicating a sentence break included in the document data. In this embodiment, the unit sentence is defined as one paragraph separated by blank lines. For this reason, the sentence separation unit 44 performs a process of dividing the document data into paragraph-by-paragraph sentences based on an identifier indicating a line break included in the document data and an identifier indicating the presence of a blank line following the identifier. Of course, a blank line may not be requested, and the sentence may be separated based on the presence of an identifier indicating a line break. Alternatively, the sentence may be separated by relying only on a tag indicating a blank line or the like without detecting an identifier indicating a line feed.

  The selection unit 45 designates one word of interest from the word group extracted by the document analysis unit 42 and stored in the storage unit 40M, and is a paragraph unit sentence (unit sentence) divided by the sentence separation unit 44. Is selected into a first unit sentence in which the attention word appears and a second unit sentence in which the attention word does not appear.

  FIG. 6A is a schematic diagram for explaining processing by the sentence separation unit 44 and the selection unit 45. Here, an example is shown in which a sentence described in the document file X is divided into seven paragraphs of paragraphs 1 to 7 by the sentence separation unit 44. Then, the selection unit 45 sets “word A” as the attention word, paragraphs 1 and 5 including this “word A” are set as the first unit sentence, and paragraph 2 does not include “word A”. An example in which Paragraph 3, Paragraph 4, Paragraph 6 and Paragraph 7 are classified as second unit sentences is shown.

  The degree-of-association calculation unit 46 evaluates the degree of sentence relevance between the first unit sentence and the second unit sentence. Specifically, in the description order of document data, the first first unit sentence that appears first is used as a reference, and the degree of sentence relevance with one or more second unit sentences that appear subsequently is evaluated. When a new first unit sentence appears, the degree of sentence relevance with one or a plurality of second unit sentences appearing after the first unit sentence is evaluated. The evaluation of the sentence relevance is based on the appearance frequency of the word group included in the paragraph as the first unit sentence (for example, the word A = 3 times, the word B = 1 time, the word C = 5 times, etc. Between the words extracted from the table and the number of occurrences thereof) and the appearance frequency of the word group included in the paragraph as the second unit sentence, and using a vector calculation method such as a cosine scale, the similarity between the two A method of evaluating (text relevance) numerically can be adopted.

  In the example of FIG. 6A, the relevance calculation unit 46 calculates the sentence relevance between paragraph 1 (first unit sentence) and subsequent paragraphs 2, 3, and 4 (second unit sentences). , Respectively, using the above-described vector calculation method or the like. On the other hand, since the following paragraph 5 is a paragraph that becomes a first unit sentence including “word A”, the relevance calculation unit 46 thereafter performs paragraph 5 (first unit sentence), and subsequent paragraphs 6 and 7. The sentence similarity with (second unit sentence) is calculated. In the “relevance” column of FIG. 6B, the numerical value of the relevance obtained by the relevance calculation unit 46 is displayed.

The word density calculation unit 47 obtains the word density in the first unit sentence of the attention word based on the ratio of the appearance frequency of the attention word (for example, word A) to the appearance frequency of all the words included in the first unit sentence. For example, in Paragraph 1, assuming that a table such as “word A = 3 times, word B = 1 time, word C = 5 times,...” Is obtained, the following formula: word density of word A = word A Appearance frequency (3 times) / Total appearance frequency of words A, B, C... (3 times + 1 time + 5 times +...)
Find the word density at.

  This word density is based on the assumption that there is a high possibility that the description affected by the attention word will follow the paragraph (first unit sentence) where the occurrence frequency of the attention word is high. Is used as a coefficient for evaluating a sentence set with one or a plurality of second unit sentences. In the example of FIG. 6B, an example is shown in which the word density of the word A in the paragraph 1 is 0.05 and the word density of the word A in the paragraph 5 is 0.03.

  The paragraph weight calculator 48 multiplies the sentence relevance for each second unit sentence calculated by the relevance calculator 46 by the word density obtained by the word density calculator 47 to obtain a paragraph weight value for each second unit sentence. calculate. As shown in FIG. 6B, the word density of the word A in the paragraph 1 is set to 0.05, and the sentence relevance levels of the following paragraphs 2, 3, and 4 are set to 0.82, 0.62, and 0.41. By multiplying each, the paragraph weight value of each paragraph is derived. For paragraph 5 and subsequent paragraphs, multiply the word density = 0.03 of word A in paragraph 5 by the sentence relevance = 0.66 and 0.33 of paragraph 6 and paragraph 7 following this, respectively. A weight value is derived.

  The weight setting unit 49 is obtained by the paragraph weight calculation unit 48 with the normal weight value of the attention word (basic weight value; for example, the appearance frequency of the attention word or a value obtained by multiplying the appearance frequency by the reciprocal of the document frequency). The sum of the paragraph weight values for each second unit sentence is added to set the weight value for the attention word. In the example of FIG. 6B, the weight setting unit 49 adds the basic weight value based on the appearance frequency of the word A to the total weight of the paragraph weights of the paragraph 2, the paragraph 3, the paragraph 4, the paragraph 6, and the paragraph 7 = 0. 13 is added to set the weight value for word A. Thereby, the creation of the document index of the word A in the document file X is completed.

  Next, the document index creation processing operation by the data processing apparatus 40 of the present embodiment will be described based on the flowcharts shown in FIGS. The data processing device 40 sets a counter p corresponding to the numbering assigned to each of the document file groups in the document database 10 to be processed to “0” (step S11), and then counters p = p + 1. (Step S12). Then, the p-th document file (the document file to which the first number is assigned for the first time) is extracted by the document extraction unit 41 (step S13).

  Next, the document analysis unit 42 performs document analysis processing on the p-th document file, and extracts a word group from the document data (step S14). Then, the document analysis unit 42 obtains the number of overlapping appearances for the extracted words, and obtains the word frequency of each word (step S15). This word group and its appearance frequency data are stored in the storage unit 40M.

  Subsequently, the sentence separation unit 44 divides the document data of the p-th document file into paragraphs (step S16). Then, a counter q corresponding to the numbering assigned to each of the word groups for document index extracted for the p-th document file is set to “0” (step S17), and q = q + 1. The counter is advanced to (step S18). Thereby, the q-th word is set as the attention word.

  Next, the sorting unit 45 sorts the divided paragraphs into a first unit sentence (paragraph) in which the q-th word appears and a second unit sentence (paragraph) in which the q-th word does not appear ( Step S19). Then, the relevance calculation unit 46 evaluates the text relevance between the first unit sentence and each of the second unit sentences using a vector calculation method or the like (step S20).

  Thereafter, the word density calculation unit 47 obtains the word density in the first unit sentence of the qth word (step S21). Subsequently, the paragraph weight calculation unit 48 multiplies the word density obtained in step S21 by the sentence relevance of each second unit sentence obtained in step S20, thereby obtaining the paragraph weight of each second unit sentence. Calculate (step S22). Finally, the weight setting unit 49 adds the paragraph weight value to the basic weight value based on the appearance frequency of the qth word (step S23), and determines the qth word weight value (step S24). The obtained weight value is stored in the storage unit 40M.

  In step S24, it is desirable that the weight setting unit 49 performs a process of normalizing the weight value of the qth word based on the number of characters constituting the document data of the pth document file. This is in order to suppress the problem that the longer the document data, the greater the number of paragraphs and the larger the added value of the paragraph weight value. For example, a method of dividing the obtained weight value by the number of characters can be given as a simple example of the normalization process.

  Subsequently, it is confirmed whether or not the word group counter q is final (step S25). If it is not final (NO in step S25), the process returns to step S18 and the same weight value calculation processing is performed for the next attention word. Is repeated. On the other hand, if the counter q is final (YES in step S25), it is confirmed whether or not the document file counter p is final (step S26). If the counter p is not final (NO in step S26), the process returns to step S12 and the same processing is repeated for the next document file. On the other hand, if the counter p is final (YES in step S26), the process ends.

  According to the network system S according to the present embodiment described above, the document index corresponding to the description content of the document file can be created in the document search system for the document database 10. As a result, it is possible to improve the search accuracy, increase the probability that the document file required by the user can be provided at the top of the search ranking, and to improve the efficiency of various processing operations using the document search system. Become.

S network system 10 database 20 search engine 21 document index storage unit 22 search index creation unit 23 search processing unit 24 ranking display processing unit 30 terminal device 40 data processing device 41 document extraction unit 42 document analysis unit (document analysis unit)
43 Document index creation unit (creation means)
44 sentence separation part (sentence separation means)
45 Sorting part (sorting means)
46 relevance calculator (evaluation means)
47 Word density calculation unit 48 Paragraph weight calculation unit 49 Weight setting unit

Claims (5)

A document database for storing multiple document files;
Extraction means for extracting one document file as a document index creation target from the document database;
Sentence separation means for dividing the document data of the one document file into a plurality of unit sentences based on an identifier indicating a sentence break included in the document data;
Document analysis means for performing a document analysis process on the document data and extracting a word group;
Selecting means for designating one attention word from the word group, and selecting the plurality of unit sentences into a first unit sentence in which the attention word appears and a second unit sentence in which the attention word does not appear; ,
An evaluation means for evaluating a sentence relevance between the first unit sentence and the second unit sentence;
Creating means for creating a document index by giving a weight value to each of the word groups extracted by the document analysis means,
The document index creation device, wherein the creation unit sets the weight value for the attention word using the appearance frequency of the attention word and the evaluation value of the sentence relevance. The evaluation means evaluates the sentence relevance based on the similarity between the appearance frequency of the word group included in the first unit sentence and the appearance frequency of the word group included in the second unit sentence;
The document index creation device according to claim 1. The evaluation means includes
As the evaluation value of the sentence relevance level, the word density of the attention word is obtained based on the ratio of the appearance frequency of the attention word to the appearance frequency of all words included in the first unit sentence.
The creating means includes
Multiplying the sentence relevance by the word density to obtain a weight value of the second unit sentence,
Adding a weight value of the second unit sentence to a basic weight value based on the appearance frequency of the attention word, and setting a weight value for the attention word;
The document index creation device according to claim 2. The sentence separating means divides the sentence into paragraphs based on an identifier indicating a line break and / or an identifier indicating the presence of a blank line.
The document index creation apparatus according to claim 1. And a normalizing unit for normalizing a weight value for the attention word based on the number of characters constituting the document data of the one document file.
The document index creation device according to any one of claims 1 to 4.

Images