JPH11259515A - Similar document retrieval device and method and recording medium recording similar document retrieval program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve both inter-document similarity calculation accuracy and similar document retrieval accuracy via the optimization of a list of unnecessary words by deciding some of extracted words as unnecessary words, deleting the unnecessary words from a retrieval key document and a retrieval object document and calculating the similarity between both documents. SOLUTION: Some of words extracted by a word extraction means are decided as unnecessary words based on the occurrence frequency of each designated unnecessary word. Then the unnecessary words are deleted from a retrieval key document and a retrieval object document, and the similarity is calculated between both documents. An unnecessary word deletion part 28 of this similar document retrieval device deletes the words equivalent to the unnecessary words stored in an unnecessary word buffer 45 from a retrieval keyword information storing buffer 47 and a retrieval object word information storing buffer 42. A similarity calculation part 29 calculates the similarity between the retrieval key document and the retrieval object document based on the information which are stored in the buffer 47, the buffer 42 and a common word information storing buffer 48.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a similar document search apparatus, a similar document search method, and a similar document search program for searching a document database for document data based on the similarity between documents. It relates to a recording medium.

[0002]

2. Description of the Related Art In recent years, a large amount of digitized document data has been distributed, and similar to a document specified in a document database (hereinafter referred to as a retrieval key document) for the purpose of automatic classification and the like. Systems for automatically searching documents have been put to practical use. In this document search system, a word included in a search key document is compared with a word included in a search target document (hereinafter, referred to as a search target document), and a type of a common word, an appearance location, The similarity is calculated from the number of appearances by the vector space method, and a search target document having a high similarity is output as a search result.

At this time, calculation of similarity including unnecessary words (common words that do not characterize the contents of a document) in performing similar document search may cause a decrease in search accuracy. A method is adopted in which a word list is created, and when extracting words from a document, the unnecessary word list is referred to so that words corresponding to unnecessary words are not extracted from the document.

[0004] However, in general, when creating an unnecessary word list, it is necessary for the user to determine the types of words to be unnecessary words one by one, and a separate unnecessary word list is required for each type of document database to be searched. It is necessary to prepare. The work of creating such an unnecessary word list not only imposes a heavy burden on the user, but also has a problem that the accuracy of the similar document search greatly varies due to individual differences in the selection of the unnecessary word.

[0005]

As described above, in order to perform a highly accurate similar document search, it is preferable to exclude unnecessary words from word targets to be extracted from the document. It is necessary to manually create an unnecessary word list for each type, and this imposes a burden on the user. Also, of course, unnecessary words are not selected,
There is a problem that the individual difference in the selection of the unnecessary word by the user is strongly reflected in the search result, and the accuracy of the similar document search is likely to vary.

SUMMARY OF THE INVENTION The present invention has been made to solve such a problem. An optimum unnecessary word list can be automatically created, and the similarity calculation between documents can be improved and the similarity can be improved by optimizing the unnecessary word list. It is an object of the present invention to provide a similar document search device, a similar document search method, and a recording medium on which a program for searching for a similar document is recorded, which can improve the document search accuracy.

[0007]

According to a first aspect of the present invention, there is provided a similar document search apparatus according to the present invention, wherein a certain document is used as a search key document. In a similar document search apparatus for searching for a document to be searched from a plurality of search target documents, a document data storage unit storing a plurality of document data including the search key document and the search target document, and an arbitrary word are designated. Word designation means, word extraction means for extracting a word from each document data stored in the document data storage means, and arbitrary words designated by the word designation means and words extracted by the word extraction means An appearance frequency calculation means for calculating the appearance frequency in each document data, and an appearance frequency of the arbitrary word calculated by the appearance frequency calculation means as a reference And an unnecessary word determining unit that determines at least some of the words extracted by the word extracting unit as unnecessary words, and an unnecessary word determining unit that determines the unnecessary words from the search key document and the search target document. Means for calculating the similarity between the two documents by removing the unnecessary words.

In the present invention, word extraction means is selected based on an appearance frequency calculated for a word arbitrarily designated by a user as a representative of unnecessary words from a group of words extracted from a plurality of document data. A similar document search is performed by determining at least some of the words extracted by the above as unnecessary words and calculating the similarity between the two documents by removing the unnecessary words from the search key document and the search target document, respectively. .

For example, as described in claim 2, of the words extracted by the word extracting means, words whose appearance frequency calculated is higher than the appearance frequency calculated for the arbitrary word are regarded as unnecessary words. In the determination, or as described in claim 3, of the words extracted by the word extraction means, the calculated appearance frequency is equal to or higher than the minimum appearance frequency of the appearance frequencies of a plurality of arbitrary words. The word is determined as an unnecessary word. Further, as described in claim 4,
6. A method according to claim 5, wherein, out of the words extracted by the word extracting means, a predetermined number of words are determined as unnecessary words in preference to those having a higher calculated appearance frequency. Are determined as unnecessary words, the words whose calculated appearance frequency is equal to or higher than an arbitrary appearance frequency specified in advance among the words extracted by.

According to the above-mentioned invention, extraction of unnecessary words from a word group included in document data can be automated. That is, for example, the user can input one or several arbitrary words corresponding to representative unnecessary words, input the number of arbitrary unnecessary words, or input the frequency of appearance of the reference, and can obtain the desired one. A list of unnecessary words close to the above can be obtained, the overall efficiency of similar document search can be improved, and the optimum and appropriate unnecessary words can be quickly extracted without omission for each type of search target document database. Therefore, it is possible to improve and stabilize the accuracy of the similar document search.

[0011] Further, by determining a word having a frequency equal to or higher than the minimum frequency among the frequencies of occurrence of a plurality of arbitrary words as an unnecessary word, the degree of individual differences for each user in the difference in the unnecessary word list is reduced. Also from this point, it is possible to improve and stabilize the accuracy of the similar document search.

[0012]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing a hardware configuration of a similar document search apparatus according to an embodiment of the present invention. As shown in FIG. 1, a similar document search device according to the present embodiment includes a control device 1 including a CPU, a memory, and the like, an input device 2 such as a keyboard, and a display device 3 that displays a process and a result of similar document search. , And an external storage device 4 for storing document data and various data necessary for similar document search.

FIG. 2 is a functional block diagram showing the configuration of the control device 1 of the similar document search device of the present embodiment. As shown in FIG. 1, the control device 1 includes a control unit and a memory unit.

The control unit comprises a main processing unit 11, an initialization unit 1
2, input unit 13, output unit 14, search target document reading unit 1
5. Search target document word extraction unit 16, search target word appearance frequency calculation unit 17, search target word information calculation unit 18, all search target word statistics calculation unit 19, unnecessary word setting unit 20, unnecessary word list creation unit 21, unnecessary Word list reading unit 22, search key document input unit 23, search key word extraction unit 24, search key word appearance frequency calculation unit 25, search target word information reading unit 26, common word extraction unit 27, unnecessary word removal unit 28, similar Degree calculator 2
9, a search result output unit 30 and the like.

The memory section includes a search target document storage buffer 4.
1. Search target word information storage buffer 42, all search target word information storage buffer 43, unnecessary word setting buffer 44, unnecessary word buffer 45, search key document storage buffer 46, search key word information storage buffer 47, common word information storage buffer 48, a similarity storage buffer 49, a search result output buffer 50, a work buffer 51, and the like.

The initialization unit 12 includes buffers 41, 42,
.., 51 are initialized. The input unit 13 inputs data input by a user through the input device 2 to the control unit. The output unit 14 outputs output data of the control unit to the display device 3.

The search target document reading unit 15 reads the search target document specified by the user from the external storage device 4 and stores the read search target document in the search target document storage buffer 41.

The search target document word extraction unit 16 extracts words from the document data stored in the search target document storage buffer 41, extracts words characterizing the contents of the extracted document from the extracted word group, and extracts the extracted words. The searched word type is stored in the search target word information storage buffer 42. Here, word extraction is performed by morphological analysis or the like, and extraction of words characterizing the contents of the document is performed by selecting words such as “noun” or “sa-variant noun” based on the word class information of the word. Will be

The search target word appearance frequency calculation unit 17 calculates the appearance frequency (the number of appearances) of the individual words stored in the search target word information storage buffer 42 in the extraction source document, and calculates the calculated occurrence frequency. It is stored in the search target word information storage buffer 42 in association with the word.

The search target word information writing unit 18 reads out the word information and the appearance frequency information of each search target document stored in the search target word information storage buffer 42 and writes them to the external storage device 4.

The all-search-target-word statistics calculating unit 19 sequentially reads out the word appearance frequency information of each search target document stored in the external storage device 4 and writes the information in the search target document storage buffer 41. For each type of word extracted from the document, the statistical value of the appearance frequency, for example, the number of appearing documents, is calculated, and the result is stored in the all-search-target-word information storage buffer 43 as the all-search target word statistics.

The unnecessary word setting unit 20 allows a user to set a reference for setting a type of word (unnecessary word) to be excluded from a group of words extracted from each document data in calculating similarity between documents. Upon reception, the set reference is stored in the unnecessary word setting buffer 44. At this time, there are a method of setting one or more arbitrary words (unnecessary words), a method of specifying the number of unnecessary words, and a method of specifying a reference value of an appearance frequency.

The unnecessary word list creating section 21 creates and creates an unnecessary word list based on all the search target word information stored in the all search target word information storage buffer 43 and the setting contents in the unnecessary word setting buffer 44. The unnecessary word list is stored in the external storage device 4.

In the creation of the unnecessary word list, if any one of the unnecessary words is designated by the unnecessary word setting unit 20,
Words having a frequency equal to or higher than the appearance frequency calculated for the unnecessary word are stored in the unnecessary word buffer 45 as unnecessary words, and when a plurality of unnecessary words are specified, the minimum appearance frequency of the frequency calculated for the unnecessary word is used. The above words are stored in the unnecessary word buffer 45 as unnecessary words. Unnecessary word setting unit 2
When the number of unnecessary words is designated by 0, the words having the specified unnecessary words are stored in the unnecessary word buffer 45 as unnecessary words with priority given to those having a high appearance frequency among all the words. When the reference value of the appearance frequency is specified by the unnecessary word setting unit 20, words that are equal to or more than the specified reference value of the appearance frequency among all the words are stored in the unnecessary word buffer 45 as unnecessary words.

The unnecessary word list reading section 22 reads the unnecessary word list stored in the external storage device 4 and stores it in the unnecessary word buffer 45.

The search key document input section 23 stores the search key document input from the input device 2 in the search key document storage buffer 46.

The search key word extracting section 24 cuts out a word from the search key document stored in the search key document storage buffer 46, and selects a word characterizing the contents of the search key document from the cut out word group. The seed is extracted, and the extracted word type is stored in the search key word information storage buffer 47. Here, the extraction of words is performed by morphological analysis or the like, and the extraction of words characterizing the contents of the document is performed based on the part of speech information of the words, for example, “noun” or “sa-variable noun”.
This is done by selecting a word.

The search key word appearance frequency calculation unit 25 calculates the appearance frequency (appearance number) in the extraction source document for each word extracted by the search key word extraction unit 24,
The calculated appearance frequency is stored in the search key word information storage buffer 4
7 is stored.

The search target word information reading unit 26 retrieves the word information of each search target document in the document database stored in the external storage device 4 and the information of its appearance frequency for each document, and stores the search target word information. The data is stored in the buffer 42.

The common word extraction unit 27 reads word information commonly present in the search key document and the search target document and information on the frequency of occurrence thereof from the search key word information storage buffer 47 and the search target word information storage buffer 42, It is stored in the common word information storage buffer 48.

The unnecessary word removing unit 28 includes a search key word information storage buffer 47 and a search target word information storage buffer 4.
From 2, the word corresponding to the unnecessary word stored in the unnecessary word buffer 45 is deleted.

Based on the information stored in the search key word information storage buffer 47, the search target word information storage buffer 42, and the common word information storage buffer 48, the similarity calculation unit 29 compares the search key document with the search key document by a vector space method or the like. The similarity with the search target document is calculated, and the calculated similarity is stored in the similarity storage buffer 49.

The search result output unit 30 calculates the similarity of each search target document stored in the similarity storage buffer 49 from
Document information (for example, a document ID) serving as a similar search result is stored in the search result output buffer 50, and the contents of the search result output buffer 50 are output to the display device 3 through the output unit 14.

Next, the operation of the similar document search apparatus according to this embodiment will be described.

First, the operation of creating a document database and an unnecessary word list will be described with reference to FIGS.

First, the initialization unit 12 is started, and all buffers are initialized (step 301). Subsequently, the unnecessary word setting unit 20 is activated, and a reference for setting the unnecessary word is set (step 302). The criterion for setting the unnecessary word is set by a method arbitrarily selected by the user from the following three methods.

In the first method, the user specifies an arbitrary number of words (unnecessary words), and calculates the appearance frequency calculated for the specified words (or the appearance frequency calculated for each unnecessary word when there are a plurality of specified words). Among the words extracted from the document data, all words having a calculated appearance frequency equal to or higher than the reference value are regarded as unnecessary words. For example, as shown in FIG. 4, a general word having no property to characterize a document, for example, "koto"
"Apparatus" or the like is designated, and the minimum appearance frequency among the appearance frequencies calculated for these words is used as a reference value, and words whose appearance frequency is equal to or higher than this reference value are used as unnecessary words.

In the second method, the user arbitrarily designates the number of unnecessary words (or the order of appearance frequency values), and, of words extracted from document data, gives priority to words having a higher calculated appearance frequency. In addition, there is a method of making the specified number of words unnecessary words. For example, as shown in FIG.
2 (the number of unnecessary words = 2) ", the words whose calculated appearance frequency is in the top two are regarded as unnecessary words.

In a third method, the user arbitrarily specifies a reference value of the appearance frequency, and the calculated appearance frequency of the word group extracted from the document data is equal to or higher than the specified reference value of the appearance frequency. This is a method of making all words unnecessary words. For example, as shown in FIG. 6, when “designated appearance frequency = 500 or more” is designated, all words whose appearance frequency is 500 or more are regarded as unnecessary words.

The unnecessary word setting criterion set by any of these three methods is stored in the unnecessary word setting buffer 44.

Next, the retrieval target document reading section 15 is started. The search target document reading unit 15 determines whether there is a search target document which has not been processed yet in the external storage device 4 (step 303), and if there is a search target document, as shown in FIG. Search target document storage buffer 4 for target document
1 (step 304).

Next, the search target document word extraction unit 16 is activated. The search target document word extraction unit 16 cuts out a word from the search target document stored in the search target document storage buffer 41 by morphological analysis or the like, and extracts the content of the document such as “noun” or “sa noun” from the cut out word group. Are extracted, and the extracted words are stored in the search target word information storage buffer 42 (step 305).

Subsequently, the search target word appearance frequency calculation unit 17
Is started. The search target word appearance frequency calculation unit 17 calculates the appearance frequency of each word stored in the search target word information storage buffer 42 in the extraction source document, for example, as shown in FIG. The information on the appearance frequency thus obtained is stored in the search target word information storage buffer 42 in association with the word. Hereinafter, the information on the word and the appearance frequency is referred to as “word information”. In FIG. 8, “frequency 2” described in correspondence with the word “document” indicates that the word “document” appears twice in the extraction source document.

Next, the search target word information writing unit 18 is started, and the contents (word information) of the search target word information storage buffer 42 are stored in the external storage device 4 (step 30).
6). Thereafter, the process returns to step 303 and the external storage device 4
Then, the next search target document stored in the search target document is read, and a word is extracted from the search target document and the appearance frequency is calculated. As described above, the extraction of the words and the calculation of the appearance frequency are performed for all the search target documents stored in the external storage device 4, and the results are stored in the external storage device 4.

When the word information of all the search target documents stored in the external storage device 4 is stored in the external storage device 4,
Next, the all search target word statistics calculation unit 19 is activated. The all-search-target-word-statistics calculating unit 19 sequentially reads out word information (information of appearance frequency) of all the search target documents stored in the external storage device 4 and stores the word information in the search target document storage buffer 41.
Based on the word information (information on the appearance frequency) of all the search target documents stored in the search target document storage buffer 41, the statistical value of the appearance frequency of each word (for example, the number of appearing documents) is calculated. I do. Then, as shown in FIG. 9, the statistical values calculated in this manner, such as the number of appearing documents of the appearance frequency of each word, are stored in the all search target word information storage buffer 43 as all search target word information (step 3).
07). Next, the unnecessary word list creation unit 21 is activated. The unnecessary word list creation unit 21 creates an unnecessary word list based on all the search target word information stored in the all search target word information storage buffer 43 and the unnecessary word setting criteria stored in the unnecessary word setting buffer 44, The created unnecessary word list is stored in the external storage device 4 (step 308).
The generation of the unnecessary word list is performed based on the selection criteria of the unnecessary word arbitrarily specified by the user.

When an arbitrary number of words (unnecessary words) are specified by the user in the unnecessary word setting unit 20 (in the case of the first method), the unnecessary word list creating unit 21 determines the frequency of occurrence of the unnecessary words. It is read from the all search target word information storage buffer 43, and all words whose appearance frequency is equal to or more than a reference value are extracted from the all search target word information storage buffer 43 based on the minimum occurrence frequency among the read unnecessary word appearance frequencies. Then, this is stored in the unnecessary word buffer 45 as an unnecessary word. FIG.
9 shows an example of the unnecessary word stored in the unnecessary word buffer 45.

When the number of unnecessary words (or the order of appearance frequency values) is designated by the user in the unnecessary word setting unit 20 (in the case of the second method), the unnecessary word list creating unit 21
Among all the search target word information, words having a high frequency of appearance and up to a designated number (designated order) are determined as unnecessary words and stored in the unnecessary word buffer 45.

Further, when the reference value of the appearance frequency is specified by the user in the unnecessary word setting section 20 (in the case of the third method), the unnecessary word list creating section 21 sets all unnecessary words whose appearance frequency is higher than the reference value. The word is determined as an unnecessary word and stored in the unnecessary word buffer 45.

Thus, the creation of the document database and the unnecessary word list is completed.

Next, the operation of similar document retrieval will be described with reference to FIG.
This will be described with reference to FIGS. First, the initialization unit 12
Is started, and all buffers are initialized (step 401). Next, the unnecessary word list reading unit 22 is activated,
The unnecessary word list is read from the external storage device 4 and stored in the unnecessary word buffer 45 (step 402).

Next, when the search key document input section 23 is activated, the search key document specified by the user is read from the external storage device 4 and the read search key document is stored in the search key document storage buffer 46. It is stored (step 403). FIG. 12 shows an example of the search key document stored in the search key document storage buffer 46.

Subsequently, the search key word extraction unit 24 is activated. The search key word extraction unit 24 cuts out a word from the search key document stored in the search key document storage buffer 46 by morphological analysis or the like, and extracts the contents of the document such as “noun” or “sa variable noun” from the cut out word group. Characteristic words are extracted, and the extracted words are stored in the search key word information storage buffer 47 (step 404).

Next, the unnecessary word removing unit 28 is activated. The unnecessary word removing unit 28 includes a search key word information storage buffer 47.
A word that matches the unnecessary word stored in the unnecessary word buffer 45 is found from the word group of the search key document stored in the search key document and deleted (step 405).

Subsequently, the search key word appearance frequency calculation unit 25
Is started. The search key word appearance frequency calculation unit 25 calculates the appearance frequency of each word stored in the search key word information storage buffer 47 in the extraction source document, and uses the calculated appearance frequency information as a figure. As shown in FIG. 13, it is stored in the search key word information storage buffer 47 in association with the word (step 406).

Next, the retrieval target document reading section 15 is activated. The search target document reading unit 15 determines whether there is a search target document that has not been processed yet in the external storage device 4 (step 407). If there is a search target document, the search target document is searched. The data is stored in the storage buffer 41.
Thereafter, the search target document word extraction unit 16 cuts out words from the search target document stored in the search target document storage buffer 41 by morphological analysis or the like, and selects “noun” or “ Words that characterize the contents of the document, such as "sa noun," are extracted, and information on the extracted words is stored in the search target word information storage buffer 42 (step 408).

Subsequently, the unnecessary word removing section 28 is activated.
The unnecessary word removing unit 28 stores the search target word information storage buffer 4.
A word that matches the unnecessary word stored in the unnecessary word buffer 45 is found out of the word group of the search target document stored in 2 and is deleted (step 409).

Next, the common word extraction unit 27 is activated.
The common word extraction unit 27 detects words commonly stored in the search target word information storage buffer 42 and the search key word information storage buffer 47 from which the unnecessary words have been deleted, and as shown in FIG. The detected word is stored in the common word information storage buffer 48 (step 41).
0).

Thereafter, the similarity calculating section 29 is started.
The similarity calculation unit 29 stores the search target word information storage buffer 4
2. Based on the contents of the search key word information storage buffer 47 and the common word information storage buffer 48, the similarity between the search key document and the search target document is calculated by the vector space method or the like, and the calculated similarity is calculated. It is stored in the degree storage buffer 49 (step 411). FIG. 15 shows an example of similarity information between the search key document stored in the similarity storage buffer 49 and each search target document.

Thereafter, returning to step 407, if there is a search target document in the external storage device 4 that has not been processed yet, the same processing as described above is performed on the search target document, and the search key document thus calculated Is stored in the similarity storage buffer 49.

After all the similarities between the search target documents and the search key documents stored in the external storage device 4 are stored in the similarity storage buffer 49, the search result output unit 30 is activated. The search result output unit 30 arranges the IDs of documents to be searched in the order of the highest similarity, as shown in FIG. 16, for example, from the contents of the similarity storage buffer 49, and stores the result in the search result output buffer 50. . Thereafter, the output unit 14
Thus, the contents of the search result output buffer 50 are output to the display device 3 (step 412).

Thus, according to the similar document search apparatus of the present embodiment, the work load of the user when creating the unnecessary word list is greatly reduced, and overall similar document search efficiency can be improved. That is, in the similar document search device of the present embodiment, the user operation required to create the unnecessary word list includes one or a small number of unnecessary words,
Alternatively, either the designation of the number of unnecessary words or the reference value of the appearance frequency may be performed, and only such a simple designation operation is performed in advance by the user, and the optimum and appropriate unnecessary words can be listed without omission, The accuracy of similar document search can be improved and stabilized.

In this embodiment, a case will be described in which the appearance frequency of a word specified by the user as a representative of the unnecessary word is set as a reference value, and all words having an appearance frequency equal to or higher than the reference value are set as unnecessary words. However, a certain margin may be secured on the lower side than this reference value, and a word having an appearance frequency within this margin may be determined as an unnecessary word.

[0064]

As described above, according to the present invention, it is not necessary for the user to register each word to be an unnecessary word in the creation of the unnecessary word list. By simply specifying the unnecessary words, the number of unnecessary words, and inputting the reference value of the appearance frequency, a desired unnecessary word list can be created. As a result, the overall efficiency of the similar document search can be improved, and the optimum and appropriate unnecessary words can be extracted without omission for each type of the search target document database, thereby improving the accuracy of the similar document search. And stabilization.

[Brief description of the drawings]

FIG. 1 is a diagram showing a hardware configuration of a similar document search device according to an embodiment of the present invention.

FIG. 2 is a functional block diagram showing a configuration of a control device of the similar document search device of FIG. 1;

FIG. 3 is a flowchart showing a procedure for creating a document database and an unnecessary word list.

FIG. 4 is a diagram showing an example of an unnecessary word specified by a user.

FIG. 5 is a diagram showing an example of the number of unnecessary words specified by the user (order of appearance frequency).

FIG. 6 is a diagram illustrating an example of a reference value of an appearance frequency specified by a user;

FIG. 7 shows an example of a search target document.

FIG. 8 is a diagram showing an example of words stored in a search target word information storage buffer and appearance frequencies;

FIG. 9 is a diagram illustrating an example of words of all documents to be searched and statistical values of their appearance frequencies stored in a buffer for storing all words to be searched.

FIG. 10 is a diagram illustrating an example of unnecessary words stored in an unnecessary word buffer.

FIG. 11 is a flowchart showing a procedure of a similar document search operation;

FIG. 12 illustrates an example of a search key document.

FIG. 13 is a diagram showing an example of words stored in a search key word information storage buffer and appearance frequencies;

FIG. 14 is a diagram showing an example of common words stored in a common word information storage buffer and appearance frequencies;

FIG. 15 is a diagram showing an example of the similarity between the search key document stored in the similarity storage buffer and the search target document.

FIG. 16 is a diagram illustrating an example of a similar document search result.

[Explanation of symbols]

 200: Main processing unit 201: Initializing unit 202: Input unit 203: Output unit 204: Search target document reading unit 205: Search target document word extraction unit 206: Search target word appearance frequency calculation unit 207: Search target word information calculation unit 208: Search target word Statistical calculation unit 209 Unnecessary word setting unit 210 Unnecessary word list creation unit 211 Unnecessary word list reading unit 212 Search key document input unit 213: Search key word extraction unit 214: Search key word appearance frequency calculation unit 215: Search target word information reading unit 216: Common word extraction unit 217: Unnecessary word removing unit 218: Similarity score calculating unit 219 ...... search result output unit

Continued on the front page (72) Inventor Yukio Nakamoto 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd. (72) Inventor Takuya Nishina 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering (72) Inventor Naohide Kubota 1381-1, Shinmachi, Ome-shi, Tokyo Toshiba Computer Engineering Co., Ltd.

Claims (7)

[Claims] 1. A similar document search apparatus for searching a document similar to a search key document from a plurality of search target documents using a certain document as a search key document, wherein a plurality of search target documents including the search key document and the search target document are included. Document data storage means in which document data is stored; word specification means for specifying an arbitrary word; word extraction means for extracting a word from each document data stored in the document data storage means; An appearance frequency calculation unit that calculates an appearance frequency of each of the specified arbitrary word and the word extracted by the word extraction unit in each of the document data; and an occurrence frequency of the arbitrary word calculated by the appearance frequency calculation unit. It is unnecessary to determine at least some of the words extracted by the word extracting means as unnecessary words based on the frequency of appearance. Determination means; and means for calculating a similarity between the two documents by removing the unnecessary words determined by the unnecessary word determination means from the search key document and the search target document, respectively. Document search device. 2. A similar document search apparatus for searching a document similar to a search key document from a plurality of search target documents using a certain document as a search key document, comprising: a plurality of search target documents including the search key document and the search target document. Document data storage means in which document data is stored; word specification means for specifying an arbitrary word; word extraction means for extracting a word from each document data stored in the document data storage means; An appearance frequency calculation means for calculating an appearance frequency of each of the specified arbitrary word and the word extracted by the word extraction means in each of the document data; and, among the words extracted by the word extraction means, A word whose appearance frequency calculated by the frequency calculation means is equal to or higher than the appearance frequency calculated for the arbitrary word is determined as an unnecessary word. Key word determination means, and means for calculating a similarity between the two documents by removing the unnecessary words determined by the unnecessary word determination means from the search key document and the search target document, respectively. Similar document search device. 3. A similar document search apparatus for searching a document similar to a search key document from a plurality of search target documents using a certain document as a search key document, comprising: a plurality of search target documents including the search key document and the search target document. Document data storing means for storing document data; word specifying means for specifying a plurality of arbitrary words; word extracting means for extracting words from each document data stored in the document data storing means; A plurality of arbitrary words specified by a means and an appearance frequency calculating means for calculating an appearance frequency of each of the words extracted by the word extracting means in each of the document data; The appearance frequency calculated by the appearance frequency calculation means is the appearance of a plurality of arbitrary words calculated by the appearance frequency calculation means. Unnecessary word determining means for determining a word having a frequency equal to or higher than the minimum frequency of occurrence as an unnecessary word; and removing the unnecessary words determined by the unnecessary word determining means from the search key document and the search target document. Means for calculating the similarity of the similar document search device. 4. A similar document search apparatus for searching a document similar to a search key document from a plurality of search target documents using a certain document as a search key document, comprising: a plurality of search target documents including the search key document and the search target document. Document data storing means for storing document data; word extracting means for extracting words from each document data stored in the document data storing means; and unnecessary words among words extracted by the word extracting means. Unnecessary word number specifying means for arbitrarily specifying the number of words; appearance frequency calculating means for calculating the appearance frequency of each word extracted by the word extracting means in each of the document data; extracted by the word extracting means Of the words, words having a higher appearance frequency calculated by the appearance frequency calculation means are given priority, and the number of words designated by the unnecessary word number designation means is given priority. Unnecessary word determining means for determining as a word; and means for calculating a similarity between the two documents by removing the unnecessary words determined by the unnecessary word determining means from the search key document and the search target document. A similar document search device characterized by the following. 5. A similar document search apparatus for searching a document similar to a search key document from a plurality of search target documents using a certain document as a search key document, comprising: a plurality of search target documents including the search key document and the search target document. Document data storage means in which document data is stored; word extraction means for extracting a word from each document data stored in the document data storage means; in each of the document data of words extracted by the word extraction means An appearance frequency calculating means for calculating an appearance frequency of, an appearance frequency designating means for designating an arbitrary appearance frequency, and, among words extracted by the word extracting means, an appearance frequency calculated by the appearance frequency calculating means, Unnecessary word determining means for determining a word having an arbitrary appearance frequency or more designated by the appearance frequency designating means as an unnecessary word; And a means for calculating the degree of similarity between the respective documents by excluding the unnecessary words determined by the unnecessary word determining means from the search target document. 6. A similar document search method for searching a document similar to the search key document from a plurality of search target documents using a certain document as a search key document, wherein an arbitrary word is specified; And extracting words from a plurality of document data including the search target document; calculating the appearance frequency of each of the designated arbitrary words and the extracted words in each of the document data; Determining at least a part of the extracted words as unnecessary words based on the calculated appearance frequency of the arbitrary word; and determining the words from the search key document and the search target document. Calculating a similarity between the two documents by removing unnecessary words from each other. 7. A word designation for designating an arbitrary word in a recording medium in which a program for retrieving a document similar to the search key document from a plurality of search target documents using a certain document as a search key document is recorded. Means, word extraction means for extracting a word from a plurality of document data including the search key document and the search target document, and an arbitrary word designated by the word designation means and a word extracted by the word extraction means An appearance frequency calculation unit for calculating the appearance frequency in each of the document data; and a word extracted by the word extraction unit based on the appearance frequency of the arbitrary word calculated by the appearance frequency calculation unit. Unnecessary word determining means for determining at least a part of the words as unnecessary words; and Recording medium comprising a program and a means for calculating a similarity between the determined unnecessary words both documents except each is recorded by a main character discriminating unit.