JP2005099972A - Concept search method and system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a conceptual search method which improves search accuracy compared with a conventional conceptual search by performing a more accurate search based on information necessary for a user, since in a conceptual search, the search is carried out without deciding whether the information included in documents is necessary information for a user, there are cases that search accuracy may deteriorate, if the conceptual search is performed for huge documents, and that although decision of necessity and propriety of information included in the document is made by a feature term, the discrimination may be difficult. <P>SOLUTION: A keyword which becomes a feature of a document is extracted from the document is inputted as a search condition. The keyword used as the feature and individual sentences obtained by breaking down a document are classified into an equivalence class, matched and held. A sentence representing a sentence group in each classified equivalence class is shown as an important sentence, and is made to be choosen. The conceptual search processing is carried out by the use of a keyword group matched with the equivalence class shown by the important sentence and its weight. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a concept search in which an electronic document is searched with a search condition registered by the user and a document satisfying the condition is returned to the user. In particular, by scanning the contents of the electronic document, the user can set the search condition as The present invention relates to a document search method and system having an inter-document fitness level calculation function for calculating a fitness level for a registered document.

  In recent years, a large amount of digitized documents (hereinafter referred to as text) has been delivered to users from time to time by e-mail, electronic news, and the like. In addition, information sources that transmit information using the WWW (World Wide Web) are rapidly increasing, and a large amount of text is collected from these information sources using an information collecting robot or the like. For this reason, there is an increasing need to search for texts including information that the user really wants from among these texts.

  In the conventional search system, a search is performed by assembling a search expression for a word that the user thinks is necessary for the search according to a certain syntax and inputting it. However, it is difficult for a user unfamiliar with searching to enter a suitable word to obtain the desired information, or to extract only necessary information and assemble a complicated search expression to shake out unnecessary information. is there. For this reason, Patent Document 1 proposes a technique (hereinafter referred to as a concept search) in which a document including information desired by a user (hereinafter referred to as a seed document) is input instead of assembling a search expression. . In this technique, a word necessary for a search (hereinafter referred to as a feature term) is automatically extracted from a seed document, an appropriate weight is assigned to the extracted feature term, and the degree of matching of a search result sentence is calculated. A search result having a fitness value greater than a certain value is used.

  However, when the seed document is a long document, not only the document information desired by the user but also a seed document including an unnecessary concept may be input. In this case, since there is an error with the information desired by the user, the user is not satisfied with the search result, so he tries to perform a search again with some adjustments. In general, a search result document is referred to, and a document (or sentence) including the concept desired by the user is found more appropriately than the seed document. By repeating this operation, it is possible to approach the search result desired by the user. Alternatively, with respect to the weight for the feature term, the search is performed again by increasing the weight of the feature term considered necessary for the user or lowering the weight of the feature term that is not necessary.

Hereinafter, other related literature techniques related to the present invention will be described.
Non-Patent Document 1 touches on a technique for performing clustering (automatic classification) by comparing the number of appearances of keywords in each classification item and the number of appearances of keywords in each document and classifying them into the closest classification item.

  Non-Patent Document 2 mentions a technique for extracting an important sentence in a document based on keyword weights.

JP-A-11-143902

Automatic document clustering system gnmz (http://icrouton.as.wakwak.ne.jp/pub/kks/cnamazu.html) "Technology for automatically summarizing texts-Part 1-Extracting important sentences in texts", computer science magazine bit February, Kyoritsu Shuppan, pp. 37-42, 2000.2

  In the concept search, the search is executed without determining whether the information contained in the document is necessary information or unnecessary information for the user. Therefore, when performing a concept search using a long document, only the conventional method is used. Then, the search accuracy may decrease. In order to prevent this search accuracy degradation, the following two points are problems.

1) Means for classifying information contained in a document 2) Means for allowing a user to accurately determine whether the information is necessary In the prior art, as means for 2), there are selection of feature terms and weight adjustment. It may be difficult to determine whether or not the information is desired by the user only with feature terms lacking peripheral information.

  An object of the present invention is to provide a concept search method in which search accuracy is improved as compared with a conventional concept search by executing a more accurate search based on information necessary for a user.

  In order to achieve the above-mentioned object, the process consisting of the following steps automatically presents candidates for seed documents (sentences) suitable for the search without the user knowing the optimum feature terms for the search. By selecting an appropriate sentence, the above problem can be solved and the search accuracy can be improved. The processing procedure will be described below.

  Step 1: Extract feature terms and their weights from the input seed document.

  Step 2: The input seed document is decomposed into sentences.

  Step 3: Treat each sentence decomposed in Step 2 as one document, classify each sentence into equivalence classes using the feature terms extracted in Step 1 and their weights (hereinafter referred to as clustering), and equivalence classes Is determined from the feature terms extracted in step 1.

  Step 4: Select a sentence (hereinafter referred to as an important sentence) that is most suitable as a feature of the equivalence class among the equivalence classes classified in Step 3 above.

  Step 5: Present the important sentence extracted in Step 4 to the user, and ask the user to select a sentence including necessary information.

  Step 6: A concept search is executed using the feature terms representing the equivalence classes classified in Step 3 and their weights corresponding to the sentence selected in Step 5 above.

  In executing the concept search, the concept search is executed using only information necessary for the user, so that a more accurate search can be executed. As a result, the search accuracy is improved as compared with the conventional concept search.

  FIG. 1 shows the system configuration of the present invention. The concept search device 30000 is a client server type search system that communicates with the client 10000 via the network 20000. When performing a search, the user inputs a seed document 40000 from the client 10000. The client 10000 transmits the input seed document 40000 to the concept search device 30000 through the network 20000, and executes the processing of the present invention.

  The concept search device 30000 has the following configuration. A feature term extraction unit 31000 that extracts a feature term and its weight from the seed document 40000 using the document information DB 38000, and a similarity that calculates the similarity between the document in the document DB 37000 and the seed document 40000 using the feature term and its weight Degree calculation unit 32000, sentence decomposition unit 33000 that decomposes seed document 40000 into sentences, clustering unit 34000 that classifies each sentence in seed document 40000 into equivalence classes with similar contents using feature terms and their weights, clustering unit 34000 An important sentence extraction unit 35000, a similarity calculation unit 32000, and an important sentence extraction unit that output an important sentence with respect to a search condition (a feature term extracted by the feature term extraction unit and its weight), using sentences classified as equivalent classes by Screen data that receives 35000 results and sends to client 10000 Consisting screen data generating unit 36000 to generate. The concept search procedure of the present invention will be described below according to the processing flow of FIG.

  Step 60000: The user inputs the document 41000 having the contents illustrated in FIG. 3 into the seed document input BOX 11000 from the client screen shown in FIG. After the input, when the user presses the search start button 12000, the client 10000 transmits the input data as a seed document 40000 to the concept search device 30000 via the network 20000.

  Step 61000: The concept retrieval apparatus 30000 receives the seed document 40000 transmitted from the client 10000 and inputs it to the feature term extraction unit 31000. The feature term extraction unit 31000 uses the document information DB 38000 to extract the feature term 42000 and its weight 43000 shown in FIG. Here, as an example of the feature term extraction algorithm, the method of Patent Literature 1 can be used. In this case, the document information DB 38000 is assumed to store necessary data shown in Patent Literature 1. As another processing method, morphological analysis may be used for feature term extraction, and the number of appearances of feature terms in the seed document may be used for weight calculation.

  Step 62000: Next, the concept search device 30000 inputs the seed document 40000 to the sentence decomposition unit 33000, and obtains a sentence group 44000 in the seed document 40000 shown in FIG. The sentence decomposition unit 33000 operates according to the processing flow shown in FIG. When processing relatively free-form sentences such as business daily reports, the sentence length tends to be irregular. Therefore, the sentence disassembly unit 33000 combines several short sentences into one sentence or has a long document. You may adjust so that it may cut out at the break of the word which does not exceed fixed length.

  Step 63000: The feature term 42000 extracted by the above step, its weight 43000, and the sentence group 44000 are input to the clustering unit 34000, and each sentence is classified into equivalence classes, and the feature term group 45000 representing the classification shown in FIG. And the classification sentence group 46000 is obtained. Although there are various document clustering methods, the method of Non-Patent Document 1 can be used here. The similarity of each sentence may be calculated by using the term appearance frequency, the product of the term appearance frequency and the term uniqueness, or the product of the term appearance frequency and the number of types of terms. FIG. 7 shows the clustering result when the number of classifications is 3. Necessary information is acquired from the document information DB 38000. The feature term group 45000 representing the classified sentence group 46000 is assumed to be all included in each sentence group among the feature terms 42000 extracted in the above step. Further, the target sentence for executing clustering is only a sentence including the feature term shown in FIG.

  Step 64000: Next, an important sentence of each classification group is determined. The important sentence is determined by the important sentence extraction unit 35000 using the similarity calculation unit 32000. Each sentence in the classification is input to the similarity calculation unit 32000 using the feature terms 45000 and the weight obtained from the table of FIG. Here, data is acquired from the document information DB 38000 as necessary. The similarity calculation unit 32000 may obtain the similarity of each sentence in accordance with Patent Document 1 or may use the method of Non-Patent Document 2. However, in this case, it is assumed that the document information DB 38000 stores data necessary for Non-Patent Document 2. FIG. 9 shows a sentence having the highest similarity among the obtained similarities 47000 as the important sentence 48000 of the classification group.

  Step 65000: When the obtained important sentence 48000 is input to the output screen data generation unit 36000, the concept search device 30000 transmits necessary output data to the client 10000 via the network 20000, and the client 10000 outputs to the search result screen 13000. (FIG. 10).

  Step 66000: The search result screen 13000 displays the important sentence 48000 of each classification group and the number of sentences classified into the classification group as the score 15000 of the classification group. The higher the score, the more the concept of the classification group is included in the seed document 40000. Further, by using the detailed browsing button 18000, all documents in the classification group shown in FIG. 8 can be browsed. The user checks a necessary concept according to a screen instruction using a check box 17000, and when a search execution button 14000 is pressed, a selection number 49000 is transmitted. In FIG. 10, “2” is selected, and the client 10000 transmits “2”, which is the selection number 49000, to the concept search device 30000.

  Step 67000: After receiving the selection number 49000, the concept search device 30000 obtains a feature term group 45000 corresponding to the selection number 49000 and its weight 43000 from the table shown in FIG. 9 from the table shown in FIG.

  The feature term 45000 and its weight 43000 are input to the similarity calculation unit 32000, and the similarity to each document stored in the document DB 37000 is obtained. As an example of the similarity calculation method, Patent Document 1 is used here.

  Step 68000: After calculating the similarity, the document title is input to the output screen data generation unit 36000 in descending order of similarity. The output screen data generation unit 36000 transmits the output data to the client 10000.

  Step 69000: The client 10000 displays the search result screen shown in FIG. When the search condition storage button 19000 is pressed, the search condition is stored in the client using the selected sentence as a key. Thereby, the user can save the trouble of assigning an appropriate storage name, and whether or not the user can use it can be determined based on the contents of the text even when the user uses it later.

  Further, the sentence decomposition process shown in FIG. 12 will be described.

  Step 50000: The text operation unit 33100 reads the processed document into the work storage area 33200. The operation initial character position 33300 and the operation initial position (first character of the document) of the operation character position 33400 are set to 0 in the sentence number 33500, and the process proceeds to Step 50010.

  Step 50010: The text operation unit 33100 determines whether or not the character at the character position stored in the operation position storage area 33400 among the processed documents read into the work storage area 33200 is a delimiter (.). If it is a delimiter, the process of step 50020 is performed. If it is not a delimiter, the process proceeds to step 50060.

  Step 50020: The characters from the operation start character position 33300 to the operation character position 33400 are copied from the work storage area 33200 to the sentence group storage area 33600, and the process proceeds to Step 50030.

  Step 50030: The sentence number 33500 is incremented by 1, and the process proceeds to Step 50040.

  Step 50040: The operation character position 33400 is incremented by 1, and the process proceeds to Step 50050.

  Step 50050: The value of the operation character position 33400 is set in the operation start character position 33300, and the process proceeds to Step 50060.

  Step 50060: The value of the operation character position 33400 is incremented by 1, and the process proceeds to Step 50070.

  Step 50070: If the value of the operation character position 33400 is smaller than the number of characters of the processed document, the process proceeds to Step 50010. If the value of operation character position 33400 is equal to the number of characters of the processed document, the process proceeds to step 50080.

  Step 50080: The characters from the operation start character position 33300 to the operation character position 33400 are copied to the sentence group storage area 33600. The number of sentences 33500 is incremented by 1, each sentence in the sentence group storage area is output, and the process is terminated.

1 shows a system configuration of the present invention. The client screen in the Example of this invention is shown. The seed document input in the Example of this invention is shown. The feature terms and their weights are shown. The sentence decomposition part in the Example of this invention is shown. Indicates the sentence group output by the sentence decomposition unit. The classification sentence group in the Example of this invention is shown. The classification sentence group in the Example of this invention and the importance of each sentence are shown. Each classification and the important sentence in the Example of this invention are shown. The concept selection screen in the Example of this invention is shown. The search result screen in the Example of this invention is shown. The sentence decomposition part process flow in the Example of this invention is shown. The flow of the process in the Example of this invention is shown.

Explanation of symbols

10000: client, 20000: network, 30000: concept search device,
31000: Feature term extraction unit, 32000: Similarity calculation unit,
33000: sentence decomposition unit, 34000: clustering unit, 35000: important sentence extraction unit,
36000: output screen data generation unit, 37000: document DB, 38000: document information DB, 40000: seed document

Claims (7)

In a concept search method for searching for similar documents by inputting a document as a search condition,
Extracting a keyword that is characteristic of the document from the document input as the search condition;
Break the document into individual sentences,
Classify the keyword as the feature and the decomposed individual sentence into equivalence classes, hold them in correspondence,
The sentence representative of the group of sentences in each classified equivalence class is an important sentence, the important sentence is presented instead of the equivalence class, and the user is selected,
A concept search method comprising performing a concept search process using a keyword group associated with an equivalence class indicated by the important sentence and a weight of the keyword group.   When disassembling the document into sentences, to make the length of each sentence as uniform as possible, two sentences are regarded as one sentence when it is a short sentence, and word separation is used when a sentence is too long. 2. The concept retrieval method according to claim 1, wherein the concept is extracted as one sentence in the middle of the sentence in consideration of the above.   In calculating similarity when classifying each sentence into equivalence classes, the frequency of keyword occurrence in each sentence, the product of the keyword appearance frequency and its uniqueness, or the product of the keyword appearance frequency and the number of keyword types The concept retrieval method according to claim 1, wherein either of them is used.   2. The concept retrieval method according to claim 1, wherein when an important sentence is selected instead of the equivalence class, the number of sentences classified into the equivalence class is presented next to the important sentence.   2. The concept retrieval method according to claim 1, wherein when the user selects an important sentence instead of the equivalence class, each sentence in the equivalence class is browsed.   2. The concept retrieval method according to claim 1, wherein when a keyword group representing the equivalence class indicated by the important sentence and the weight are left as a search profile, the important sentence is stored as a key. A concept search system that searches for similar documents by entering documents as search criteria.
Means for extracting a keyword that is a feature of the document from the document input as the search condition;
Means for decomposing the document into individual sentences;
Means for classifying the keyword as the feature and the decomposed individual sentences into equivalent classes and associating them with each other;
Means that presents the important sentence instead of the equivalence class and makes the user select the sentence representative of the group of sentences in each classified equivalence class as an important sentence;
A concept search system comprising: a keyword group associated with an equivalence class indicated by the important sentence; and means for performing a concept search process using a weight of the keyword group.

Images