JP4298550B2 - Word extraction method, apparatus, and program - Google Patents

Description

  The present invention relates to a method and apparatus for extracting words from a target document set.

  Currently, a large number of electronic documents in various fields and types continue to be born on the Internet. However, it is impossible for a user to see all recently created documents quantitatively, and it is often difficult even if limited by the field or information source. Therefore, for example, if there is a simple access method to a document including the word from the topic word by sequentially listing words having higher topicality than the past from a set of recently created documents, the user can You can efficiently view documents that contain useful information from time to time. In addition, even in a scene where it is desired to know recent topic words and their meanings, if there is a mechanism as described above, the time and effort of searching for topic words and examples of their use are reduced.

As a conventional technique for extracting words from a document set, a word candidate is obtained from an arbitrary character string in the target document set, a character string statistic of the word candidate is calculated from the target document set and another document set, and Some candidates extract word candidates whose values are equal to or greater than a threshold (see Non-Patent Document 1).
Takayuki Adachi, Seto Yamada, Masaaki Nagata, “Lexical Acquisition Method from a Small Document Collection”, Proc. 274-277, March 2003.

  In this technique, word candidates are narrowed down in advance from an arbitrary partial character string in a target document set, and the likelihood of a word is independently examined. However, due to rough narrowing, there are cases in which an incorrect character string partially including the word remains in the word candidates instead of removing the correct word.

  Therefore, it is conceivable to perform morphological analysis on the word candidate. However, if the word candidate includes a word not included in the dictionary for morphological analysis, the correct word cannot be extracted because the result of the morphological analysis is incorrect.

  Also, with this technique, the topic level of a word is the difference between the appearance probability of a recent document set and the appearance probability of a past document set, and a higher topic level indicates a higher topic level. However, there is a possibility that unimportant words are extracted even if the latest degree is high.

  Moreover, even if it is the latest topic word, there may be a case where a characteristic word which appears biased in a certain document is desired, and there is a case where the latest topic word is desired regardless of the bias. However, the topic level calculation method cannot be changed as appropriate according to the purpose of the user.

  Moreover, what is obtained is a word list arranged in order of topic level, and has means for presenting information useful for the user to understand the meaning of the word and means for easily accessing a document containing the word. Absent.

  An object of the present invention is to provide a word extraction method, apparatus, and program capable of extracting a word that can handle even the latest word with higher accuracy than before and that can obtain a topic word useful for a user. Is to provide.

The word extraction method of the present invention is a word extraction method performed in a word extraction device that extracts topic words in a specified period from a sequentially updated document set,
A document collection step in which the document collection means collects documents updated since the last collection from the document storage source at a specified cycle;
A target document set creation step in which a target document set creation unit creates a document set that is a target of word extraction by collecting documents updated from a collected document within a period specified independently of the period;
A word candidate extraction step in which the word candidate extraction means extracts word candidates except for a character string that appears multiple times in the target document set and that is nested by a character type;
The word dividing unit is a value obtained by multiplying the residual IDF of the character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate output by the word candidate extracting unit. Calculate a word score,
Among the partial character strings, a partial character string in which any one character type of katakana, English letters, Arabic numerals, or Chinese numerals continues is the longest length of the partial character string in which any one kind of character type continues. Correcting the word score of each of the partial character strings so that the word score is higher than the word score of the partial character string in which the continuous length of any one type of character is shorter than the longest length ,
Among the partial character string sequences that become the word candidates when the partial character strings are arranged, the partial character string sequence that maximizes the sum of the word scores of the partial character strings constituting the sequence. A word division step for outputting as a word division result ;
Word word selection means, among the word segmentation result of said word candidates outputted by the word segmentation step, the word segmentation result the word score matches a threshold value or character types and string length condition word scores predetermined A word selection step to extract as
Having.

The word extraction device of the present invention is a word extraction device that extracts topic words in a specified period from a sequentially updated document set,
Document collection means for collecting documents updated since the last collection from the document storage source at a specified cycle;
A target document set creation means for creating a document set as a word extraction target by collecting documents updated within a period specified independently of the cycle from the collected documents;
Word candidate extraction means for extracting word candidates except for a character string that appears multiple times in the target document set and a character string that is nested or divided by a certain character type;
Calculate a word score that is a value obtained by multiplying the residual IDF of the character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate output by the word candidate extraction unit. And
Among the partial character strings, a partial character string in which any one character type of katakana, English letters, Arabic numerals, or Chinese numerals continues is the longest length of the partial character string in which any one kind of character type continues. Correcting the word score of each of the partial character strings so that the word score is higher than the word score of the partial character string in which the continuous length of any one type of character is shorter than the longest length ,
Among the partial character string sequences that become the word candidates when the partial character strings are arranged, the partial character string sequence that maximizes the sum of the word scores of the partial character strings constituting the sequence. A word dividing means for outputting as a word division result ;
Of word segmentation result of the word candidates the word dividing means is outputted, the word selection for extracting the word segmentation result the word score matches a threshold value or character types and string length condition word score predetermined as a word Means,
Having.

1) By performing word segmentation based on a character string statistic, a character string length, and a character type from a document set, it is possible to extract a word that can deal with the latest word with higher accuracy than before.
2) Useful for users by calculating the topic level based on the importance of words in documents in a certain period and previous periods and the latest degree of words calculated from the appearance status of certain periods and earlier periods. You can get a topic word.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a word extraction device according to an embodiment of the present invention. In FIG. 1, a single line arrow indicates a data flow, and a double line arrow indicates a process flow.

  The word extraction device 1 includes a crawler 11, a target document set data creation unit 12, a word extraction unit 13, a topic level calculation unit 14, a display unit 15, a document DB 16, a target document set DB 17, and a topic word DB 18.

  The crawler 11 collects documents that have been updated since the last collection from the Internet 2 at a designated cycle, processes a collected document with a tag such as HTML (Hyper Text Markup Language) into a plain text, and outputs it to the document DB 16. The target document set data creation unit 12 inputs processed documents updated from the document DB 16 within a specified period independently from the specified period from the document DB 16, and collects a plurality of processed documents as a topic word. An extraction target document set is created, a statistic is calculated from the target document set, and the target document set and statistic data are output to the target document set DB 17. A URL (Uniform Resource Locator) and a title are also stored in association with the target document set. The word extraction unit 13 inputs the target document set of the target document set DB 17, divides the word, and outputs the word to the target document set DB 17. The topic level calculation unit 14 inputs the word output from the word extraction unit 13 from the target document set DB 17, and the previous target document set and the current target document set in the target document set DB 17 (the time and period are the previous time and the current time). The recent topic level is calculated by a method specified based on the statistic of each character string, and the topic word list arranged in the topic level order is output to the topic word DB 18. The display unit 15 receives the topic word list output from the topic level calculation unit 14, and further inputs the target document set and statistics data of the target document set DB 17, and the context in the target document set for the topic word. Or the original URL (Uniform Resource Locator) containing the topic word, the title of the document in which the topic word appears, the appearance frequency of the topic word in the document set, the number of documents appearing, the topic level, and a certain period The transition of statistics related to the topic word is shaped into a display format and output as a topic word list and related information.

  FIG. 2 is a block diagram illustrating the configuration of the word extraction unit 13. In FIG. 2, single-line arrows indicate data flows, and double-line arrows indicate process flows.

  The word extraction unit 13 includes a word candidate extraction unit 13A, a word division unit 13B, and a word selection unit 13C. The word candidate extraction unit 13A reads the target document set from the target document set DB 17, and excludes a character string that appears multiple times in the target document set and that is nested or divided by a certain character type (Non-Patent Document 1). (Similar method to word candidate extraction)) Word candidates are extracted and stored in the target document set DB 17. The word dividing unit 13B inputs word candidates from the target document set DB 17, performs word division on the word candidates based on the character string statistics, the character string length, and the character type, and then outputs the word division result to the target document set DB 17. The word selection unit 13C inputs word division results of a plurality of word candidates from the target document set DB 17, performs narrowing by a predetermined word score threshold, character type, and character string length, and outputs the remaining words to the topic word DB 18. .

  Next, the operation of this embodiment will be described with reference to the flowchart of FIG.

  First, the crawler 11 collects documents updated to the difference of the cycle (after the previous collection) from the Internet 2 at a designated cycle (step 101). The crawler is a means for visiting a page on the Internet 2 and collecting documents. For example, when collecting documents updated every day (designated cycle) on the previous day (documents updated since the previous collection), if today is January 1, 2004, the update date and time is updated by the crawler 11. Only the page of December 31, 2003 yesterday is collected and stored in the document DB 16 that manages the collected data at the update date and time. If today is January 2, 2004, the crawler 11 similarly collects only the pages with the update date yesterday January 1, 2004, and stores them in the document DB 16 that manages the collected data at the update date. To do. The collection period is specified in advance. For example, if “every day” is set, it is collected every day, and if “every day” is set, it is collected once every two days. Furthermore, since the collected pages contain tags and unnecessary character strings that are not related to the contents of the document, they are removed, a formatted document is created, and the document is stored in the document DB 16 in association with the collected document. Managed.

  Next, the target document set data creation unit 12 reads out the pre-formatted documents updated within the specified period independently from the specified period from the document DB 16, and collects the collected documents as a target word set for topic word extraction. The target document set is managed with the specified date and time and stored in the target document set DB 17 (step 102). For example, if you specify “Yesterday” for the date and time and “1 day” for the period, you can read out the formatted documents whose update date and time match the date and time, and put the documents together (for example, each document as one line Are concatenated as a target document set, managed by a specified date and time, and stored in the target document set DB 17. The date / time and period are initially set in advance. For example, if the date is set to “Yesterday”, the period is set to “1 day”, and if today is January 1, 2004, the formatted document with the update date of December 31, 2003 is applicable. .

It is also possible to create a more limited set of target documents by specifying not only the date / time and period but also the site name and field. The site name may not be a complete URL but may be defined as a part of the URL (for example, domain name in the case of “http://www.sample.com/sports”+partial character string). Further, the field is determined by a character string (for example, “sports”) included in the URL, a character string appearing in the document, or a result of estimating the field of the document by the field estimation unit learned in advance. Further, a character string statistic is calculated from the target document set, and is stored and managed in the target document set DB 17 in association with the used target character set. When calculating a string statistic, for example, by creating a suffix array (suffix array (see Reference 1)) that is also used in Non-Patent Document 1, the appearance of an arbitrary character string in a document The frequency and the number of documents in which the character string appears can be calculated. The suffix array is an array of all character strings (pointers to documents) from an arbitrary character in a sentence to the end of the sentence, sorted by character string. When the same character string appears in a plurality of places, the sorted arrays are adjacent to each other, so the appearance frequency can be obtained by counting the number. In addition, by obtaining the leading position of the document in advance, it is possible to know in which document the partial character string has appeared, so the number of documents in which each character string appears can be calculated. The data used in creating the suffix array is stored in the target document set DB 17 as statistical data. Note that if the same calculation is possible without using the method of creating a suffix array, it may be used.
Reference 1: Mikio Yamamoto, Kenneth W. Church, “Using Suffix Arrays to Compute Term Frequency and Document Frequency for All Substrings in Corpus”, Computational Linguistics, Vol27, No.1, 2001, pp.1-30.

  Next, the word extraction unit 13 reads the target document set of the target document set DB 17, divides the target document set into words, and extracts words (step 103). As a word extraction method, a method of performing morphological analysis and extracting a part-of-speech pattern that matches can be considered, but the latest topic word may include an unknown word. Since general morphological analysis uses a dictionary, it is impossible to deal with unknown words.

  Therefore, in the following, the word division based on the character string statistics will be described with reference to FIG. First, the word candidate extraction unit 13A reads the target document set from the target document set DB 17, extracts word candidates by the same method as the word candidate extraction of Non-Patent Document 1, and stores the word candidates in the target document set DB 17. In this example, it is possible to divide the target document set for each sentence and divide one sentence into words, but in order to perform the processing quickly, word candidates are extracted by the method of Non-Patent Document 1. For example, it is assumed that “... Tategami wolf ...” appears in the target document set a plurality of times, and as a result of word candidate extraction, “Vertical wolf” is one of the word candidates. Note that a character string such as “Tategamio” is not a word candidate if it is nested in “Tategamio wolf” or if the last character of “Tategamio” is divided by the same type of character (katakana). . Next, the word division unit 13B inputs word candidates from the target document set DB 17, performs word division on the word candidates based on the character string statistics, the character string length, and the character type, and then obtains the word division results as the target document set DB 17 Output to. For example, when one of the word candidates “vertical wolf's” is input, the character string input in order from the top, such as “vertical”, “vertical”,. Word scores are calculated for all substrings included.

For the word score, any statistic may be used as long as the input character string can be divided into words well. In addition, a plurality of methods may be prepared in advance, and a method used in the initial setting may be used. For example, tf * idf, which is often used to extract a characteristic keyword, which is obtained by multiplying the appearance frequency (tf) and the reciprocal of the document frequency (idf) as a statistic, can be considered here. The word score S (s) of the column s is calculated according to ridf (s) × (length (s) −1). Ridf (s) is called a residual IDF (ridf) of the character string s, and the word related to the contents is a statistic that tends to take a large value (reference document 2). Also, length (s) is the length of the character string s. When the appearance frequency is low, the statistic does not take an accurate value. For example, when the appearance frequency (tf) is 10 or less, ridf (s) = 0.
Reference 2: Kenji Kita, Kazuhiko Tsuda, Masato Sasabori, “Information Exploration Algorithm”, Kyoritsu Shuppan, published the first edition, January 1, 2002, pp. 43-45.

  In order to calculate the residual IDF of the character string s, the appearance frequency of the character string s, the number of documents including the character string s (document frequency), and the total number of documents in the document set are calculated in advance. Or read and calculate at runtime. For example, it is possible to obtain the appearance frequency and document frequency of a character string in a document set using Reference Document 1. However, in Reference Document 1, it is limited to an arbitrary character string having an appearance frequency of 2 or more. Therefore, it can be seen that when the character string is not found by the method of Reference 1 and appears in the document set, the appearance frequency is 1 (= document frequency 1). The total number of documents in the document set may be checked once in advance. From the above, using the string statistic of the target document set and the string statistic of another document set prepared in advance, add the string statistic (appearance frequency and document frequency) of each document set for a certain character string. Can be sought. The reason why not only the target document set but also the statistics of another document set is used is that errors can be reduced by supplementing the statistics when the appearance frequency of the target document set is low. Also, by calculating the statistics for other documents in advance, only the target document set needs to be calculated at the time of execution, rather than calculating the statistics at the time of execution for the entire target document set and another document set. Can reduce processing time.

  The left side of FIG. 4 is a part of the result of adding the statistics of the target document set and the separate document set, and the appearance frequency and document frequency of each character string from the total number of documents D = 100000 are added. Assume that you have asked for it. For example, in the case of “ta” in the word candidate “vertical wolf's”, the appearance frequency tf = 80000 and the document frequency df = 30000 are obtained, so that rdf (ta) = 0.74. From these, the word score S (ta) = ridf (ta) × (length (ta) −1) = 0. Similarly, word scores S (s) of all partial character strings of the input character string are obtained (right side in FIG. 4).

Next, word score correction will be described. A word string having a certain character type does not have an appropriate word score, and erroneous division is likely to occur. Therefore, the calculated word score is corrected. In the correction, the word score of a character string that is unlikely to be a word is set to 0, and a higher value is given to a character string that is likely to be a word and has a low word score. For example, if katakana, English letters, Arabic numerals, and Chinese numerals are consecutive in the same character type, the word score is 0 except for the longest one, and if it is the longest and the word score is 0, it may be greater than 0. give. In addition, the symbol score is set to 0 for two or more symbols. In the case of a character string including hiragana, if the length is one character and the document frequency is greater than or equal to the threshold, the word score is set to w (s) (w (s) is a function of s). In the case of a character string including hiragana, the word score is set to 0 when the length is 2 characters or more and ridf (s) is less than or equal to the threshold value. For example, as shown in FIG. 3, in the case of “vertical”, the word score is S (vertical) = 0.48, but in the word candidate “vertical wolf”, the Katakana partial character string of “vertical wolf” Therefore, S (vertical) = 0 (right side in FIG. 4). Further, the word score of “vertical wolf” is 0, but it becomes a very small value (for example, 0.0001) by the correction. Next, by dynamic programming (DP (see Non-Patent Document 3)), a word arrangement that maximizes the sum of the word scores is obtained from the word scores of the character strings. For example, the word candidates “Late wolf wolf” are arranged as “Late wolf (0.0001) / of (0.05)” (the number after the word is the word score). If correction of the word score is not performed, “vertical (0.48) / gummy (0.53) / wolf (3.99) / (0.05)” is obtained, and correct word division is obtained. I can't.
Reference 3 Kiyoshi Ishihata, “Algorithms and Data Structures”, Iwanami Shoten, first printed on March 30, 1989, pp. 399-401.

  Next, the word selection unit 13C in FIG. 2 inputs word division results of a plurality of word candidates from the target document set DB 17, narrows down by a predetermined word score threshold, character type, and character string length, and the remaining words Is output to the DB 18 after the topic. For example, if the word score is limited to a certain threshold value or Katakana and the character string length is 2 or more, “Tategami wolf” is output because it is Katakana and is stored in the target document set DB 17.

  Next, the topic level calculation unit 14 inputs the word output by the word extraction unit 13 from the target document set DB 17 and the previous target document set and the current target document set (time, period, etc.) in the target document set DB 17. The recent topic level is calculated based on each character string statistic (which differs between the previous time and the current time), and a topic word list arranged in the order of the topic level is output to the topic word DB 18 (step 104).

  The recent topic level is a statistic indicating the importance level of the words in the previous and current target document sets, and this time it is considered that the importance level has increased sharply compared to the previous time. It can be obtained by the ratio of the importance of the previous time and this time. It should be noted that the topic level may be obtained by a method other than that described here as long as the recent topic level is required, such as a difference instead of a rate, or a case where both the rate and the difference are considered.

  By preparing multiple statistics, you can select the topic level that suits the user. As statistics, appearance frequency (tf), tf * idf, and the like are conceivable. The more tf appears, the more important it is. Also, tf * idf is a product of the appearance frequency (tf) and the reciprocal of the document frequency (idf), and tf * idf has a higher value for words that appear biased in the document than the appearance frequency. . This means that if the frequency of appearance is high, the number of words that appear evenly tends to increase, and therefore, the word specialized to the content becomes a high value by considering the bias. If the number of documents in the target document set is D, tf * idf is calculated by tf * idf = tf * (log (D / df) +1). The statistics are not limited to these, and may be calculated by other methods. For example, when calculating the topic level of the word “vertical wolf” by tf * idf, assuming that the current day is one day and the previous day is the previous day, there are 100 documents in the target document set on the previous day, of which two documents If there were 120 “Tategami wolf” on the same day, and there were 120 documents in the target document set on that day, and 3 of those documents had “Tategami wolf” appeared 6 times, Tf * idf) / (tf * idf of the previous day) = 6 * (log (120/3) +1) / 2 * (log (100/2) +1) = 15.6 / 5.4 = 2.89 Become. Then, a topic word list is created in descending order of topic level and stored in the topic word DB 18.

  Finally, the display unit 15 inputs the topic word list output from the topic level calculation unit 14, and further inputs the target document set and statistics data of the target document set DB 17, and the target document set for the topic word. Context, the original URL containing the topic word, the title of the document in which the topic word appears, the frequency of appearance of the topic word in the document set, the number of documents that appear, the topic level, and the topic for a certain period The transition of statistics about words is formatted into a display format and output as a topic word list and related information.

  For example, as shown in FIG. 5A, when the user selects the date, date, number, field, site, topic level calculation method, the latest topic word list under the selected condition is output. If the topic word "baby" is in the topic word list and the highest word that appears in the same document as "baby" is "vertical wolf", it is included in the document containing the same information. Therefore, by excluding "baby" from the topic word list, it is possible to save the trouble of viewing the same information multiple times from the top of the list.

  Further, for example, when the fourth “vertical wolf” is clicked, as shown in FIG. 5B, the number of occurrences in the target document set of “vertical wolf” on January 1, 2004, which is the designated date and time. (Tf), the number of documents including the word (df), topic level, topic words included in the same document, statistics of topic words for a certain period (in this example, topic level), a transition graph, and topics The original URL including the word, the title of the document in which the topic word appears, and the context before and after the topic word in the target document set are displayed. If you want to see the original document, click the URL to display the actual document. As a result, when it is difficult to determine whether or not it is a topic word simply by looking at a topic word list at a certain date and time, it is possible to see information that can be used as a reference for the determination based on the transition graph of the statistic and the appearance frequency. In this example, the transition graph is only the topic level, but a plurality of statistics may be displayed, or a graph desired to be viewed may be selected from a plurality of statistics by the selection means. Moreover, you may change the width | variety and time of a period to look at by a selection means.

  In addition, in the case of a new word whose meaning is not understood, it is possible to help understanding by looking at the context of the topic word and the original document. Note that FIG. 5 is an example of display, and the data, order, and configuration of information to be displayed may be changed as appropriate.

  The word extracting method of the present invention is not only realized by dedicated hardware, but also a program for realizing the function is recorded on a computer-readable recording medium, and the program recorded on the recording medium is recorded. May be read by a computer system and executed. The computer-readable recording medium refers to a recording medium such as a floppy disk, a magneto-optical disk, a CD-ROM, or a storage device such as a hard disk device built in the computer system. Furthermore, a computer-readable recording medium is a server that dynamically holds a program (transmission medium or transmission wave) for a short time, such as when transmitting a program via the Internet, and a server in that case. Some of them hold programs for a certain period of time, such as volatile memory inside computer systems.

It is a block diagram which shows the structure of the word extraction apparatus which is one Embodiment of this invention. It is a block diagram which shows the structure of the word extraction part in FIG. It is a flowchart showing the outline | summary of a process of the word extraction device of FIG. It is a figure which shows an example of the word score of the arbitrary character strings calculated from the document set, and the word score corrected based on the input character string. It is a display example of a topic word list and related information.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Word extraction apparatus 2 Internet 11 Crawler 12 Target document set data preparation part 13 Word extraction part 13A Word candidate extraction part 13B Word division part 13C Word selection part 14 Topic degree calculation part 15 Display part 16 Document DB
17 Target document collection DB
18 Topic DB
101-105 steps

Claims (7)

A word extraction method performed in a word extraction device that extracts topic words in a specified period from a sequentially updated document set,
A document collection step in which the document collection means collects documents updated since the last collection from the document storage source at a specified cycle;
A target document set creation step in which a target document set creation unit creates a document set that is a target of word extraction by collecting documents updated from a collected document within a period specified independently of the period;
A word candidate extraction step in which the word candidate extraction means extracts word candidates except for a character string that appears multiple times in the target document set and that is nested by a character type;
The word dividing unit is a value obtained by multiplying the residual IDF of the character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate output by the word candidate extracting unit. Calculate a word score,
Among the partial character strings, a partial character string in which any one character type of katakana, English letters, Arabic numerals, or Chinese numerals continues is the longest length of the partial character string in which any one kind of character type continues. Correcting the word score of each of the partial character strings so that the word score is higher than the word score of the partial character string in which the continuous length of any one type of character is shorter than the longest length ,
Among the partial character string sequences that become the word candidates when the partial character strings are arranged, the partial character string sequence that maximizes the sum of the word scores of the partial character strings constituting the sequence. A word division step for outputting as a word division result ;
Word word selection means, among the word segmentation result of said word candidates outputted by the word segmentation step, the word segmentation result the word score matches a threshold value or character types and string length condition word scores predetermined A word selection step to extract as
A word extraction method comprising: Processing for calculating a word score that is a value obtained by multiplying the residual IDF of the partial character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate in the word division step in word extraction method according to claim 1, wherein the frequency of occurrence of the partial character string, characterized in that a zero residual IDF substring to be 10 or less. The topic level calculation means calculates the importance in the target document set for the extracted word and the importance in the document set for a certain period before the target document set. A topic level calculation step for calculating the topic level of a word;
A topic word list creating means for creating a topic word list in which words are arranged in descending order of topic degree;
The word extraction method according to claim 1 or 2, wherein: A word extraction device that extracts topic words in a specified period from a sequentially updated document set,
Document collection means for collecting documents updated since the last collection from the document storage source at a specified cycle;
A target document set creation means for creating a document set as a word extraction target by collecting documents updated within a period specified independently of the cycle from the collected documents;
Word candidate extraction means for extracting word candidates except for a character string that appears multiple times in the target document set and a character string that is nested or divided by a certain character type;
Calculate a word score that is a value obtained by multiplying the residual IDF of the character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate output by the word candidate extraction unit. And
Among the partial character strings, a partial character string in which any one character type of katakana, English letters, Arabic numerals, or Chinese numerals continues is the longest length of the partial character string in which any one kind of character type continues. Correcting the word score of each of the partial character strings so that the word score is higher than the word score of the partial character string in which the continuous length of any one type of character is shorter than the longest length ,
Among the partial character string sequences that become the word candidates when the partial character strings are arranged, the partial character string sequence that maximizes the sum of the word scores of the partial character strings constituting the sequence. A word dividing means for outputting as a word division result ;
Of word segmentation result of the word candidates the word dividing means is outputted, the word selection for extracting the word segmentation result the word score matches a threshold value or character types and string length condition word score predetermined as a word Means,
A word extraction device. The word dividing means includes
In the process of calculating a word score that is a value obtained by multiplying the residual IDF of the partial character string by the character string length of the partial character string for one or more consecutive partial character strings included in the word candidate, 5. The word extraction device according to claim 4, wherein a residual IDF of a partial character string having a character string appearance frequency of 10 or less is set to zero . For the extracted word, calculate the importance in the target document set and the importance in the document set for a certain period before the target document set, and calculate the degree of increase in importance compared to before as the topic level of the word Topic level calculation means to
A topic level list creation means for creating a topic word list in which words are arranged in descending order of topic level;
6. The word extraction device according to claim 4 or 5, wherein: The program for functioning a computer as each means of the word extraction apparatus of any one of Claims 4 thru | or 6 .

Images