JPH09120401A - Thesaurus preparing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically prepare thesaurus by inputting materials. SOLUTION: A morpheme analysis part 2 reads a character string, analyzes the character string into a string of words, and prepares and outputs a list of parts of speech of the respective words to a coocurrence extraction part. The cooccurrence extraction part 3 analyzes the list and extracts and outputs cooccurrence data to a coocurrence storage part 4. The cooccurrence storage part 4 stores the coocurrence data and appearance frequencies by data kinds in a storage device. A semantic distance calculation part 6 calculates the semantic distances between the words by a quantifying means from the data stored in the cooccurrence storage part 4 and outputs the calculation results to a thesaurus preparation part 7. The thesaurus preparation part 7 divides the words into groups on the basis of the semantic distances received from the semantic distance calculation part 6 and further divides the groups into groups. After operation is completed for all the words, the thesaurus preparation part 7 outputs the words which are hierarchically grouped as a thesaurus.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for automatically creating a thesaurus in which words are grouped, and can be applied to a natural language processing system such as machine translation.

[0002]

2. Description of the Related Art A machine translation system that translates from Japanese to English or from English to Japanese, and nature of Japanese input FEP that converts voice input, kana input, or romaji input into kana-kanji mixed sentences In a language processing system, in order to eliminate polysemy (ambiguity) resulting from simple syntactic analysis, a semantic code is given to each word in the dictionary in advance, and the polysemy is resolved or reduced by referring to this semantic code. The method was taken.

For example, the expression "pork does not eat meat" means "meat does not eat pork" in addition to the interpretation that "pork does not eat meat" unless it is judged by referring to a meaning code. The interpretation which does not do is also approved.

Further, if a human gives the meaning code only by intuition, for example, from the expression "meat is attached to the body", it is judged that "meat" has a meaning close to weight, and the meaning code of food is given. Forgetting can happen. In this case, if the judgment is made by referring to the meaning code, only the interpretation that does not mean "meat does not eat pork" is established.

Therefore, a method for automatically generating a more appropriate semantic code system so that the semantic processing by the semantic code always succeeds has been researched (reference "Rei Sugimura,
Koji Kakihara, Masahiko Ishikawa, Riku Kawagoe, Shoichi Aoyama, "Automatic Generation of Semantic Code System", Information Processing Society of Japan, Natural Language Processing 78
-4, July 19, 1990 ”).

[0006]

However, the conventional attempt to automate the creation of the thesaurus is to create from the examples collected by working manually, and compared with the case where the meaning code is directly added manually. Therefore, the influence of the difference in the interpretation by the worker can be reduced, but which example is extracted depends on the worker, and the bias due to the worker cannot be avoided. Here, the thesaurus is a systematic collection of words for each similar concept, organizing them, and developing them into superordinate concepts.

Further, since a great deal of work is required to collect examples for creating a thesaurus, it is not necessary to create a thesaurus dedicated to specific documents / materials, and various documents / materials can be collected in one thesaurus. I was trying to cover. For this reason, for example, in a sentence about “fat” or “thinning”, it is not possible to deal with “meat” being closer to “weight” rather than food, and semantic processing often fails.

[0008]

According to a first aspect of the present invention, there is provided a thesaurus that includes a morpheme analysis unit that analyzes an input sentence to decompose it into words and specifies a part of speech of each word, and groups the words into a thesaurus. The creation device is characterized by including the following means.

That is, co-occurrence extraction storage means for extracting co-occurrence data from the information analyzed by the morphological analysis means, and storing the appearance frequency for each co-occurrence data type together with the co-occurrence data.
The first co-occurrence data is constructed from the co-occurrence data stored in the co-occurrence extraction storage means and the appearance frequency for each co-occurrence data type.
Between the constituent words and / or the semantic distance calculating means for calculating the semantic distance information between the second constituent words forming the co-occurrence data, and the first constituent word based on the semantic distance information, and Or a thesaurus generating means for grouping the second constituent words.

According to a second aspect of the present invention, in the thesaurus creating apparatus according to the first aspect, there is provided frequency management means for managing distribution information of appearance frequency for each co-occurrence data type stored in the co-occurrence extraction storage means. The co-occurrence extraction storage means is characterized in that, when a certain condition is satisfied, the co-occurrence data information having a low appearance frequency is deleted based on the information obtained from the frequency management means.

In the inventions of claims 1 and 2, the co-occurrence extraction storage means extracts the co-occurrence data and records the appearance frequency for each co-occurrence data type, and the semantic distance calculation means configures the co-occurrence data. The semantic distance information between the first constituent words and / or the second constituent words is calculated for all combinations, and the thesaurus generating means calculates the first constituent words and / or the first constituent words based on the semantic distance information. By grouping the two constituent words, the entire process from morphological analysis of the input sentence to thesaurus generation can be performed automatically.

Further, in the invention of claim 2, the frequency management means manages the distribution of the appearance frequency for each co-occurrence data type,
When a certain condition is satisfied, the co-occurrence extraction storage unit deletes the information of the co-occurrence data having a low frequency of appearance based on the information obtained from the frequency management unit, thereby saving the resources related to the co-occurrence storage.

[0013]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of a thesaurus generator according to the present invention will be described in detail below with reference to the drawings.

(First Embodiment) FIG. 1 shows a functional configuration of a thesaurus creating apparatus according to the first embodiment. Actually, the thesaurus creating apparatus according to the first embodiment is realized by a computer system such as a workstation or a personal computer equipped with a large-capacity auxiliary storage device, and its hardware configuration is omitted.

As shown in FIG. 1, the thesaurus creating apparatus according to the first embodiment functionally has a control unit 1, a morphological analysis unit 2, a co-occurrence extraction unit 3, a co-occurrence storage unit 4, and a semantic distance. It is composed of a calculation unit 6 and a thesaurus generation unit 7.

The control unit 1 receives a read destination of a character string which is input data and a write destination of a thesaurus which is output data, and controls and operates each unit in the thesaurus creating apparatus.

The morphological analysis unit 2 inputs a character string from a reading destination of the character string, analyzes the character string into a string of words, and specifies each part of speech.

The co-occurrence extraction unit 3 extracts a portion in which three words of a noun, a particle, and a verb are arranged in this order from the morpheme analysis result by the morpheme analysis unit 2. Hereinafter, this part is called co-occurrence.

The co-occurrence storage unit 4 stores the co-occurrence extracted by the co-occurrence extraction unit 3 and the appearance frequency of the co-occurrence in a storage device.

The semantic distance calculation unit 6 reads the information stored in the storage device by the co-occurrence storage unit 4 and calculates the semantic closeness of the co-occurring words by a quantification method.

The thesaurus generator 7 includes a semantic distance calculator 6
A group of words that are close in meaning is created based on the value of the closeness of meaning between the words obtained by, and the group is output to the specified writing destination of the thesaurus.

The operation of the thesaurus creating apparatus according to the first embodiment will be described below.

Upon receiving the read destination of the character string which is the input data, the control unit 1 sends the read destination of the input character string to the morpheme analysis unit 2 and the co-occurrence accumulation of the character string from the morpheme analysis unit 2 is performed. A series of processes up to the part 4 is activated.

The morphological analysis unit 2 reads the character string from the reading destination of the character string, analyzes the character string into a sequence of words, creates a list specifying the part of speech of each word, and outputs it to the co-occurrence extraction unit 3. To do.

The co-occurrence extraction unit 3 analyzes the list input from the morpheme analysis unit 2, and if there is a part in which three words of a noun, a particle, and a verb are arranged in this order, the co-occurrence list is used as a co-occurrence. It is output to the origination / accumulation unit 4.

The co-occurrence storage unit 4 retrieves the co-occurrence input from the co-occurrence extraction unit 3 from the storage device, and if already stored, increases the appearance frequency of the retrieved record by one, updates it, and stores it. If not, the input co-occurrence frequency is set to 1 and stored in the storage device as a new record. The co-occurrence storage unit 4 notifies the control unit 1 of the end of processing when all processing of the input data is completed.

When the control unit 1 receives the output destination of the thesaurus which is the output data, the control unit 1 activates a series of processes from the semantic distance calculation unit 6 to the thesaurus generation unit 7 and, at the same time, sets the thesaurus write destination to the thesaurus generation unit. Send to 7.

First, the semantic distance calculation unit 6 stores the logical occurrence frequency of co-occurrence and the actual co-occurrence storage unit 4 stores it in a storage device assuming that each word appears randomly without having a semantic relationship. The bias with respect to the appearance frequency of the co-occurrence is calculated in the same manner as the calculation of χ (chi) square. Next, the semantic closeness between words is calculated by comparing the distribution of words (groups) that have a co-occurrence relationship. These calculation methods will be described below.

First, the degree of co-occurrence relation Wmn (m specifies the type of the first constituent word, n is the value of all the combinations of the first constituent word and the second constituent word that form the co-occurrence). (Specifying the type of the second constituent word) is calculated, and then the semantic distance between the two kinds of first constituent words or the semantic distance between the two kinds of second constituent words is calculated. Distance Dij (i
And j respectively specify different types of first constituent words or different types of second constituent words) for all combinations. In addition,
In the present embodiment, the first constituent word is a noun,
The second constituent word is a verb.

The total number of co-occurrence data is T, the number of input co-occurrence data whose word m is the first constituent word is Cm,
The number of input co-occurrence data in which the word n is the second constituent word is Cn, the word m is the first constituent word, and
When the number of input co-occurrence data in which the word n is the second constituent word is Cmn, the co-occurrence relation degree Wmn between the word m and the word n first obtained by the semantic distance calculation unit 6 is
It can be expressed by equation (1).

[0031]

(Equation 1) Here, wmn in the equation (1) follows the following equation (2).

[0032]

(Equation 2) This equation (2) is based on the following concept. The number of input co-occurrence data (appearance frequency) Cmn in which the word m is the first constituent word and the word n is the second constituent word is very small in comparison with the total number T of the co-occurrence data. Therefore, the number Cmn is considered to follow the Poisson distribution.

The expected value of the number of co-occurrence data in which the word m is the first constituent word and the word n is the second constituent word is given by the following expression (3).

[0034]

(Equation 3) Since it follows the Poisson distribution, its standard deviation is given by equation (4) below.

[0035]

(Equation 4) Therefore, how much the actual co-occurrence relation data number Cmn with the word m as the first constituent word and the word n as the second constituent word deviates from its expected value is determined by a multiple of the standard deviation. What is shown (co-occurrence degree) is the following expression (5).

[0036]

(Equation 5) The probability that the appearance frequency becomes 0 is given by the following equation (6).

[0037]

(Equation 6) By the way, since it follows the Poisson distribution, even if the expected value has a certain value, the probability that the actual appearance frequency becomes 0 is high, and the value changes depending on the expected value. Also, the Poisson distribution is not symmetrical about the expected value. Therefore, the degree of co-occurrence relation of the above equation (2) is modified in consideration of the probability that the appearance frequency is zero.

Based on such an idea, the degree of co-occurrence relation between the first constituent word m and the second constituent word n was quantified by wmn according to the equation (2). Quite wide. Therefore, the co-occurrence degree w
It is decided to convert the co-occurrence degree Wmn in which the range of possible values of mn is narrow. The expression (1) also intends that the range of the co-occurrence degree Wmn can be -1 to -1.

The semantic distance Dij between the word i and the word j calculated by the semantic distance calculation unit 6 can be expressed by the following equation (7). The expression (7) is an expression for obtaining the semantic distance Dij between the two types of first constituent words i and j. The formula for obtaining the semantic distance Dij between the two types of second constituent words i and j is almost the same as the formula (7), and therefore its description is omitted.
In addition, the meaning of the equation will also be explained for equation (7).

[0040]

(Equation 7) However, Σ is performed on the second constituent word k in which at least one of the co-occurrence degree Wik and Wjk is positive.

This expression (7) is a modification of the semantic similarity dij shown in the following expression (8).

[0042]

(Equation 8) In this embodiment, as described in the proviso in the expression (7), the first constituent word i of the input certain co-occurrence data and the co-occurrence relation degree Wik are strong to some extent, or the other first
Considering the set K of the second constituent word k having a relatively high degree of co-occurrence relation Wjk with respect to the first constituent word j and the first constituent words i and j
It aims to capture the semantic distance between them. Only the elements of this set K have the value of the co-occurrence relation degree Wik for the word i, and the value of the co-occurrence relation degree Wjk for the word j. That is, for a certain second constituent word k, there is a co-occurrence relation degree Wik with the word i and a co-occurrence relation degree Wjk with respect to the word j, and such a pair of data has only elements of the set K. Equation (7) is used to obtain a correlation coefficient value dij between one element (data relating to word i) and the other element (data relating to word j) of the pair with respect to such plural pairs of data. It is what Therefore, if this value dij is large, both elements, and thus the second constituent word k
The relationship (semantic similarity) between the first constituent words i and j mediated by is high, and conversely, if this value dij is small, the first constituent word mediated by the second constituent word k is The relationship (semantic similarity) between i and j is low.

As described above, the equation (7) is obtained by modifying the equation (8). In order to prevent the denominator in equation (7) from becoming 0, 1 is added. Also (7)
The reason that the part corresponding to the expression (8) is subtracted from 1 in the expression is that the concept of the name of distance, that is, the smaller value means the closer. Further, in the expression (7), the reason why it is divided by a predetermined number is to set the range of values that the distance Dij can take to a range of 0 to 1. Defining the range in this way facilitates relative comparison of distances.

As described above, based on the co-occurrence degrees Wik and Wjk with the second constituent word k, the first expression is obtained by the expression (7).
It is possible to obtain the semantic distance Dij between the two types of words i and j that are the constituent words of. Although the description of the formula is omitted, the degree of co-occurrence relation Wki with the first constituent word k, Wki
Based on kj, the semantic distance Dij between the two types of words i and j, which are the second constituent words, can be obtained.

The semantic distance calculation unit 6 reads the information stored in the storage device by the co-occurrence storage unit 4, calculates the semantic distance Dij using the above calculation method, and calculates the word i, the word j, and the semantic distance D.
The list having the set of ij as elements is sent to the thesaurus generator 7. In addition, the semantic distance calculation unit 6 does not calculate the semantic distance between the same words or the semantic distance between words whose appearance frequency is extremely low. This is because the number obtained by calculation becomes meaningless when the appearance frequency is extremely low.

The thesaurus generator 7 divides each word so that the number of groups is the smallest within a predetermined number of groups per layer (groups do not overlap each other). The term "group" used here means a semantic distance Dab between a word a and a word b belonging to the same group.
, Always satisfies the condition of Dab <Dac for the semantic distance Dac with the word c that does not belong to the same group. Further, these groups are sequentially divided so that the number of groups is the smallest within the range of the number of groups per layer, and the subdivision is repeated up to the number of layers which repeats a predetermined subdivision, or the one layer Subdivision operations are recursively performed until no more groups can be created within the number of groups. Also, the above Dab <D
In addition to the ac condition, it is determined whether or not grouping can be performed within a predetermined number of constituent words per group. In the above grouping, the semantic distance between the same words, which has not been calculated by the semantic distance calculation unit 6, is set to 0, and the semantic distance between words having an extremely low appearance frequency is set to 1.

Such grouping will be described with reference to FIG. 2 by taking as an example the operation of the thesaurus generator 7 when grouping the words in Table 1. Table 1 shows cucumber,
It shows the semantic distance between each word of tomato, pumpkin, melon, apple, watermelon, and peach, and it has already been recursively grouped up to N-1 hierarchical levels.
Suppose you are trying to group a hierarchy. Here, N is a positive integer.

[0048]

[Table 1] Here, the number of groups per layer ≧ 2, the number of constituent words per group ≧ 2, the number of layers in which subdivision is repeated ≦ M
(M is a positive integer and N <M).

If cucumber is a and tomato is b, D
ab = 0.2, and Dab <Dac holds for any of melon, apple, watermelon, and peach as c. Also, if the pumpkin is b, Dab = 0.3, and if any of melon, apple, watermelon, and peach is c, Dab <Dac holds. Furthermore, when tomato is c, Dab = 0.
3 and any c of melon, apple, watermelon, peach
Also, Dab <Dac holds. Therefore, cucumber,
It can be seen that tomatoes and pumpkins belong to the same group (group 1), and melons, apples, watermelons, and peaches do not belong to this group. Further, since the number of constituent words is three, the condition of the number of constituent words per group ≧ 2 is also satisfied. Similarly, if any one of melon, apple, watermelon, and peach is designated as a and b and any one of cucumber, tomato, and pumpkin is designated as c, Dab <Dac holds. Therefore,
It can be seen that melons, apples, watermelons and peaches belong to the same group (group 2), while cucumbers, tomatoes and pumpkins do not belong to this group. Further, since the number of constituent words is four, the condition of the number of constituent words per group ≧ 2 is also satisfied (Nth hierarchy: step 201).

As described above, each word in Table 2 could be divided into two groups.
Can be further divided into two groups. However, since grouping is performed so that the number of groups is minimized within a predetermined number of groups per layer, the Nth layer is divided into two groups instead of three groups.

When the creation of the group is completed, the identification number of the word that does not belong to the group is set to 0 (in the example of Table 1, there is no word that does not belong to the group), and the words belonging to the group are further subdivided into subdivided groups. It is checked whether it is possible, and if it can be subdivided, the group is passed as a population and recursively divided into groups. If it cannot be subdivided, an identification number corresponding to the group is added to each word. Group 1 consists of 3 words and cannot be subdivided any further, so [cucumber, 1], [tomato,
1] and [pumpkin, 1].
To further subdivide, the number of groups per layer ≧ 2
Since the number of constituent words per group is ≧ 2, at least 4 words (2 × 2) are required for subdivision, and N + 1 ≦
It is necessary to satisfy M. Since the group 2 can be subdivided (the number of words forming the group = 4, N + 1 ≦
M), a partial list of words shown in Table 2 is created, and this partial list is delivered to perform recursive grouping processing (Nth hierarchy: steps 202 to 207).

[0052]

[Table 2] Grouping in the (N + 1) th layer when the list of Table 2 is delivered from the Nth layer will be described.

In Table 2, assuming that melon is a and watermelon is b, Dab = 0.1, and Dab <Dac holds for both apple and peach. If apple is a and peach is b, Dab = 0.1,
Dab <Dac holds regardless of whether melon or watermelon is c. Further, since the number of constituent words is two, the condition of the number of constituent words per group ≧ 2 is also satisfied. Thus, melons and watermelons form one group (group 1), and apples and peaches form one group (group 2). Since all four words belong to any group, there are no words that do not belong to the group. In addition, since each group consists of 2 words, it cannot be subdivided any further, so [melon, 1], [watermelon,
1], [apple, 2], and [peach, 2] are assigned identification numbers corresponding to groups (N + 1st layer: steps 201 to 205).

Even if the constituent words of each group are 4 words or more in the (N + 1) th hierarchy, each group cannot be subdivided when N + 2> M.

Returning to the Nth hierarchy, [melon, 21],
[Watermelon, 21], [Apple, 22], [Peach, 22]
As described above, the identification number of the Nth layer is added to the identification number given in the (N + 1) th layer. Although the example in Table 2 does not apply, when no group can be created in the (N + 1) th hierarchy, [melon, 20], [watermelon, 20],
Instead of using [apple, 20] and [peach, 20], the identification number 0 assigned in the (N + 1) th layer is deleted and [melon, 2], [watermelon, 2], [apple, 2], [ peach,
2] (Nth layer: step 208).

As shown in FIG. 3, the pairs of words and identification numbers obtained by the above-mentioned operation are obtained by grouping words having a close semantic distance and further subdividing the groups into a systematic group. .

After the operation is completed for all the words, the thesaurus generator 7 outputs the pair of the word and the identification number grouped by the above operation to the writing destination of the thesaurus received from the controller 1 to the controller 1. Notify the end of processing.

As described above, according to the first embodiment, the morphological analysis unit reads the input sentence from the read destination of the input sentence, decomposes the input sentence into words, and specifies the part of speech of the words. The co-occurrence storage unit stores the occurrence frequency for each co-occurrence and co-occurrence data type, and the semantic distance calculation unit calculates the semantic distance between words from the appearance frequency for each co-occurrence and co-occurrence data type. Since the thesaurus calculation unit calculates the grouped words based on the semantic distance and outputs the word-identification number pair to the thesaurus writing destination, the thesaurus can be automatically obtained from the material.

(Second Embodiment) The functional configuration of the thesaurus generator in the second embodiment shown in FIG. 4 is the first.
The frequency management unit 5 is added to the thesaurus creating apparatus of the embodiment, and the same or corresponding parts as those in FIG. 1 are designated by the same reference numerals. The components other than the co-occurrence storage unit 4 and the semantic distance calculation unit 6 in this embodiment function in the same manner as in the first embodiment, and therefore their explanations are omitted.

The co-occurrence storage unit 4 stores the co-occurrence extracted by the co-occurrence extraction unit 3 and the appearance frequency of this co-occurrence in a storage device, and informs the frequency management unit 5 that the appearance frequency has changed. When the capacity of the storage device is approached, the co-occurrence with a low appearance frequency is deleted from the storage device by referring to the information from the frequency management unit 5.

The frequency management unit 5 sets the appearance frequencies to 1, 2, 3,
4, 5, 6, 7, 8, 9, 10 or more, less than 15, 15 or more, less than 20, 20 or more, less than 25, 25 or more, less than 30, 30
It is divided into 16 layers of 40 or more, 40 or more, less than 50, and 50 or more, and the number of records recorded in each layer is stored.

The semantic distance calculation unit 6 uses the information from the co-occurrence storage unit 4 and the frequency management unit 5 to determine the semantic closeness of words that form co-occurrence based on the frequency of occurrence of co-occurrence by a quantification method. Is to calculate the

The operation of the thesaurus generator in this embodiment will be described below.

Upon receipt of the read destination of the character string which is the input data, the control unit 1 sends the read destination of the input character string to the morpheme analysis unit 2, and the morpheme analysis unit 2 for the relevant character string transmits the read data to the frequency management unit. A series of processes up to 5 is activated.

Since the morphological analysis unit 2 and the co-occurrence extraction unit 3 operate in the same manner as in the first embodiment, their explanation is omitted.

The co-occurrence storage unit 4 retrieves the co-occurrence input from the co-occurrence extraction unit 3 from the storage device and, if it is already stored, updates the appearance frequency of the retrieved record by one and updates it. If not, the appearance frequency of the input co-occurrence is set to 1 and stored in the storage device as a new record, and the usable amount of the storage device is inspected. When the usable amount is less than the first predetermined reference amount, the co-occurrence storage unit 4
If the frequency management unit 5 obtains the frequency distribution information of the co-occurrence records and deletes the frequency from 1 to any frequency, the usable amount of the storage device is equal to or more than a second reference amount which is set in advance (second 2 reference amount> first reference amount) is calculated, the co-occurrence record of the frequency is deleted from the storage device based on the calculation result, and the next record is input. At the same time, the co-occurrence storage unit 4 sends the change information of addition / update / deletion of the co-occurrence record stored in the storage device to the frequency management unit 5 each time the change information is changed.

The frequency management unit 5 is added from the co-occurrence storage unit 4.
When change information of update / deletion is input, the appearance frequency is 1, 2, 3, 4, 5, 6, 7, 7, 8, 9, 10 or more and less than 15, 15 or more and less than 20, 20 or more and less than 25, 25 or more. Three
When the co-occurrence unit 4 refers to, managing the number of records of co-occurrence for each layer divided into 16 layers of less than 0, 30 or more, less than 40, 40 or more, less than 50, 50 or more,
The number of records of each layer having the appearance frequency is sent to the co-occurrence storage unit 4. When the processing of the input data is completed, the frequency management unit 5 notifies the control unit 1 of the end of processing.

When the control unit 1 receives the output destination of the thesaurus which is the output data, the control unit 1 activates a series of processes from the semantic distance calculation unit 6 to the thesaurus generation unit 7 and, at the same time, sets the thesaurus write destination to the thesaurus generation unit. Send to 7.

The semantic distance calculating unit 6 refers to the information from the co-occurrence accumulating unit 4 and the frequency managing unit 5 and calculates the semantic distance Dij by using the same calculation method as that of the first embodiment. A list including elements of i, word j, and semantic distance Dij as elements is sent to the thesaurus generator 7. The semantic distance calculation unit 6 includes the frequency management unit 5 and the co-occurrence storage unit 4 when calculating the semantic distance.
Since the same operation as that of the first embodiment is performed except that the information from is referred to, the description thereof will be omitted.

Since the operation of the thesaurus generator 7 is exactly the same as that of the first embodiment, its explanation is omitted.

As described above, according to the second embodiment, the frequency management unit manages the distribution of the co-occurrence frequency of appearance, and the co-occurrence storage unit deletes the co-occurrence information having a low frequency of occurrence. Therefore, the capacity of the storage device necessary for accumulating co-occurrence can be reduced, and high-speed processing can be performed.

Therefore, the thesaurus generator can be realized even in a low-speed computer system with a small storage capacity.

(Other Embodiments) The following (a) to (d) will be given to give some examples of other embodiments in which a part of the first and / or second embodiment is modified. .

(A) The data sent from the semantic distance calculation unit 6 to the thesaurus generation unit 7 may be limited to data with a small semantic distance. In this case, speeding up of processing can be expected.

(B) When calculating the semantic distance, the semantic distance calculation unit 6 sets the appearance frequency for each co-occurrence data type to a predetermined ratio with respect to the sum of the appearance frequencies for each co-occurrence data type. If not, the co-occurrence may be excluded from the calculation target. Alternatively, co-occurrence that is less than a predetermined appearance frequency may be excluded from the calculation target. By doing so, it is possible to obtain an appropriate thesaurus for a frequent word in a specific document or material, even in a device with a relatively small storage capacity or calculation speed.

(C) Although the noun-verb relation co-occurrence is extracted, the noun-noun relation co-occurrence, the noun-adjective relation co-occurrence, etc. may be extracted and processed.

(D) The definition of the group is such that the semantic distance Dac between a word a and a word b belonging to the same group is always Dab <Dac with respect to the semantic distance Dac with a word c not belonging to the same group. However, the group may be defined by another statistical method.

In the second embodiment, the appearance frequency is 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or more 1
Less than 5, 15 or more and less than 20, 20 or more and less than 25, 25 or more and less than 30, 30 or more and less than 40, 40 or more and less than 50, 5
Although the number of records of co-occurrence is recorded for each layer by dividing it into 16 layers of 0 or more, the delimiter of each layer and the number of layers can be set arbitrarily, and the data amount of the input sentence can be set. You may make it change according to it.

Furthermore, in the first and second embodiments, the case where Japanese co-occurrence data is processed is shown.
It can be applied to other languages.

[0080]

As described above, according to the present invention, the semantic distance between the constituent words is obtained from the co-occurrence data extracted from the material, and the words are grouped based on this semantic distance. By doing so, the thesaurus can be automatically obtained, and natural language processing can be made more appropriate by performing the semantic processing of the material using the thesaurus created in advance from the material. .

[Brief description of the drawings]

FIG. 1 is a functional configuration diagram showing a function of a thesaurus creating device according to a first embodiment.

FIG. 2 is a flowchart of a grouping process according to the first embodiment.

FIG. 3 is a conceptual diagram of a thesaurus according to the first embodiment.

FIG. 4 is a functional configuration diagram showing functions of the thesaurus creation device according to the second embodiment.

[Explanation of symbols]

 2 Morphological analysis unit 3 Co-occurrence extraction unit 4 Co-occurrence storage unit 5 Frequency management unit 6 Semantic distance calculation unit 7 Thesaurus generation unit

Claims (2)

[Claims] 1. A thesaurus creating device for analyzing a input sentence to decompose the input sentence into words and for specifying a part of speech of a word, and grouping the words to create a thesaurus. Co-occurrence data stored in the co-occurrence extraction storage means for extracting co-occurrence data from information and storing the appearance frequency for each co-occurrence data type together with the co-occurrence data The first to construct co-occurrence data from the appearance frequency for each
Between the constituent words and / or the semantic distance calculating means for calculating the semantic distance information between the second constituent words forming the co-occurrence data, and the first constituent word based on the semantic distance information. And / or a thesaurus generating means for grouping the second constituent words into a group. 2. A frequency management means for managing distribution information of appearance frequency for each co-occurrence data type stored in the co-occurrence extraction storage means, the co-occurrence extraction storage means satisfying a certain condition. The thesaurus creating apparatus according to claim 1, wherein the information of the co-occurrence data having a low appearance frequency is deleted based on the information obtained from the frequency managing means.