JPH03131967A - Method for sorting japanese word - Google Patents

Abstract

PURPOSE:To efficiently sort words even in the case of a large word group by clustering words by an inter-word distance indicating the conceptional nearness of respective words to separate words. CONSTITUTION:The modification structure of words is obtained by a modification analysis part 1 and the combination of different modification and the appearance frequency of respective combinations are extracted by a statistic data extraction part as statistic data. After analyzing the numerized IV sort of a modifying word or a modified word in the statistic data, the Euclidean distance of values applied to the words is calculated and held. Processing is executed by a clustering part 6 and a hierarchical processing part 7 by using the calculated result as a distance matrix and a sorted result is obtained from a word sorting result output part 8.

Description

DETAILED DESCRIPTION OF THE INVENTION FIELD OF INDUSTRIAL APPLICATION The present invention relates to a Japanese word classification method in the field of language processing.

In conventional natural language processing, a dictionary of words classified by semantic concepts is essential for semantic analysis.

Conventionally, as a method for classifying words according to concepts, there is a top-down method in which a classification system is set by humans based on their knowledge and experience. However, since this method relies on human power, the production and design costs are high, and it is not easy to change the word classification according to the task.

To solve this problem, a method has been proposed to automatically classify words based on data obtained as a result of linguistic analysis of Japanese sentences. For example, Information Processing Society of Japan Transactions, V
OL, 26. No. 4, ``Dictionary structure and its learning function for dependency analysis'' and Japanese Patent Application Laid-Open No. 17237-1983.
As shown in Publication No. 2, the number of occurrences of combinations of word dependencies is counted, and from this data, the distance between words (=dissimilarity), which indicates the conceptual closeness of words to each other, is defined as follows. ), and then perform appropriate rastering (Centroid method, Single
This method uses a linkage method, etc.) to classify the information.

Here, the distance between words d(A, B) is d(A, B)=1-N(AB)/(N(A)+N(B)
)) N(A): Number of occurrences of dependent word A N(B): Number of appearance of dependent word B N(AB): Sum of the number of times dependent words A and B are the same adjunct word and relate to the same -object word, or, N, (AB): Sum of the number of times that the dependent words A and B are attached words with the same role and are related to the same receiving word. Total sum of the number of times N, (AB) related to the receiving word of , (AB): Total sum of the number of times related to the receiving word of the same similar word group belonging to level j, where the dependent words A and B are attached words of the same role, W: As a weighting coefficient defined.

In addition, clustering progresses from the integration of word groups with small distances to the integration of word groups with large distances. Therefore, in the hierarchization process, the upper limit value of the distance between word groups is associated with the predetermined synonymous and similar hierarchy levels, and the clusters generated based on these are hierarchized into each level.

Furthermore, by repeatedly performing this grouping process,
The number of words that are grouped is gradually increased.

Problems to be Solved by the Invention However, such conventional methods can be applied to a small number of word groups, especially in a limited field, but as the word group becomes larger, the classification accuracy decreases and grouping becomes difficult. The percentage is low.

Means for Solving the Problem In the Japanese word classification method, which uses statistical data obtained by analyzing Japanese sentences with a dependency analysis unit to classify words into groups based on semantic concepts, Quantifying the statistical data of words consisting of different combinations of dependencies and their frequencies ■ Quantifying by the class analysis unit ■ Giving quantities to words through class analysis, and at least the conceptual closeness of words to each other determined from this quantity The words are classified by clustering based on the distance between words.

Quantification of word statistical data consisting of different combinations of dependencies between action words and their frequencies Since we defined the distance between words using the quantity given by N-type analysis, we can efficiently calculate the distance even for large word groups. Can be classified.

Embodiment A first embodiment of the present invention will be explained based on FIG. This FIG.
The functions other than the class analysis section are the same as those of the conventional method.

First, a Japanese sentence example is subjected to morphological analysis if necessary, and then its dependency structure is obtained by the dependency analysis unit l. Next, the statistical data extraction unit 2 extracts a three-term set of "dependent word, attached word, and dependent word" or "dependent word,
Different combinations of case types, ternary sets of dependent words j, and the frequency of appearance of each combination are extracted as statistical data.

Next, the dependent word quantification type analysis section 3 performs quantification type analysis on the dependent words of this statistical data. Quantification type analysis is a method of assigning quantities to objects based on the affinities eij between the objects and positioning them in Euclidean space. That is, the two-dimensional quantity given to the object l is (xi
, yi), then (xi, yi) is determined so as to maximize (see the modern mathematics pillar ``Multivariate statistical analysis method'', etc.).

Therefore, for example, affinity elJ may be defined and analyzed as follows.

However, N(i): Frequency of appearance of dependent word i N(j): Frequency of appearance of dependent word l N(i, j): Frequency of occurrence of dependent word i and j in the same - dependent word The dimension of the quantity to be summed may be appropriately determined depending on the size of the eigenvalue obtained as a result of the analysis.

Similarly, for received words, the affinity eiJ may be defined and analyzed as follows.

However, N(i): Frequency of appearance of target word i N(j): Frequency of appearance of target word i N(t+j): Target word 'r J is the same - dependent on the same -
Summation of Frequencies from Dependent Words Quantification of Dependent Words or Subordinate Words After the class analysis, the inter-word distance calculation unit 5 calculates and holds the Euclidean distance of the quantity given to the words between each word. 2
If it is a dimension, d(Lj)=((x+-x+)"+(y+-y+
)')''. Using this calculation result as a distance matrix, the clustering unit 6 and the hierarchization processing unit 7
By performing the processing in the same manner as conventionally, the classification results are obtained from the word classification result output section 8.

Next, a second embodiment of the present invention will be described with reference to FIG. In this embodiment, the processing according to the embodiment described above is repeatedly executed. First, the word distance calculation unit 5,
The clustering unit 6 and the hierarchization processing unit 7 are provided separately for the dependent word side and the dependent word side, and the first classification process is performed for each dependent word and dependent word, and each result is sent to the dependent word knowledge unit 9, It is held in the received word knowledge section 10. Next, using these results, for example, the following affinity eij is defined by the recipient word and the dependent word, and the dependent words are quantified. ■Perform class analysis.

Dependent word quantification ■Finality in class analysis However, N(i): Frequency of appearance of dependent word l N(j): Frequency of appearance of dependent word j N, (i, J): Dependent word i + J is the same - related In uke, the sum of the frequencies related to the same dependent word Nk (l+, +): When the dependent word i + J is the same - dependent word, the uke of the same group belonging to level k is the sum of the frequencies related to the word w, doubleness coefficient Categories quantification ■Finality in class analysis However, N(i): Frequency of occurrence of subject word 1 N(j): Frequency of appearance of subject word j, (i, j): Same - dependency of subject word 1 + J Then, the sum of frequencies N, (l, J) from the same dependent word: sum of frequencies wk from the dependent words of the same group belonging to the same group, where J has the same - dependency, wk: weighting coefficient is defined as Using the number of words obtained in this manner, calculation of distance between words, clustering, and hierarchization processing are executed in the same manner as in the above embodiment.

Such processing is repeated until the word groups are saturated to a certain condition.

Furthermore, as before, the relationship between dependent words and dependent words is
The relationship between a noun and a verb through the same adjunct can also be expressed as the relationship between a modifier and a modified word in a compound word, or the dependency relationship between a noun and a verb using the same case type.

Effects of the Invention As described above, the present invention defines the distance between words using the quantity given by the quantification of statistical data of words consisting of different combinations of dependencies between words and their frequencies. Therefore, even large word groups can be efficiently classified.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing a first embodiment of the present invention, and FIG.
The figure is a block diagram showing a second embodiment of the present invention. l... Dependency analysis section, 3, 4... Quantification ■ Type analysis section 2nd high school 1 Diagram

Claims (1)

[Claims] In the Japanese word classification method, which uses statistical data obtained by analyzing Japanese sentences with a dependency analysis unit to classify words into groups based on semantic concepts, different combinations of dependencies between words and The statistical data of words consisting of their frequency is quantified by the Class IV analysis unit.
A Japanese word characterized in that the words are classified by performing class IV analysis and giving a quantity to each word, and then clustering and classifying the words based on the inter-word distance that indicates the conceptual closeness between the words determined from this quantity. Classification method.