JP2001331514A - Document classification device and document classification method - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide a document classification device and a document classification method that can prevent the classification of documents that do not meet the user's intention and can easily generate an initial classification representative feature vector. I do. A document input unit, a document analysis unit that analyzes words of document data, a document feature vector generation unit that calculates a document feature vector for a document,
A classifying representative vector generating unit 104 for generating a classifying representative vector having the same number of dimensions as the document feature vector, a refining exclusion vector specifying unit 105 for specifying a classifying representative vector for which refining processing is not to be performed, Document data allocating unit 10 to be allocated to any one of
6 and a classification representative vector refining unit 1 for recalculating the classification representative vector based on the document feature vector allocated by the document data allocation unit except for the refinement exclusion vector.
07 and a classification result storage unit 108.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a document classification device and a document classification method.

[0002]

2. Description of the Related Art In recent years, the spread of the Internet and the like has made it possible to access a large amount of document information, and the collected large amount of document information is classified into meaningful groups, and intellectual work such as grasping the structure of a document set. Is beginning to take place. When analyzing a large document set, the document set is first classified into several topics, and various tasks are performed using the obtained partial document set (a plurality of documents collected according to a certain standard) as a unit. Work can be performed efficiently. When a user manually classifies a large amount of document information, human / time costs become enormous. Therefore, an apparatus that can automatically classify a document set according to the contents of the document is desired.

[0003] By using natural language processing such as Japanese morphological analysis to extract the words that compose them from a document, the document becomes a word frequency vector (document feature vector).
It is possible to express as a space. This is called a vector space model of the document and is widely used. In the vector space model, a document is mapped in a measurable space, so that automatic classification based on the content of the document can be performed using a statistical method.

[0004] As a typical technique for performing automatic classification based on the contents of documents using a statistical technique, there is a clustering technique as described in Japanese Patent Application Laid-Open No. Hei 7-114572.

[0005] However, the vector space model in the clustering method has a problem in that it has to hold and calculate high-dimensional vector information for each of a large number of documents, so that there is a large restriction on computational resources.

Therefore, as in the non-hierarchical clustering method, first, some initial classification representative feature vectors are set, and a document is appropriately classified based on the similarity between the classification representative feature vector and the document feature vector. It is efficient to use a method of classifying documents by assigning them to values.

There is also an advantage that the user can specify the initial classification representative feature vector to perform the document classification intended by the user.

[0008]

However, in the conventional non-hierarchical clustering method, a refinement process (the repetition process) that dynamically changes the classification representative feature vector and repeatedly assigns a document in order to improve the classification accuracy is performed. For example, it is common to obtain the centroid of the vector, classify the vector of the vector centroid as a new classification representative feature vector, and repeat this classification operation a predetermined number of times or until the classification error is minimized.) By performing the refining process, the classification representative feature vector initially specified by the user changes, and as a result, a document classification result that does not match the user's intention is generated (for example, by the refining process,
As a result of the movement of the vector centroid, the original classified representative feature vector may be classified into another classified representative feature vector. ).

[0009] When the user can specify the classified representative feature vector, the information usable by the user may be, for example, a document and words constituting the document. As a method of designating the initial classification representative feature vector, the user can easily specify the initial classification representative value by using a logical expression having elements of a document and a word widely used in the conventional document search. At this time, the quality of the obtained classification result is considered to be different depending on how the initial classification representative feature vector is generated from the specified logical expression, but this is not mentioned in the conventional method.

In view of the above problems, the present invention excludes the initial classification representative feature vector specified by the user,
By performing the refining process, it is an object to prevent the refining process from being performed while the initial classification representative feature vector specified by the user is fixed, and to prevent the document classification not meeting the user's intention from being performed.

The specified logical expression is developed as a logical sum expression in units of a logical product expression of a document or a word, and each logical product expression is synthesized with a feature vector of a document or a word constituting the logical product expression. It is an object of the present invention to easily generate an initial classification representative feature vector for a user by using each element (logical product expression) constituting the logical sum expression as an initial classification representative feature vector.

That is, an object of the present invention is to provide a document classifying apparatus and a document classifying method which can prevent the classification of documents that do not meet the user's intention and can easily generate an initial classification representative feature vector. Is what you do.

[0013]

The present invention has means, functions and effects for solving the above problems as described below.

According to a first aspect of the present invention, there is provided a document classification apparatus for setting a plurality of initial classification representative feature vectors and classifying a document based on the similarity between the classification representative feature vector and the document feature vector. And a refining processing unit (for example, a classification representative vector refining processing unit 107 in FIG. 1) for dynamically changing a classification representative feature vector and repeatedly assigning a document in order to improve classification accuracy. The refining processing unit does not perform the refining processing on some or all of the classification representative feature vectors.

According to the first aspect of the present invention, the refining process is bypassed for some or all of the initial classification representative feature vectors, so that the partial or all of the initial classification representative vectors are represented. It is possible to obtain a classification result that clearly indicates the viewpoint of performing the classification.

According to a second aspect of the present invention, there is provided a document classification apparatus for classifying a document according to the content of the document.
A document input unit for inputting document data (for example, document input unit 101 in FIG. 1) and a document analysis unit for analyzing words of the document data (for example, document analysis unit 10 in FIG. 1)
2) a document feature vector generation unit (for example, the document feature vector generation unit 103 in FIG. 1) that calculates a document feature vector for the document based on the document analysis result of the document analysis unit; A classification representative vector generation unit (for example, the classification representative vector generation unit 104 in FIG. 1) that generates a classification representative vector having a number,
A refined exclusion vector designating unit (for example, a refined exclusion vector designating unit 105 in FIG. 1) for designating a classified representative vector for which no refining process is performed, and a similarity between the document feature vector and the classified representative vector. The document data allocating unit (for example, the document data allocating unit 106 in FIG. 1) that allocates document data to one of the classified representative vectors, and the classification other than the classified representative vector specified by the refinement exclusion vector specifying unit. For the representative vector, a classification representative vector is recalculated based on the document feature vector allocated by the document data allocation unit, and the document data allocation and the recalculation of the classification representative vector are repeated until a specific criterion is satisfied. A representative vector refining unit (for example, the classified representative vector refining unit 107 in FIG. 1); Classification result storage unit for storing a class result (e.g., the classification result storing unit 108 in FIG. 1) and having a.

According to the second aspect of the present invention, the refining process is bypassed for some of the specified initial classification representative feature vectors. Can be obtained.

According to a third aspect of the present invention, in the document classification apparatus according to the second aspect, the classification representative vector generation unit generates some classification representative vectors based on information designated by a user. Features.

According to the third aspect of the present invention, in addition to the features of the document classification apparatus according to the second aspect, the user specifies the initial classification representative vector that bypasses the refining process, thereby reflecting the user's classification intention. A classified result can be obtained.

According to a fourth aspect of the present invention, in the document classification apparatus according to the third aspect, the refinement exclusion vector designation section is generated by information designated by a user in the classification representative vector generation section. The method is characterized in that only the classified representative vectors obtained are classified representative vectors for which refining is not performed.

According to the fourth aspect of the present invention, in addition to the features of the document classifying apparatus according to the third aspect, the classification representative feature vector designated by the user is not subjected to refining processing, so that the user's classification Can be obtained.

According to a fifth aspect of the present invention, in the document classification apparatus of the fourth aspect, in the classification representative vector generation unit, the information specified by the user is a logical combination of the document data and the words existing in the document data. It is a formula.

According to a fifth aspect of the present invention, in addition to the features of the document classification apparatus of the fourth aspect, an initial classification representative feature vector is generated from a logical expression of a document specified by a user and a word included in the document. This makes it possible to realize a document classification device that allows a user to easily describe a classification intention.

According to a sixth aspect of the present invention, there is provided the document classification apparatus according to the fifth aspect, wherein a classification representative vector generation unit stores information for generating a classification representative vector specified by the classification representative vector generation unit. An information storage unit (for example, the classification representative vector generation information storage unit 201 in FIG. 8), and a classification representative vector generation information reading unit that reads information stored in the classification representative vector generation information storage unit (for example, the classification representative in FIG. 8) A vector generation information reading unit 202) and a refined exclusion vector information storage unit (for example, a refined exclusion vector in FIG. 8) that stores information on the classified representative vector not subjected to the refining process specified in the classified representative vector generation unit Information storage unit 203) and information stored in the refining exclusion vector information storage unit. Refining of exclusion vector information reading part for reading (e.g., refining of exclusion vector information reading section 204 in FIG. 8), characterized in that it further comprises a.

According to a sixth aspect of the present invention, in addition to the features of the document classification apparatus according to the fifth aspect, a designated initial classified representative feature vector and a classified representative feature vector not subjected to refining processing are stored. By providing a mechanism to read out information later, it is possible to obtain a new classification result by slightly modifying the setting of the document classification performed in advance, and to maintain the same classification even when the classification target document set changes. You can get the classification result in the setting.

According to a seventh aspect of the present invention, there is provided a document classification apparatus for classifying a document according to the content of the document.
A document input unit for inputting document data (for example, the document input unit 101 in FIG. 11), a document analysis unit for analyzing words of the document data (for example, the document analysis unit 102 in FIG. 11), and A document feature vector generation unit (for example, the document feature vector generation unit 103 in FIG. 11) that calculates a document feature vector for each document from a document analysis result, and a logic of a plurality of information primitives specified as some classification representative vectors A classification representative vector that expands the expression into a combination expression based on a logical sum in which each element is a logical product of information primitives, and generates a classification representative vector having the same number of dimensions as the document feature vector from each logical product of the generated information primitives The designation generation unit (for example, the classification representative vector designation generation unit 30 in FIG. 11)
1), a classifying representative vector information storage unit for storing information on the designated logical formula and classifying representative vectors generated therefrom, and a classifying representative vector using the information generated by the document feature vector generating unit. Based on the automatically generated classification representative vector generation unit (for example, the classification representative vector automatic generation unit 303 in FIG. 11) and the similarity between the document feature vector and the classification representative vector,
Generating a classification result based on information stored in the document classification unit (for example, the document classification unit 304 in FIG. 11) for classifying the document data and the classification representative vector information storage unit;
A document classification device comprising a classification result storage unit (for example, a classification result storage unit 305 in FIG. 11) for storing.

According to the present invention, the logical expression of the designated information primitive is developed into a logical sum expression, and an initial classification representative vector is generated from each of the logical products of the information primitives constituting the logical sum expression. Thus, a document classification result suitable for the meaning of the described logical expression can be generated.

According to an eighth aspect of the present invention, in the document classification apparatus according to the seventh aspect, a classification representative vector information reading unit (for example, the classification representative vector in FIG. 11) for reading the information about the stored classification representative vector. (Information reading unit 305).

According to an eighth aspect of the present invention, in addition to the features of the document classifying apparatus according to the seventh aspect, information relating to a logical expression of an information primitive stored in advance and information relating to a generated document representative vector is reused. As a result, it is possible to realize a document classification device capable of classifying documents based on the same classification standard even for different document sets.

According to a ninth aspect of the present invention, in the document classification apparatus according to the eighth aspect, the classification representative vector generation unit calculates an arithmetic average of each information primitive included in a logical product of the generated information primitives. A classification representative vector is generated.

According to the ninth aspect of the present invention, in addition to the features of the document classifying apparatus according to the eighth aspect, in the logical product of the generated information primitives, a feature vector appropriately representing each information primitive constituting the information primitives is provided. By generating a document representative vector from the arithmetic mean of, the AND can be quantified appropriately, and thereby a document classification result suitable for the meaning of the logical expression can be generated.

[0032] The invention described in claim 10 is the ninth invention.
Wherein the information primitive specified by the classification representative vector generation unit is a document and a word included in the document.

According to the tenth aspect of the present invention, in addition to the features of the document classifying apparatus according to the ninth aspect, a document in which a logical expression can be easily generated by using a document and words included in the document as information primitives. A classification device can be realized.

The invention described in claim 11 is the first invention.
0, wherein the logical expression of the information primitive specified by the classification representative vector generation unit is specified by a user.

According to the eleventh aspect, in the tenth aspect,
In addition to the features of the document classification device described in (1), since the user can specify a logical expression, a document classification result that clarifies the user's intention can be generated.

According to a twelfth aspect of the present invention, there is provided a document classification method for setting a plurality of initial classification representative feature vectors and classifying a document based on the similarity between the classification representative feature vector and the document feature vector. Has a refining processing step of dynamically changing a classification representative feature vector to improve classification accuracy and repetitively assigning a document, and the refining processing step includes a part or all of the classification representative feature. Refining is not performed on vectors.

According to a thirteenth aspect of the present invention, in the document classification method for classifying documents according to the contents of the documents, a document input step of inputting document data and a document analysis step of analyzing words of the document data. A document feature vector generation step of calculating a document feature vector for the document based on the document analysis result of the document analysis step; a classification representative vector generation step of generating a classification representative vector having the same number of dimensions as the document feature vector;
Refining exclusion vector designating step of designating a classified representative vector for which refining processing is not performed; and assigning document data to one of the classified representative vectors based on the similarity between the document feature vector and the classified representative vector. For the classification representative vector other than the classification representative vector specified in the document data allocation step and the refinement exclusion vector specification step, the classification representative vector is re-created based on the document feature vector allocated in the document data allocation step. A classification / representation vector refining step of repeating calculation and recalculation of the document data allocation and classification / representation vector until a specific criterion is satisfied; and a classification result storage step of storing a classification result.

The invention described in claim 14 is the first invention.
3. The document classification method according to 3, wherein in the classifying representative vector generating step, some classifying representative vectors are generated based on information specified by a user.

The invention described in claim 15 is the first invention.
4. In the document classification method according to 4, the refined exclusion vector designating step includes, in the classified representative vector generating step, only the classified representative vector generated by the information specified by the user in the classified representative vector, which is not refined. It is characterized by doing.

The invention described in claim 16 is the first invention.
In the document classification method of the fifth aspect, in the classification representative vector generation step, the information specified by the user is a logical expression of the document data and words existing in the document data.

The invention described in claim 17 is the first invention.
6. In the document classification method described in 6, the classification representative vector generation information storing step for storing information for generating the classification representative vector specified in the classification representative vector generation step, and the classification representative vector generation information storing step Classifying representative vector generation information reading step for reading the classified information, a refining exclusion vector information storing step for storing information related to the classifying representative vector that is not subjected to the refining process specified in the classification representative vector generating step, A refining exclusion vector information reading step for reading the information stored in the exclusion vector information storage step.

According to the invention described in claim 18, in a document classification method for classifying documents according to the contents of the document, a document input step of inputting document data, and a document analysis step of analyzing words of the document data. A document feature vector generation step of calculating a document feature vector for each document from the document analysis result by the document analysis step; and a plurality of information primitives, each of which has a logical expression of a plurality of designated information primitives as elements of the information primitive. A classification representative vector specification generating step of generating a classification representative vector having the same number of dimensions as the document feature vector from each of the logical products of the generated information primitives, Stores information on logical expressions and the classification representative vectors generated from them A classifying representative vector information storing step, a classifying representative vector automatic generating step of automatically generating a classifying representative vector using the information generated in the document feature vector generating step, A document classification step of classifying the document data based on the similarity; and a classification result storage step of generating and storing a classification result based on the information stored in the classification representative vector information storage step. It is characterized by.

The invention described in claim 19 is the first invention.
8. The document classification method according to item 8, further comprising: a classification representative vector information reading step of reading information on the stored classification representative vector.

The invention described in claim 20 is the first invention.
9. The document classification method according to 9, wherein in the classifying representative vector generating step, a classifying representative vector is generated as an arithmetic average of each information primitive included in a logical product of the generated information primitives.

The invention described in claim 21 is the second invention.
0, wherein the information primitive specified in the classification representative vector generation step is a document and a word included in the document.

The invention described in claim 22 is the second invention.
1. The document classification method according to 1, wherein a logical expression of the information primitive specified in the classification representative vector generation step is specified by a user.

The document classification method according to the twelfth to twenty-second aspects is a document classification method suitable for the document classification apparatus according to the first to eleventh aspects.

[0048]

Next, embodiments of the present invention will be described with reference to the drawings.

In the present invention, if a set of one or more sentences described in a natural language is to be classified, it is called a document. At the end of one document, it is assumed that a document end symbol that can be identified is placed.

To give a concrete example, a patent publication and a specific newspaper article are also documents, and even if a claim or a specific sentence is extracted therefrom, it is regarded as a document. (First Embodiment) FIG. 1 is a configuration diagram of a document classification device for explaining a first embodiment.

In the document input unit 101, a keyboard, an OC
A document or a document group is obtained via an auxiliary storage device such as an R device, a hard disk, or a network, and document data is input.

The document analysis unit 102 performs a natural language analysis on each of the input documents, and extracts words and their parts of speech. In addition, the order in which words appear in the document,
The meta information of the document such as the creator and the date of creation of the document is also extracted. After the extraction, a unique word ID is assigned to the word that has appeared in the document group, and the number of word appearances in the document and for the document group is counted.

The document feature vector generation unit 103 generates a document feature vector based on document analysis data such as a word, a word ID, the number of word appearances, and part of speech information generated by the document analysis unit 102.

The document feature vector will be described with reference to FIG. FIG. 2 shows a case where a line is stored in a document (doci [i = 1 to
0]), and columns as words (wordj [j = 1 to 10]). At the intersection of the document and the word (wordji),
The appearance frequency of the word in the document is shown.
For example, a word (word1) in a document (doc1)
Is 1, and the appearance frequency of the word (word1) in the document (doc3) is 5.

As described above, the document-word frequency matrix data is generated such that the appearance frequency of the word (wordj) in the document (doci) becomes the (i, j) element, and each column vector of the document-word frequency matrix is generated. Is a document feature vector.

In the description of the present invention, the case where the document data shown in FIG. 2 is classified into three document sets will be considered.

Further, it is possible to improve the classification accuracy by simultaneously performing normalization processing on the document-word frequency matrix and using a document feature vector in consideration of the influence of the length between documents.

Further, by applying a multidimensional scaling method such as factor analysis, quantification type III, and singular value decomposition to the document-word matrix, a document feature vector in consideration of the ambiguity and synonymity of a word is obtained. It can also be used.

As an example, assuming that the effective singular value dimension is 10,
FIG. 3 shows a document feature vector obtained as a result of applying the singular value decomposition to the document-word frequency matrix of FIG.

By applying singular value decomposition to the document-word frequency matrix, words are also placed in the same feature space as the document. FIG. 4 shows these word feature vectors.

In FIGS. 3 and 4, dimk [k
= 1 to 6] indicate each feature dimension. In the examples shown in FIGS. 3 and 4, SVDPACKC [http: // www.
w. cs. utk. edu / berry /] to determine the singular value decomposition.

The classification representative vector generation unit 104 generates a classification representative vector having the same number of dimensions as the document feature vector as preprocessing for performing document classification using a statistical method.

As a method of generating the classification representative vector, for example, a fixed rule such as selecting at equal intervals from the document feature vector, selecting at random from the document feature vector, selecting from the beginning of the document feature vector, and the like. Some document feature vectors are selected based on the above, and those document feature vectors are used as classification representative vectors.

Further, by generating a classification representative vector from information of a document or word specified by the user, a classification representative vector that clearly reflects the user's classification intention can be generated.

The information specified by the user may be any information as long as it relates to a document or a word included in the document. However, each piece of information needs to have a specific correspondence with a document or a word in the document.

Further, the user can designate a classification representative vector by a logical expression of a document and words existing in the document. Thus, the user can easily describe the classification intention. It should be noted that a specific specification method for specifying the classification representative vector by a logical expression does not matter.

Here, a case will be described in which two classified representative vectors are generated from information designated by the user, and one classified representative vector is automatically generated.

For the sake of simplicity, it is assumed that the user designates a classification representative vector by a logical product of words or a document. When a word set is specified, the classified representative vector is set as the arithmetic average of the feature vectors of the included words, and when a document is specified, the corresponding document feature vector is set as the classified representative vector.

As the classification representative vector 1, the word: wor
The figure shows an initial document representative vector in a case where d1 and the word: word3 are specified, the document: doc5 is specified by the user as the classification representative vector 2, and the document feature vector of the document: doc7 is automatically selected as the classification representative vector 3. It is shown in FIG.

The refined exclusion vector specifying unit 105 specifies whether or not to perform the refining process on the classified representative vector generated by the classified representative vector generating unit 104 based on a specific reference.

The criterion for determining whether or not to perform the refining process can be automatically determined from information related to documents and words, daily tendency of the user, and the like. here,
Any method can be used as long as the determination can be made for all the classification target vectors.

The classification representative vector generation unit 104
By using the criterion that only the classification representative vector specified by the user is the refining exclusion vector,
The classification representative vector expressing the classification intention specified by the user can be prevented from changing due to refinement.

The document data allocating unit 106 calculates the similarity between each document feature vector and the classified representative vector, and allocates each document data to the classified representative vector having the highest similarity. Here, for calculating the similarity, an inner product between vectors, a cosine, a Euclidean distance, or the like can be used, but any method is applicable here.

In the classification representative vector refining section 107, the document data allocating section 106 allocates all the classification representative vectors for which refining exclusion has not been specified in the refining exclusion vector specifying section 105. The classification representative vector is recalculated based on the feature vector of the document data, and the document data is re-calculated until a specific criterion is satisfied (for example, until a refining process is performed a predetermined number of times, a classification error becomes a predetermined value or less, etc.) The assignment and the recalculation of the classification representative vector are repeatedly performed.

As the recalculation method, the k-means method (a method of recalculating each time one document to be classified is allocated) used in non-hierarchical clustering, or a method of re-calculation is used.
A method (a method of recalculating after all the documents to be classified are assigned) can be used, but the method is not limited here.

The classification result storage unit 108 stores the generated classification result in an appropriate format.

Next, a case will be described in which some classified representative vectors are used as refined exclusion vectors.

First, among the three classified representative feature vectors shown in FIG. 5, the classified representative vector designated by the user, that is, the classified representative vector 1 and the classified representative vector 2 are defined as refined exclusion vectors. When the cosine is used as the similarity measure of the document data allocating unit 106 and the k-means method is used as the method of the document representative vector refining unit, and the number of repetitions is set to 5 as the refining stop criterion, the classification representative vector shown in FIG. become that way.

As is clear from FIG. 6, only the classification representative vector 3 changes in value. FIG. 7 shows the classification result.

In FIG. 7, the value in parentheses of each belonging document is the degree of similarity with the belonging classification representative vector.

According to the classification result, the classified representative vector 2 excluding the refinement forms a document set centering on the designated document: doc5. Although the document: doc7 is generated based on: doc7, the document: doc7 belongs to the document set of the classification representative vector 2 instead of this document set.

Therefore, if the classification representative vector 3 is specified by the user, the generated document set does not center on the document: doc7.
It can be predicted that the result did not match the user's intention. (Second Embodiment) FIG. 8 is a configuration diagram of a document classification device for explaining a second embodiment.

The same components as those in FIG. 1 are denoted by the same reference numerals.

The classification representative vector generation information storage unit 201 stores information used when the classification representative vector generation unit 104 generates a classification representative vector, such as rules used for generating the classification representative vector, documents and words specified by the user. Information and the like and the classification representative vector itself are stored.

Classification representative vector generation information reading section 20
In step 2, when the classified representative vector generation unit 104 generates a classified representative vector, the information stored in the classified representative vector generation information storage unit 201 is read.

The refined exclusion vector information storage unit 203 stores information on the exclusion classification representative vector used in the refined exclusion vector designation unit 105.

Refining exclusion vector information reading section 204
Then, when the exclusion classification representative vector is designated by the refinement exclusion vector designation unit 105, the information on the exclusion classification representative vector stored in the refinement exclusion vector information storage unit 203 is read.

With this configuration, it is possible to easily generate a classification representative vector by changing various settings, or to change the refined exclusion classification representative vector to obtain a classification result.

Next, among the three classification representative feature vectors shown in FIG. 5, classification representative vector 1 and classification representative vector 2
Next, a case will be described in which after refining is performed by classifying documents using the refinement exclusion vector, all the three classification representative feature vectors shown in FIG. 5 are refined.

First, the classification / representation vector generation unit 104 stores the value of the generated classification / representation vector. At this time, the storage method does not matter. Further, the classification representative vector 1 and the classification representative vector 2
Is also stored as a refining exclusion vector.

Then, by performing document classification with this setting, the classification result shown in FIG. 7 is obtained.

Next, the same processing is performed again on the same document data. At this time, the document tone vector and the word feature vector are stored in an appropriate format, and the re-execution of the singular value decomposition can be bypassed by reading them in the case of reprocessing.

The classification representative vector generation unit 104 reads the value of the classification representative vector stored in advance. Also,
The refining exclusion vector specifying unit 105 reads information on the smelting exclusion vector, and in this example, changes the setting to perform the smelting process for all the classified representative vectors.

FIG. 9 shows the values of the classification representative vectors and FIG. 1 shows the classification results when the document classification is executed with this setting.
0 is shown.

As described above, the classification representative vector generation information storage unit 201 and the classification representative vector generation information reading unit 20
2. By adding the refined exclusion vector information storage unit 203 and the refined exclusion vector information reading unit 204, various classification results can be easily obtained. (Third Embodiment) FIG. 11 is a configuration diagram of a document classification device for explaining a third embodiment. 1 and 2 are denoted by the same reference numerals.

In the classification / representative vector designation generation unit 301,
The logical expression of the specified information primitive is developed into a logical OR combination of the primitive logical product composed only of the logical product of the information primitives.

For example, each of A, B, C, D, E, and F is defined as an information primitive, and these logical expressions are expressed as: ((A
Given * B) + (C * D) + E) * F, this logical expression can be expressed as (A * B * F) + (C * D * F) + (E * F)
Expand like

Here, * indicates a logical product and + indicates a logical sum.

Then, a classification representative vector is generated from each of these primitive logical products. Here, the method of generating the classification representative vector from the primitive logical product does not matter. However, the generated classification representative vector must have the same number of dimensions as the document feature vector.

If each of the information primitives constituting the primitive logical product can be represented by a vector having the same dimension as the document feature vector, a classification representative vector may be generated by arithmetic mean of the vectors. That is, the vectors expressing the information primitives of A, B, C, D, E, and F are Va, Vb, Vc, Vd, Ve, V
f, the generated classification representative vector is (Va + Vb + Vf / 3, (Vc + Vd + Vf) /
3, (ve + Vf) / 2.

A document and words contained in the document can be used as information primitives. That is, vectors Va, Vb,
As Vc, Vd, Ve, and Vf, a document feature vector and a word feature vector as shown in FIGS. 3 and 4 can be used.

Further, the logical formula of the information primitive can be specified by the user. However, a specific designation method does not matter here.

In the classification representative vector information storage unit 302,
Information on the designated logical expression used in the classification representative vector designation generation unit 301 and the generated document representative vector is stored in an appropriate format.

In the classification representative vector automatic generation unit 303,
Generate some classification representative vectors automatically.

As an automatic generation method, for example, based on certain rules such as selecting at equal intervals from a document feature vector, selecting at random from a document feature vector, selecting from the beginning of a document feature vector, and the like, Are selected, and those document feature vectors are set as classification representative vectors.

However, the generated document representative vector must have the same number of dimensions as the document feature vector.

The document classifying unit 304 classifies documents using the generated classification representative vector and the document feature vector.

In this case, the k-means method of the non-hierarchical clustering method,
Method can be used.

The classification result storage unit 305 stores the generated document classification result in an appropriate format. In the classification representative vector information reading unit 306, the classification representative vector information storage unit 302 reads the information on the logical expression used in the classification representative vector designation generation unit 301 and the generated document representative vector before that. With reference to the read information, the classification / representative vector specification generation unit 301 newly specifies a classification / representative feature vector.

Thus, a new logical expression can be created with reference to various logical expressions specified in advance.

As an example, when the document set shown in FIG. 2 is given and the document feature vector and the word feature vector are given in FIGS. A description will be given of a document classification when an expression and automatic generation of one document set are designated.

First, when the user enters “word1 * word”
Assuming that the logical expressions “3” and “(doc2 + doc12) * word5” are specified, w
ord1 * word3, do as classification representative vector 2
doc as c2 * word5 and classification representative vector 3
It is assumed that a document is automatically (randomly) selected as 12 * word5 and the classification representative vector 4, and doc17 is selected.

In this case, if the AND primitive is to take the arithmetic mean of the vectors representing the information primitives, the classification representative vector 1 is (word1 word feature vector + word3 word feature vector) / 2, the classification representative vector 2 is (word5 word feature vector +
The document feature vector of doc2) / 2, the classification representative vector 3 is (word feature vector of word5 + document feature vector of doc12) / 2, and the classification representative vector 4 is (doc).
17 document feature vectors), and the result is shown in FIG. FIG. 13 shows the result of performing document classification using the k-means method using the cosine measure, with the classification representative vector specified by the user being the refinement exclusion vector. (Document Classification Method) Next, a document classification method will be described.

FIG. 1, FIG. 8 and FIG.
FIG. 12 is a configuration diagram of the document classification device, which is shown in FIGS.
Describes the contents of the processing related to the document classification device. Therefore, a document classification method can be shown by performing the processing of each unit described in FIG. 1, FIG. 8, and FIG. 11 in chronological order so as to achieve the initial purpose. For example, in FIG. 1, the processing is performed by the document input unit 101, then the processing is performed by the document analysis unit 102, and then the processing of the document feature vector generation unit 103, the processing of the classification representative vector generation unit 104, and the refining exclusion Vector designator 105
Processing, processing of the document data allocation unit 106, processing of the classification representative vector refining unit 107, classification result storage unit 108
By performing the above processing, it is possible to indicate a document classification method.

Therefore, as for the method of classifying documents, if the flow of processing of each unit described in FIGS. 1, 8 and 11 can be understood, the method of classifying documents can be understood.

The flow of the processing of each unit shown in FIGS. 1, 8 and 11 will be described below. (Processing flow of each component) Document input unit 10 in FIG.
1. Document analysis unit 102, document feature vector generation unit 10
3. The process flow of the classification representative vector generation unit 104, the refinement exclusion vector designation unit 105, the document data allocation unit 106, the classification representative vector refinement unit 107, and the classification result storage unit 108 will be described with reference to FIGS. I do. The processing of the document input unit 101 will be described with reference to FIG.

A target ID is assigned to the target document.
This is performed for all target documents. The processing of the document analysis unit 102 will be described with reference to FIG.

The word information (word notation, part of speech) is extracted from the target document.

Next, if the extracted word is the first extracted word, the word information is registered as independent word information, and an ID is assigned to the registered independent word information. However, if the extracted words are already extracted words,
Extract the corresponding independent word information. Also, word information that can be replaced by reference to independent word information is replaced.

After performing this for all the extracted word information, this is performed for all the target documents. The processing of the document feature vector generation unit 103 will be described with reference to FIG.

The frequency of occurrence of all independent words in a document is counted and used as a document feature vector. This is performed for all target documents.

Next, when normalizing the document feature vector, the document feature vector is normalized.
This is performed for all document feature vectors.

Further, when performing a linear transformation, a linear transformation matrix is applied to the document feature vector. This is performed for all document feature vectors. The processing of the classification representative vector generation unit 104 will be described with reference to FIG.

The document representative vectors are sequentially generated until all the document representative vectors are generated.

The process of designating the document representative vector by the user may precede the above process. If the user specifies the logical expression of the document data and the word existing in the document data, the above-described processing (document representative vector generation processing) is performed to obtain the logical expression of the document ID and the word ID generated by the user. I do. The process of the refinement exclusion vector designation unit 105 will be described with reference to FIG.

First, refinement exclusion information is generated.

Next, if information for excluding refining is given, the flag for performing refining is lowered. If information for excluding refining is not given, a flag for performing refining is set.

This is performed for all target documents.

The process of treating the document representative vector specified by the user as a refining exclusion vector is assumed, and the process of adding information for excluding the refining of the document representative vector specified by the user is added to the above process. You may. The processing of the document data allocation unit 106 will be described with reference to FIG.

The similarity with all the document representative vectors is calculated, and the document feature vector is assigned to the document representative vector having the highest similarity.

This is performed for all document feature vectors.

Thus, the documents are classified. The process of the classification representative vector refining unit 107 will be described with reference to FIG.

The following process is repeated until the refining stop criterion is satisfied.

If a flag for refining is set for the document representative vector, the classification representative vector is recalculated based on the document feature vector belonging to the classification representative vector.

This is performed for all document representative vectors.

Thereafter, the processing in FIG. 19 is performed. The processing of the classification result storage unit 108 will be described with reference to FIG.

For the document representative vector, information on the document feature vector to which it belongs is stored. This is performed for all the document representative vectors.

The classification representative vector generation information storage unit 201 and the classification representative vector generation information reading unit 20 in FIG.
2. The refining exclusion vector information storage unit 203 and the smelting exclusion vector information reading unit 204 store or read a vector, and the storage or reading itself is different from general storage or reading processing. Since there is no description, the description is omitted.

The document input unit 101 and the document analysis unit 10
2. Document feature vector generation unit 103, classification representative vector generation unit 104, refinement exclusion vector designation unit 105, document data allocation unit 106, classification representative vector refinement unit 1
07, regarding the processing flow of the classification result storage unit 108,
The description is omitted because it is the same as the above description.

FIGS. 22 to 22 show the flow of processing in the classification / representation vector designation generation unit 301, classification / representation vector information storage unit 302, classification / representation vector automatic generation unit 303, document classification unit 304, and classification result storage unit 305 in FIG. 2
6 will be described. The processing flow of the document input unit 101, the document analysis unit 102, and the document feature vector generation unit 103 is the same as that described with reference to FIG. The processing of the classification / representative vector designation generation unit 301 will be described with reference to FIG.

First, a logical expression of a designated information primitive is obtained, and an ID is assigned to each logical expression.

Next, with respect to the acquired information primitive logical expression, the logical expression is developed into a logical OR of elements composed only of logical ANDs according to a combination rule, an exchange rule, an idempotent rule, and an absorption rule.

Next, a document representative vector is generated for an element consisting only of a logical product based on information of each information primitive forming the logical product.

At this time, if the generated document representative vector matches the previously generated document representative vector, the ID of the previously generated document representative vector is obtained. However, if the generated document representative vector does not match the previously generated document representative vector, an ID is assigned to the generated document representative vector.

Then, the ID of the document representative vector is stacked.

This is performed for all the acquired information primitive logical expressions. The processing of the classification representative vector information storage unit 302 will be described with reference to FIG.

First, the ID assigned to the logical expression of the information primitive is stored. Next, the ID of the stacked document representative vector is stored.

This is performed for the logical expressions of all information primitives. The processing of the classification representative vector automatic generation unit 303 will be described with reference to FIG.

First, information on the document feature vector is obtained.

Until all document representative vectors are generated,
A document representative vector is generated, and a document representative vector ID is assigned to the document representative vector. The processing of the document classification unit 304 will be described with reference to FIG.

The similarity between the document feature vector and all the document representative vectors is calculated. The document feature vector is assigned to the document representative vector having the highest similarity.

This is performed for all document feature vectors.

Next, the following processing is repeated until the refining stop criterion is satisfied.

The classification representative vector is recalculated based on the document feature vector belonging to the document representative vector, and the processing in FIG. 19 is performed.

This is performed for all document feature vectors. The processing of the class result storage unit 305 will be described with reference to FIG.

First, the information stored in the document representative vector information storage section is read.

Next, a document classification identifier is stored for the information primitive logical expression. Next, for all the document representative vectors, if the ID of the document representative vector is the ID of the stacked document representative vector, information on the document feature vector belonging to the document representative vector is stored.

This is performed for all information primitive logical expressions.

If the document representative vector is not stacked in any of the information primitive logical expressions, the document identifier is stored, and information on the document feature vector belonging to the document representative vector is stored.

FIGS. 27 and 28 show an example of the flow of another process of the classification / representative vector designation generation unit 301.

[0161]

According to the present invention as described above, the following various effects can be obtained. According to the first aspect of the present invention, the refining process is bypassed for some or all of the initial classification representative feature vectors, so that the partial or all of the initial classification representative vectors have a viewpoint expressed by them. A classification result as clearly shown can be obtained.

According to the second aspect of the present invention, the refining process is bypassed for some of the specified initial classification representative feature vectors. Can be obtained.

According to the third aspect of the invention, in addition to the features of the document classification apparatus according to the second aspect, the user specifies the initial classification representative vector that bypasses the refining process, thereby reflecting the user's classification intention. A classified result can be obtained.

According to the fourth aspect of the present invention, in addition to the features of the document classifying apparatus according to the third aspect, the classification representative feature vector designated by the user is not subjected to the refining process, so that the user's classification Can be obtained.

According to a fifth aspect of the present invention, in addition to the features of the document classifying apparatus according to the fourth aspect, an initial classification representative feature vector is generated from a document specified by a user and a logical expression of a word included in the document. This makes it possible to realize a document classification device that allows a user to easily describe a classification intention.

According to the invention described in claim 6, in addition to the features of the document classification device described in claim 5, a designated initial classification representative feature vector and a classified representative feature vector not subjected to refining processing are stored. By providing a mechanism to read out information later, it is possible to obtain a new classification result by slightly modifying the setting of the document classification performed in advance, and to maintain the same classification even when the classification target document set changes. You can get the classification result in the setting.

According to the present invention, the logical expression of the specified information primitive is expanded into a logical sum expression, and an initial classification representative vector is generated from each of the logical products of the information primitives constituting the logical sum expression. Thus, a document classification result suitable for the meaning of the described logical expression can be generated.

According to the eighth aspect of the present invention, in addition to the features of the document classifying apparatus according to the seventh aspect, information relating to a logical expression of an information primitive stored in advance and information relating to a generated document representative vector is reused. As a result, it is possible to realize a document classification device capable of classifying documents based on the same classification standard even for different document sets.

According to the ninth aspect of the present invention, in addition to the features of the document classifying apparatus according to the eighth aspect, in the logical product of the generated information primitives, a feature vector appropriately representing each information primitive constituting the information primitives is provided. By generating a document representative vector from the arithmetic mean of, the AND can be quantified appropriately, and thereby a document classification result suitable for the meaning of the logical expression can be generated.

According to the tenth aspect of the present invention, in addition to the features of the document classifying apparatus according to the ninth aspect, a document in which a logical expression can be easily generated by using a document and words included in the document as information primitives. A classification device can be realized.

According to the eleventh aspect, in the tenth aspect,
In addition to the features of the document classification device described in (1), since the user can specify a logical expression, a document classification result that clarifies the user's intention can be generated.

According to the document classification method of the present invention, a document classification method suitable for the document classification apparatus of the present invention can be provided.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a document classification device for describing a first embodiment.

FIG. 2 is a diagram for explaining a document-word frequency matrix.

FIG. 3 is a diagram for explaining a document feature vector.

FIG. 4 is a diagram for explaining a word feature vector.

FIG. 5 is a diagram (part 1) for describing an initial document representative feature vector.

FIG. 6 is a diagram (part 1) for describing a document representative feature vector after refining.

FIG. 7 is a diagram (part 1) for explaining a classification result;

FIG. 8 is a configuration diagram of a document classification device for explaining a second embodiment.

FIG. 9 is a diagram (part 2) for explaining a document representative feature vector after refining.

FIG. 10 is a diagram (part 2) for explaining a classification result;

FIG. 11 is a configuration diagram of a document classification device for describing a third embodiment.

FIG. 12 is a diagram (part 2) for explaining an initial document representative feature vector.

FIG. 13 is a diagram (part 3) for explaining the classification result;

FIG. 14 is a diagram for explaining a processing procedure of the document input unit 101.

FIG. 15 is a diagram for explaining a processing procedure of the document analysis unit 102;

FIG. 16 is a diagram illustrating a processing procedure of a document feature vector generation unit 103.

FIG. 17 is a diagram for describing a processing procedure of a classification representative vector generation unit 104.

FIG. 18 is a diagram for describing a processing procedure of a refining exclusion vector specifying unit 105.

FIG. 19 is a diagram illustrating a processing procedure of a document data allocating unit 106.

FIG. 20 is a diagram illustrating a processing procedure of a classification representative vector refining unit 107.

FIG. 21 is a diagram for describing a processing procedure of a classification result storage unit.

FIG. 22 is a diagram (part 1) for describing a processing procedure of the classification / representative vector designation generation unit 301;

FIG. 23 is a diagram for describing a processing procedure of a classification representative vector information storage unit 302.

FIG. 24 is a diagram for describing a processing procedure of a classification representative vector automatic generation unit 303.

FIG. 25 is a diagram illustrating a processing procedure of a document classification unit 304.

FIG. 26 is a diagram for explaining a processing procedure of a similar result storage unit 305.

FIG. 27 is a diagram (part 2) for explaining the processing procedure of the classification / representative vector designation generation unit 301;

FIG. 28 is a diagram (part 3) for describing the processing procedure of the classification / representative vector designation generation unit 301;

[Explanation of symbols]

 Reference Signs List 101 Document input unit 102 Document analysis unit 103 Document feature vector generation unit 104 Classified representative vector generation unit 105 Refined exclusion vector designating unit 106 Document data allocation unit 107 Classified representative vector refinement unit 108, 305 Classification result storage unit 201 Classified representative vector Generation information storage unit 202 Classified representative vector generation information reading unit 203 Refined exclusion vector information storage unit 204 Refined exclusion vector information reading unit 301 Classified representative vector designation generation unit 302 Classified representative vector information storage unit 303 Classified representative vector automatic generation unit 304 Document classifier

Claims (22)

[Claims] 1. A document classification apparatus for setting a plurality of initial classification representative feature vectors and classifying a document based on the similarity between the classification representative feature vector and a document feature vector. And a refining processing unit for repetitively assigning documents by changing the classification representative feature vector. The refining processing unit performs a refining process on a part or all of the classification representative feature vectors. Document classification device characterized by the absence. 2. A document classification device for classifying documents according to the contents of the documents, a document input unit for inputting document data, a document analysis unit for analyzing words of the document data, and a document analysis unit of the document analysis unit. A document feature vector generation unit that calculates a document feature vector for the document based on the result, a classification representative vector generation unit that generates a classification representative vector having the same number of dimensions as the document feature vector, and a classification representative that does not perform refining processing A refining exclusion vector designating unit that designates a vector; a document data assigning unit that assigns document data to one of the classification representative vectors based on the similarity between the document feature vector and the classification representative vector; Classification representative vectors other than the classification representative vector specified by the A classification representative vector recalculating unit that recalculates the classification representative vector based on the document feature vector allocated by the application unit and repeats the document data allocation and the recalculation of the classification representative vector until a specific criterion is satisfied; A document classification device, comprising: a classification result storage unit that stores a classification result. 3. The document classification apparatus according to claim 2, wherein the classification representative vector generation unit generates some classification representative vectors based on information specified by a user. 4. The document classification apparatus according to claim 3, wherein the refining exclusion vector designating unit refines only the classified representative vector generated by the information designated by the user in the classified representative vector generating unit. A classifying representative vector which does not perform the classifying. 5. The document classification device according to claim 4, wherein in the classification representative vector generation unit, the information specified by the user is a logical expression of document data and a word existing in the document data. Document classifier. 6. The document classification device according to claim 5, wherein the classification representative vector generation information storage unit stores information for generating a classification representative vector specified by the classification representative vector generation unit; A classification representative vector generation information reading unit that reads information stored in a vector generation information storage unit; and a refining exclusion vector information that stores information related to a classification representative vector that is not subjected to the refining process specified in the classification representative vector generation unit. A document classification device further comprising: a storage unit; and a refined exclusion vector information reading unit that reads information stored in the refined exclusion vector information storage unit. 7. A document classification device for classifying documents according to the contents of the documents, a document input unit for inputting document data, a document analysis unit for analyzing words of the document data, and a document analysis by the document analysis unit. A document feature vector generation unit that calculates a document feature vector for each document from the result, and a logical expression of a plurality of information primitives specified as some classification representative vectors, and a logical sum in which each element is a logical product of the information primitives And a classification representative vector designation generation unit that generates a classification representative vector having the same number of dimensions as the document feature vector from each of the logical product of the generated information primitives, A classification representative vector information storage unit for storing information about classification representative vectors, A classification representative vector automatic generation unit that automatically generates a classification representative vector using information generated by a generation unit; and a document that classifies document data based on a similarity between the document feature vector and the classification representative vector. A classification unit, a classification result storage unit that generates a classification result based on the information stored in the classification representative vector information storage unit, and stores the classification result;
Document classification apparatus characterized by having. 8. The document classification device according to claim 7, further comprising: a classification representative vector information reading unit that reads information on the stored classification representative vector. 9. The document classification apparatus according to claim 8, wherein the classification representative vector generation unit generates a classification representative vector as an arithmetic average of each information primitive included in a logical product of the generated information primitives. Document classification apparatus characterized by the above-mentioned. 10. The document classification device according to claim 9, wherein the information primitive specified by the classification representative vector generation unit is a document and a word included in the document. 11. The document classification device according to claim 10, wherein a logical expression of the information primitive specified by the classification representative vector generation unit is specified by a user. 12. A document classification method for setting a plurality of initial classification representative feature vectors and classifying a document based on the similarity between the classification representative feature vector and a document feature vector. And a refining process step of repeatedly assigning documents by changing the classification representative feature vector. The refining process step performs a refining process on a part or all of the classification representative feature vector. A document classification method characterized by the absence of a document. 13. A document classification method for classifying documents according to the contents of the documents, a document input step of inputting document data, a document analysis step of analyzing words of the document data, and a document analysis of the document analysis step. A document feature vector generation step for calculating a document feature vector for the document based on the result, a classification representative vector generation step for generating a classification representative vector having the same number of dimensions as the document feature vector, and a classification representative without refining processing A refining exclusion vector designating step of designating a vector; a document data assigning step of assigning document data to one of the classification representative vectors based on a similarity between the document feature vector and the classification representative vector; Classification representative vector specified in the generalization exclusion vector specification step For other classification representative vectors, the classification representative vector is recalculated based on the document feature vector allocated in the document data allocation step, and the document data allocation and the recalculation of the classification representative vector are performed until a specific criterion is satisfied. A document classification method comprising: a step of refining a classified representative vector; and a step of storing a classification result for storing a classification result. 14. The document classification method according to claim 13, wherein in the classification representative vector generation step, some classification representative vectors are generated based on information specified by a user. 15. The document classification method according to claim 14, wherein in the refining exclusion vector specifying step, only the classification representative vector generated by the information specified by the user in the classification representative vector generation step is refined. A document classification method characterized by using a classification representative vector that does not perform the classification. 16. The document classification method according to claim 15, wherein in the classification representative vector generation step, the information specified by the user is document data and a logical expression of a word existing in the document data. Document classification method. 17. The document classification method according to claim 16, wherein: a classification representative vector generation information storing step of storing information for generating a classification representative vector specified in the classification representative vector generation step; Classification representative vector generation information reading step for reading information stored in the vector generation information storage step; Refining exclusion vector information for storing information related to classification representative vectors not subjected to the refining process specified in the classification representative vector generation step A document classification method, further comprising: a storage step; and a refining exclusion vector information reading step for reading information stored in the refining exclusion vector information storage step. 18. A document classification method for classifying documents according to the contents of the documents, a document input step of inputting document data, a document analysis step of analyzing words of the document data, and a document analysis by the document analysis step A document feature vector generation step of calculating a document feature vector for each document from the result, and a logical expression of a plurality of information primitives specified as some classification representative vectors, and a logical sum in which each element is a logical product of the information primitives Classifying representative vector specification generating step of generating a classifying representative vector having the same number of dimensions as the document feature vector from each logical product of the generated information primitives; and Classification representative vector information that stores information about the classification representative vector A storage step, a classification representative vector automatic generation step of automatically generating a classification representative vector using information generated in the document feature vector generation step, and a similarity between the document feature vector and the classification representative vector. A document classification step of classifying the document data; and a classification result storage step of generating and storing a classification result based on the information stored in the classification representative vector information storage step. Document classification method. 19. The document classification method according to claim 18, further comprising: a step of reading classification representative vector information for reading information on the stored classification representative vector. 20. The document classification method according to claim 19, wherein in the classifying representative vector generating step, a classifying representative vector is generated as an arithmetic average of each information primitive included in a logical product of the generated information primitives. Document classification method characterized by the following. 21. The document classification method according to claim 20, wherein the information primitive specified in the classification representative vector generation step is a document and a word included in the document. 22. The document classification method according to claim 21, wherein a logical expression of an information primitive specified in the classification representative vector generation step is specified by a user.