CN104239479A - Document classification method and system - Google Patents

Abstract

The invention discloses a document classification method and system, which are applied to a Hadoop cluster including a Map program and a Reduce program. The method includes the following steps: the Map program analyzes the training documents and the documents to be classified, and determines according to the analysis results feature attribute, and divide the feature attribute; the Map program generates a classifier according to the feature attribute of the training document and the classification result of the training document; the Reduce program uses the classifier to classify the The document to be classified is classified, and the classification result of the document to be classified is obtained. The present invention makes full use of the distributed characteristics of the Hadoop cluster, avoids the limitations of the traditional system framework, has the characteristics of parallel and fast, can quickly realize the classification of massive documents, saves classification time, improves the efficiency of document classification, and improves the efficiency of document classification. system performance.

Description

A kind of document classification method and system

technical field

The invention relates to the field of computer technology, in particular to a document classification method and system.

Background technique

With the increasing popularity of network technology, the amount of data in the network has increased dramatically, and the types of applications are also very rich. Data mining technology makes full use of existing information resources and finds hidden knowledge from a large amount of data, which is a powerful development direction. Data mining involves fields such as machine learning, pattern recognition, statistics, intelligent databases, data visualization, and high-performance computing. Its purpose is to discover hidden, novel, and interesting relationships and laws from large amounts of data. Among them, document classification is an important direction of data mining.

In the prior art, traditional system frameworks are usually used for document classification, which will lead to long classification time and low system performance when processing massive data.

Contents of the invention

The invention provides a document classification method and system to solve the defect of low system performance in the prior art.

The present invention provides a kind of document classification method, is applied in the Hadoop cluster that comprises Map program and Reduce program, and described method comprises the following steps:

The Map program parses the training document and the document to be classified, determines the feature attribute according to the parsing result, and divides the feature attribute;

The Map program generates a classifier according to the feature attribute of the training document and the classification result of the training document;

The Reduce program uses the classifier to classify the document to be classified to obtain a classification result of the document to be classified.

Optionally, after the Map program determines the characteristic attribute according to the parsing result, it also includes:

The Map program performs format conversion on the training document and the document to be classified respectively according to the characteristic attribute, and obtains the training document and the document to be classified conforming to a preset format;

The Map program generates a classifier according to the feature attribute of the training document and the classification result of the training document, specifically:

The Map program generates a classifier according to the feature attribute of the training document after format conversion and the classification result of the training document;

The Reduce program uses the classifier to classify the document to be classified to obtain the classification result of the document to be classified, specifically:

The Reduce program uses the classifier to classify the documents to be classified after format conversion, and obtain the classification results of the documents to be classified.

Optionally, the Map program generates a classifier according to the feature attributes of the format-converted training document and the classification result of the training document, specifically:

The Map program calculates the frequency of occurrence of each category in the training document and the frequency of occurrence of each category in the training document according to the value range of each feature attribute corresponding to the format-converted training document and the classification result of the training document. conditional probability estimates for each value range of all the feature attributes, and record the frequency of occurrence and the conditional probability estimates as a classifier.

Optionally, the Reduce program uses the classifier to classify the format-converted documents to be classified to obtain a classification result of the documents to be classified, specifically:

The Reduce program obtains the value range of all feature attributes of the document to be classified after the format conversion, according to the obtained value range, the frequency of occurrence of each category in the training document, and all feature attributes under each category The conditional probability estimation of each value range of the document to be classified is calculated, and the conditional probability that the document to be classified belongs to each category is calculated, and the category corresponding to the conditional probability with the largest value is used as the classification result of the document to be classified.

Optionally, the Map program parses the training document and the document to be classified, determines the feature attribute according to the parsing result, and divides the feature attribute, specifically:

The Map program obtains the attributes contained in the training documents and the documents to be classified by parsing the training documents and the documents to be classified, and selects characteristic attributes from the attributes obtained by parsing, and divides multiple value ranges for each characteristic attribute.

The present invention also provides a kind of document classification system, is applied in Hadoop cluster, and described system comprises:

The analysis module is used to analyze the training documents and the documents to be classified, determine the characteristic attributes according to the analysis results, and divide the characteristic attributes;

A generating module, configured to generate a classifier according to the characteristic attributes of the training document determined by the parsing module and the classification results of the training document;

A classification module, configured to use the classifier generated by the generating module to classify the document to be classified, and obtain a classification result of the document to be classified.

Optionally, the system further includes:

A conversion module, configured to perform format conversion on the training document and the document to be classified respectively according to the characteristic attributes determined by the parsing module, so as to obtain the training document and the document to be classified conforming to a preset format;

The generation module is specifically configured to generate a classifier according to the characteristic attributes of the training document after format conversion by the conversion module and the classification result of the training document;

The classification module is specifically configured to use the classifier generated by the generation module to classify the documents to be classified after format conversion by the conversion module, and obtain a classification result of the documents to be classified.

Optionally, the generation module is specifically configured to calculate the value range of each feature attribute corresponding to the training document after the format conversion of the conversion module and the classification result of the training document, and calculate each category in the training document. The occurrence frequency in the document and the conditional probability estimation of each value range of all feature attributes under each category, and the occurrence frequency and the conditional probability estimation are recorded as a classifier.

Optionally, the classification module is specifically configured to obtain the value ranges of all feature attributes of the documents to be classified after format conversion by the conversion module, according to the obtained value ranges, the appearance of each category in the training documents Frequency and the conditional probability estimation of each value range of all feature attributes under each category, calculate the conditional probability that the document to be classified belongs to each category, and use the category corresponding to the conditional probability with the largest value as the document to be classified classification results.

Optionally, the parsing module is specifically configured to obtain the attributes contained in the training documents and the documents to be classified by parsing the training documents and the documents to be classified, and select feature attributes from the attributes obtained by parsing, and for each feature Attributes divide multiple value ranges.

The present invention makes full use of the distributed characteristics of the Hadoop cluster, avoids the limitations of the traditional system framework, has the characteristics of parallel and fast, can quickly realize the classification of massive documents, saves classification time, improves the efficiency of document classification, and improves the efficiency of document classification. system performance.

Description of drawings

Fig. 1 is the flowchart of a kind of document classification method in the embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a document classification system in an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

It should be noted that, if there is no conflict, the embodiments of the present invention and various features in the embodiments can be combined with each other, and all are within the protection scope of the present invention. In addition, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

A kind of document classification method is proposed in the embodiment of the present invention, is applied in the Hadoop cluster that comprises Map program and Reduce program, uses Hadoop command to place training document and document to be classified to HDFS (Hadoop Distributed File System, distributed file system ) and perform the operations shown in Figure 1:

Step 101, the Map program parses the training document and the document to be classified, determines the characteristic attributes according to the parsing results, and divides the characteristic attributes.

Specifically, the Map program can obtain the attributes contained in the training documents and the documents to be classified by parsing the training documents and the documents to be classified, and select characteristic attributes from the attributes obtained by parsing, and divide multiple values for each characteristic attribute scope.

Among them, the training documents and the documents to be classified can be located in different directories in HDFS, and are managed by the classification directory. The name of each folder is the class label, and the content under the folder is the corresponding data belonging to the class label. Class documentation.

For example, training documents are located in the /train directory in HDFS, and documents to be classified are located in the /test directory in HDFS. The Map program selects three feature attributes based on the analysis results of the training documents and the documents to be classified: a. Number of logs/number of days of registration; b. Number of friends/number of days of registration; c. Whether to use real avatars, and divide each feature attribute For: {a<=0.05, 0.05<a<0.2, a>=0.2}; {b<=0.1, 0.1<b<0.8, b>=0.8}; {c=0 (not), c=1( yes)}.

Step 102, the Map program performs format conversion on the training document and the document to be classified respectively according to the determined feature attributes, and obtains the training document and the document to be classified conforming to the preset format.

Specifically, the Map program can use the PrepareTwentyNewsgroups class in the command line Mahout to convert the training document and the document to be classified into the training document and the document to be classified conforming to the preset format. Wherein, the preset format may be a VectorWritable format, and in a document conforming to the VectorWritable format, the first character is a class label, and the remaining characters are feature attributes.

Step 103, the Map program generates a classifier according to the feature attributes of the format-converted training documents and the classification results of the training documents.

Specifically, the Map program can calculate the frequency of occurrence of each category in the training document and all feature attributes under each category according to the value range of each feature attribute corresponding to the format-converted training document and the classification results of the training document The conditional probability estimates for each value range of , record the above frequency of occurrence and conditional probability estimates as a classifier.

For example, the number of training documents is 10,000, and the classification result is: 8900 training documents belong to real accounts (ie, C=0), and 1100 training documents belong to non-real accounts (ie, C=1).

The frequency of each category in the training documents is:

P(C=0)=8900/10000=0.89;

P(C=1)=1100/10000=0.11;

The conditional probability estimates for each value range of all feature attributes under each category are:

P(a<=0.05︱C=0)=0.3

P(0.05<a<0.2︱C＝0)＝0.5

P(a>=0.2︱C＝0)＝0.2

P(a<=0.05︱C=1)=0.8

P(0.05<a<0.2︱C＝1)＝0.1

P(a>＝0.2︱C＝1)＝0.1

P(b<=0.1︱C=0)=0.1

P(0.1<b<0.8︱C＝0)＝0.7

P(b>＝0.8︱C＝0)＝0.2

P(b<=0.1︱C=1)=0.7

P(0.1<b<0.8︱C＝1)＝0.2

P(b>＝0.8︱C＝1)＝0.1

P(c＝0︱C＝0)＝0.2

P(c=1︱C=0)=0.8

P(c＝0︱C＝1)＝0.9

P(c=1︱C=1)=0.1

Step 104, the Reduce program uses the classifier to classify the documents to be classified after format conversion, and obtains the classification results of the documents to be classified.

Specifically, the Reduce program can obtain the value ranges of all feature attributes of the documents to be classified after format conversion, according to the obtained value ranges, the frequency of occurrence of each category in the training document, and all feature attributes under each category Estimate the conditional probability of each value range of , calculate the conditional probability that the document to be classified belongs to each category, and record the category corresponding to the conditional probability with the largest value as the classification result of the document to be classified to HDFS.

For example, the value ranges of the three feature attributes of the document to be classified are: 0.05<a<0.2, 0.1<b<0.8, b>=0.8, c=0, then the document to be classified belongs to a real account (that is, C=0 ) conditional probability is:

P(C＝0)P(x︱C＝0)

=P(C=0)P(0.05<a<0.2︱C=0)P(0.1<b<0.8︱C=0)P(c=0︱C=0)

=0.89*0.5*0.7*0.2

=0.0623;

The conditional probability that the document to be classified belongs to a non-real account (that is, C=1) is:

P(C＝1)P(x︱C＝1)

＝P(C＝1)P(0.05<a<0.2︱C＝1)P(0.1<b<0.8︱C＝1)P(c＝0︱C＝1)

=0.11*0.1*0.2*0.9

=0.00198

Since the conditional probability that the document to be classified belongs to the real account is the largest, the Reduce program determines that the document to be classified belongs to the real account.

The embodiment of the present invention makes full use of the distributed characteristics of the Hadoop cluster, avoids the limitations of the traditional system framework, has the characteristics of parallel and fast, can quickly realize the classification of massive documents, saves classification time, and improves the efficiency of document classification. Improved system performance.

Based on the above-mentioned web page clustering method, an embodiment of the present invention proposes a document classification system, which is applied in a Hadoop cluster, as shown in Figure 2, the system includes:

The analysis module 210 is used to analyze the training document and the document to be classified, determine the characteristic attribute according to the analysis result, and divide the characteristic attribute;

Specifically, the above parsing module 210 is specifically used to analyze the training documents and the documents to be classified to obtain the attributes contained in the training documents and the documents to be classified, and select feature attributes from the attributes obtained by parsing, and for each feature attribute Divide multiple value ranges.

A generating module 220, configured to generate a classifier according to the feature attributes of the training documents determined by the parsing module 210 and the classification results of the training documents;

The classification module 230 is configured to use the classifier generated by the generation module 220 to classify the document to be classified, and obtain a classification result of the document to be classified.

Further, the above system also includes:

The conversion module 240 is used to perform format conversion on the training document and the document to be classified respectively according to the characteristic attributes determined by the parsing module 210, so as to obtain the training document and the document to be classified conforming to the preset format;

Correspondingly, the above-mentioned generation module 220 is specifically configured to generate a classifier according to the characteristic attributes of the training document converted by the conversion module 240 and the classification result of the training document;

The above-mentioned classification module 230 is specifically configured to use the classifier generated by the generation module 220 to classify the documents to be classified after format conversion by the conversion module 240 to obtain the classification results of the documents to be classified.

Further, the above-mentioned generation module 220 is specifically used to calculate the frequency of occurrence of each category in the training document and In the conditional probability estimation of each value range of all feature attributes under each category, the above-mentioned frequency of occurrence and the above-mentioned conditional probability estimation are recorded as a classifier.

Correspondingly, the above classification module 230 is specifically used to obtain the value ranges of all feature attributes of the documents to be classified after format conversion by the conversion module 240, according to the obtained value ranges, the frequency of occurrence of each category in the training documents and Estimate the conditional probability of each value range of all feature attributes under each category, calculate the conditional probability that the document to be classified belongs to each category, and use the category corresponding to the conditional probability with the largest value as the classification of the document to be classified result.

The embodiment of the present invention makes full use of the distributed characteristics of the Hadoop cluster, avoids the limitations of the traditional system framework, has the characteristics of parallel and fast, can quickly realize the classification of massive documents, saves classification time, and improves the efficiency of document classification. Improved system performance.

The steps in the methods described in conjunction with the embodiments disclosed herein may be directly implemented by hardware, software modules executed by a processor, or a combination of both. Software modules can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other Any other known storage medium.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (10)

1. a document classification method, is characterized in that, is applied in the Hadoop cluster that comprises Map program and Reduce program, and described method comprises the following steps: The Map program parses the training document and the document to be classified, determines the feature attribute according to the parsing result, and divides the feature attribute; The Map program generates a classifier according to the feature attribute of the training document and the classification result of the training document; The Reduce program uses the classifier to classify the document to be classified to obtain a classification result of the document to be classified. 2. method as claimed in claim 1, is characterized in that, after described Map program determines feature attribute according to parsing result, also comprises: The Map program performs format conversion on the training document and the document to be classified respectively according to the characteristic attribute, and obtains the training document and the document to be classified conforming to a preset format; The Map program generates a classifier according to the feature attribute of the training document and the classification result of the training document, specifically: The Map program generates a classifier according to the feature attribute of the training document after format conversion and the classification result of the training document; The Reduce program uses the classifier to classify the document to be classified to obtain the classification result of the document to be classified, specifically: The Reduce program uses the classifier to classify the documents to be classified after format conversion, and obtain the classification results of the documents to be classified. 3. The method according to claim 2, wherein the Map program generates a classifier according to the feature attribute of the training document after format conversion and the classification result of the training document, specifically: The Map program calculates the frequency of occurrence of each category in the training document and the frequency of occurrence of each category in the training document according to the value range of each feature attribute corresponding to the format-converted training document and the classification result of the training document. conditional probability estimates for each value range of all the feature attributes, and record the frequency of occurrence and the conditional probability estimates as a classifier. 4. The method according to claim 3, wherein the Reduce program uses the classifier to classify the documents to be classified after format conversion, to obtain the classification results of the documents to be classified, specifically: The Reduce program obtains the value range of all feature attributes of the document to be classified after the format conversion, according to the obtained value range, the frequency of occurrence of each category in the training document, and all feature attributes under each category The conditional probability estimation of each value range of the document to be classified is calculated, and the conditional probability that the document to be classified belongs to each category is calculated, and the category corresponding to the conditional probability with the largest value is used as the classification result of the document to be classified. 5. The method according to claim 1, wherein the Map program parses the training document and the document to be classified, determines the feature attribute according to the analysis result, and divides the feature attribute, specifically: The Map program obtains the attributes contained in the training documents and the documents to be classified by parsing the training documents and the documents to be classified, and selects characteristic attributes from the attributes obtained by parsing, and divides multiple value ranges for each characteristic attribute. 6. a kind of document classification system, is characterized in that, is applied in Hadoop cluster, and described system comprises: The analysis module is used to analyze the training documents and the documents to be classified, determine the characteristic attributes according to the analysis results, and divide the characteristic attributes; A generating module, configured to generate a classifier according to the feature attributes of the training document determined by the parsing module and the classification results of the training document; A classification module, configured to use the classifier generated by the generating module to classify the document to be classified, and obtain a classification result of the document to be classified. 7. The system of claim 6, further comprising: A conversion module, configured to perform format conversion on the training document and the document to be classified respectively according to the characteristic attributes determined by the parsing module, so as to obtain the training document and the document to be classified conforming to a preset format; The generation module is specifically configured to generate a classifier according to the characteristic attributes of the training document after format conversion by the conversion module and the classification result of the training document; The classification module is specifically configured to use the classifier generated by the generation module to classify the documents to be classified after format conversion by the conversion module to obtain a classification result of the documents to be classified. 8. The system of claim 7, wherein: The generation module is specifically configured to calculate the occurrence of each category in the training document according to the value range of each feature attribute corresponding to the training document converted from the format of the conversion module and the classification result of the training document Frequency and the conditional probability estimates of the respective value ranges of all feature attributes under each category, and the frequency of occurrence and the conditional probability estimates are recorded as a classifier. 9. The system of claim 8, wherein: The classification module is specifically used to obtain the value ranges of all feature attributes of the documents to be classified after format conversion by the conversion module, according to the obtained value ranges, the frequency of occurrence of each category in the training document, and the Estimate the conditional probability of each value range of all feature attributes under each category, calculate the conditional probability that the document to be classified belongs to each category, and use the category corresponding to the conditional probability with the largest value as the classification result of the document to be classified. 10. The system of claim 6, wherein: The parsing module is specifically used to analyze the training documents and the documents to be classified to obtain the attributes contained in the training documents and the documents to be classified, and select characteristic attributes from the attributes obtained by parsing, and divide each characteristic attribute into multiple Ranges.