CN106649263A - Multi-word expression extraction method and device - Google Patents

Abstract

The present invention relates to a multi-word expression extraction method and its device, comprising: forming a vocabulary set after preprocessing a document library, calculating the mutual information of adjacent words in multiple documents, obtaining jump information before and after the mutual information sequence, and combining the mutual information with the jump information. Variable information constitutes two-dimensional mutual information, clustering two-dimensional mutual information screens out multi-word expressions, and builds a multi-word expression library. The present invention avoids the need to manually set the threshold for one-dimensional mutual information, and adapts to different data. At the same time, it is not limited to the binary structure of multi-words, and can obtain multi-word expressions of multi-word combinations at one time without step-by-step implementation. Effectively improve the utilization rate of multi-word expressions, and improve the accuracy of multi-word expression database construction.

Description

A multi-word expression extraction method and device thereof

technical field

The invention relates to the technical field of statistical machine translation and cross-language information retrieval, in particular to a multi-word expression extraction method and device thereof.

Background technique

Multi-word expression is a combination of multiple words with grammatical, semantic or pragmatic characteristics and complete meaning. The recognition of multi-word expressions can improve the efficiency and accuracy of word segmentation, part-of-speech tagging, and machine translation. In machine translation, correct identification of multi-word expressions in the source language is helpful to choose a suitable translation, and avoid unnatural or even unintelligible target language caused by separate translation of multiple words.

The extraction methods of multi-word expressions are basically divided into statistical methods and rule-based methods. Rule-based methods generally study a certain type such as verb phrase structure or are limited to a specific field, while statistics-based methods can extract multi-word expressions with independent forms, that is, use statistical information to extract various expressions without distinction. Multi-word expressions for structures and domains. However, the problems faced by the existing statistical methods are: one-dimensional mutual information needs to manually set the threshold, there is an adaptability problem to different data, it is limited to the binary structure of multi-words, and it is impossible to obtain multi-word expressions of multi-word combinations at one time. And it needs to be implemented step by step, and the accuracy of multi-word expression database construction is low.

Contents of the invention

The primary purpose of the present invention is to provide a multi-word expression that obtains multi-word combinations at one time without step-by-step implementation, effectively improves the utilization rate of multi-word expression extraction, and improves the accuracy of multi-word expression database construction.

In order to achieve the above object, the present invention adopts the following technical solutions, a multi-word expression extraction method, the method includes the steps of the following order:

(1) The document library adopts preprocessing of word segmentation and part-of-speech tagging to form source language documents;

(2) Calculate the mutual information of adjacent words in multiple documents, and further calculate the jump information before and after the mutual information sequence;

(3) The mutual information sequence and the jump information sequence constitute a two-dimensional mutual information set;

(4) The two-dimensional mutual information set uses a classifier to express inliers and outliers for multi-words, and selects multi-inlier links to construct multi-word expressions.

Further, in the step (1), the preprocessing of Chinese word segmentation, part-of-speech tagging, named entity recognition, and part-of-speech selection is performed on all documents in the collected document library to form a set of candidate words in a specific order.

Further, said step (2) includes the steps in the following order:

(a) Calculate the mutual information of all adjacent words in multiple documents;

(b) Calculate the jump information before and after the mutual information sequence.

Further, in the step (3), two-dimensional mutual information (MI _i , f _i ) is constructed according to the corresponding position points of the mutual information sequence and the hopping information sequence, and a plurality of two-dimensional mutual information forms a two-dimensional mutual information set.

Further, in the step (4), a classifier is used to divide all points in the two-dimensional mutual information set into two types of multi-word expression interior points and exterior points, and the adjacent vocabulary links containing interior points form multi-word expression .

Further, in the step (a), the mutual information of adjacent words in multiple documents is calculated to form a mutual information sequence MI, wherein the mutual information of adjacent words x and y is calculated as MI _i (0≤i<len(MI) -α) as follows:

Among them, x and y represent adjacent words; MI _i represents the i-th mutual information formed by adjacent words x and y; len(MI) represents the length of the mutual information sequence MI; α represents a constant; M represents the vocabulary in all documents The total number of; p(x, y) indicates the number of occurrences of vocabulary x and y in all documents; p(x) indicates the number of occurrences of vocabulary x in all documents; p(y) indicates the number of occurrences of vocabulary y in all documents; N Indicates the number of all documents in the document set; N _{x, y} indicates the number of documents containing co-occurrence of x and y.

Further, in the step (b), the jump information before and after the mutual information sequence is calculated to form the jump information sequence f, and the calculation formula of the jump information _fi of the adjacent mutual information is as follows:

Among them, f _i represents the jump information of the current mutual information and the subsequent mutual information in the mutual information sequence; || represents the absolute value.

Further, the α is 2.

Another object of the present invention is to provide a multi-word expression extraction device, comprising:

Candidate vocabulary acquisition device: perform preprocessing of Chinese word segmentation, part-of-speech tagging and named entity recognition, and part-of-speech selection for all documents in the collected document library to form a candidate vocabulary set with a specific order;

Mutual information and jump information acquisition device: calculate the mutual information of adjacent candidate words in multiple documents, and calculate the jump information before and after the mutual information sequence according to the adjacent mutual information;

The two-dimensional mutual information acquisition device: according to the information corresponding to the position of the mutual information sequence and the jump information sequence, select the mutual information and the jump information to form the two-dimensional mutual information;

Classification and screening multi-word expression device: use a classifier to classify all points in the two-dimensional mutual information set into two types of multi-word expression inner points and outer points, and link adjacent words with inner points to form multi-word expressions.

It can be seen from the above technical solution that the present invention converts the mutual information between adjacent words into two-dimensional mutual information, and clusters two-dimensional mutual information to screen out multi-word expressions, avoiding the need to manually set thresholds for one-dimensional mutual information, and different data At the same time, it is not limited to the binary structure of multi-words, and multi-word expressions of multi-word combinations can be obtained at one time without step-by-step implementation, which effectively improves the utilization rate of multi-word expressions and improves the construction of multi-word expression databases. Accuracy.

Description of drawings

Fig. 1 is a schematic flow sheet of the inventive method;

Fig. 2 is a structural block diagram of the device of the present invention.

detailed description

A multi-word expression extraction method, the method comprises the steps of the following sequence: (1) the document base adopts preprocessing such as word segmentation and part-of-speech tagging to form a source language document; (2) calculates the mutual information of adjacent words in the multi-document, and Further calculate the jump information before and after the mutual information sequence; (3) The mutual information sequence and the jump information sequence form a two-dimensional mutual information set; (4) The two-dimensional mutual information set uses a classifier to express inliers and outliers for multiple words , to filter continuous internal point links to construct multi-word expressions. As shown in Figure 1.

The present invention will be further described below in conjunction with FIG. 1 .

In the step (1), Chinese word segmentation, part-of-speech tagging, named entity recognition, and part-of-speech selection are performed on all texts in the collected document base to form a set of candidate words in a specific order.

The step (2) includes steps in the following order: (a) calculating the mutual information of all adjacent words in multiple documents; (b) calculating the jump information before and after the mutual information sequence.

In the step (a), the mutual information of adjacent words in multiple documents is calculated to form a mutual information sequence MI, wherein the mutual information of adjacent words x and y is calculated as MI _i (0≤i<len(MI)-α ) as follows:

Among them, x and y represent adjacent words; MI _i represents the i-th mutual information formed by adjacent words x and y; len(MI) represents the length of the mutual information sequence MI; α represents a constant; M represents the vocabulary in all documents The total number of; p(x, y) indicates the number of occurrences of vocabulary x and y in all documents; p(x) indicates the number of occurrences of vocabulary x in all documents; p(y) indicates the number of occurrences of vocabulary y in all documents; N Indicates the number of all documents in the document set; N _{x, y} indicates the number of documents containing co-occurrence of x and y; the constant α is 2.

In the step (b), the jump information before and after the mutual information sequence is calculated to form the jump information sequence f, and the calculation formula of the jump information _fi of the adjacent mutual information is as follows:

Among them, f _i represents the jump information of the current mutual information and the subsequent mutual information in the mutual information sequence; || represents the absolute value.

In the step (3), two-dimensional mutual information (MI _i , f _i ) is constructed according to the corresponding position points of the mutual information sequence and the hopping information sequence, and a plurality of two-dimensional mutual information forms a two-dimensional mutual information set.

In the step (4), a classifier is used to divide all points in the two-dimensional mutual information set into two types of multi-word expression inner points and outer points, and adjacent vocabulary links containing inner points are used to form multi-word expressions.

As shown in Figure 2, the device of the present invention includes: a candidate vocabulary acquisition device, which performs preprocessing such as Chinese word segmentation, part-of-speech tagging, named entity recognition, and part-of-speech selection for all texts in the collected document library to form a candidate vocabulary set with a specific order; mutual information and jump information acquisition device, calculate the mutual information of adjacent candidate words in multiple documents, and calculate the jump information before and after the mutual information sequence according to the adjacent mutual information; the two-dimensional mutual information acquisition device, according to the mutual information sequence and the jump information Information corresponding to the information sequence position, select mutual information and jump information to form two-dimensional mutual information; classify and screen multi-word expression devices, use a classifier to classify all points in the two-dimensional mutual information set into multi-word expression internal points and external points Two types, linking adjacent words with interior points to form multi-word expressions.

In summary, the present invention transforms the mutual information between adjacent words into two-dimensional mutual information, clusters the two-dimensional mutual information to screen out multi-word expressions, avoids the need for manual setting of thresholds for one-dimensional mutual information, and compares different data The problem of adaptability is not limited to the binary structure of multi-words, and the multi-word expressions of multi-word combinations can be obtained at one time without step-by-step implementation, which effectively improves the utilization rate of multi-word expressions and improves the accuracy of multi-word expression database construction Spend.

Claims (9)

1. A multi-word expression extraction method is characterized in that the method comprises the steps of the following order: (1) The document library adopts preprocessing of word segmentation and part-of-speech tagging to form source language documents; (2) Calculate the mutual information of adjacent words in multiple documents, and further calculate the jump information before and after the mutual information sequence; (3) The mutual information sequence and the jump information sequence constitute a two-dimensional mutual information set; (4) The two-dimensional mutual information set uses a classifier to express inliers and outliers for multi-words, and selects multi-inlier links to construct multi-word expressions. 2. The method according to claim 1, characterized in that: in said step (1), the preprocessing of Chinese word segmentation, part-of-speech tagging and named entity recognition, part-of-speech selection for all documents in the document library constitutes a specific order set of candidate words. 3. The method according to claim 1, characterized in that: said step (2) comprises the steps in the following order: (a) Calculate the mutual information of all adjacent words in multiple documents; (b) Calculate the jump information before and after the mutual information sequence. 4. The method according to claim 1, characterized in that: in the step (3), two-dimensional mutual information (MI _i , f _i ) is constructed according to the corresponding position points of the mutual information sequence and the hopping information sequence, multiple Two-dimensional mutual information constitutes a two-dimensional mutual information set. 5. method according to claim 1, it is characterized in that: in described step (4), adopt classifier to divide all points in the two-dimensional mutual information set into multi-word expression internal point and external point two classes, will Adjacent lexical links containing interior points form multi-word expressions. 6. The method according to claim 3, characterized in that: in the step (a), the mutual information of adjacent words in multiple documents is calculated to form a mutual information sequence MI, wherein the mutual information of adjacent words x and y Calculate MI _i (0≤i<len(MI)-α) as follows: ${\mathrm{<span\; class="notranslate">\; MI</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; \&lsqb;</span>\frac{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&times;</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&times;</span>\frac{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; ,</span>$ Among them, x and y represent adjacent words; MI _i represents the i-th mutual information formed by adjacent words x and y; len(MI) represents the length of the mutual information sequence MI; α represents a constant; M represents the vocabulary in all documents The total number of; p(x, y) indicates the number of occurrences of vocabulary x and y in all documents; p(x) indicates the number of occurrences of vocabulary x in all documents; p(y) indicates the number of occurrences of vocabulary y in all documents; N Indicates the number of all documents in the document set; N _{x, y} indicates the number of documents containing co-occurrence of x and y. 7. The method according to claim 3, characterized in that: in the step (b), the jump information before and after the mutual information sequence is calculated to form the jump information sequence f, wherein the jump information of adjacent mutual information The calculation formula of f _i is as follows: ${\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}{\mathrm{<span\; class="notranslate">\; MI</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; MI</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; <</span>\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}$ Among them, f _i represents the jump information of the current mutual information and the subsequent mutual information in the mutual information sequence; || represents the absolute value. 8. The method according to claim 6, characterized in that: said α is 2. 9. A multi-word expression extraction device, comprising: Candidate vocabulary acquisition device: perform preprocessing such as Chinese word segmentation, part-of-speech tagging, named entity recognition, and part-of-speech selection for all documents in the collected document library to form a candidate vocabulary set with a specific order; Mutual information and jump information acquisition device: calculate the mutual information of adjacent candidate words in multiple documents, and calculate the jump information before and after the mutual information sequence according to the adjacent mutual information; The two-dimensional mutual information acquisition device: according to the information corresponding to the position of the mutual information sequence and the jump information sequence, select the mutual information and the jump information to form the two-dimensional mutual information; Classification and screening multi-word expression device: use a classifier to classify all points in the two-dimensional mutual information set into two types: inner points and outer points of multi-word expressions, and link adjacent words with inner points to form multi-word expressions.