CN102629266A - Diagram text structure representation model based on harmonic progression - Google Patents

Abstract

The invention discloses a text graph structure representation model based on harmonic series. The specific steps of the method are as follows: (1) Open a single text in the domain corpus; (2) Rearrange the text content according to the degree of importance from large to small; (3) Segment the text and retain punctuation marks; (4) Count the number of occurrences of keywords and keyword pairs; (5) use keywords as nodes in the graph to connect keyword pairs whose co-occurrence times are not 0; (6) use the harmonic series method to compare keywords and keyword Calculate the weight of . This method avoids the lack of text structure information, and at the same time can calculate the weight of keywords and keyword pairs according to the structure information of a single text; the method is simple and easy to operate, the effect is good, and it can also have the function of TFIDF.

Description

A Text Graph Structure Representation Model Based on Harmonic Series

technical field

The present invention relates to a text representation model, in particular to a text representation model using a graph structure, and a model for calculating the weight of keywords and keyword pairs using a harmonic series, which is a text graph structure representation based on a harmonic series Model.

Background technique

Humans are good at dealing with unstructured text, because unstructured text conforms to human language expression habits, and more importantly, humans have strong logical reasoning ability. Machines, on the other hand, are good at processing structured text, such as graphs and tables. During human-computer interaction, it is necessary to convert human-understandable unstructured text into machine-understandable structured text, which requires a text representation model.

The most widely used text representation model is the vector space model. The vector space model represents the text as a weight vector, each item in the vector is composed of terms, and the weight of each term is determined by the TFIDF method. The TFIDF method uses the term weight formula to calculate the importance of a term to a single text in the corpus. The term weight of the TFIDF method is the product of the term frequency TF (Term Frequency) and the inverse document frequency IDF (Inverse Document Frequency). The specific formula of TFIDF is as follows:

                                                 

Among them, TF _i is the term frequency of term i, that is, the number of times term i appears in the text; IDF _i is the inverse document frequency of term i, which is calculated by log(N/n _i ); N is the text of the text set Total number; n _i is the number of texts containing term i in the text set.

However, when using the vector space model combined with the TFIDF method to represent text, there are the following shortcomings:

(1) The vector space model regards the text as a collection of terms, and regards the relationship between terms and terms as independent, thus losing a lot of text structure information.

(2) The TFIDF method does not consider the influence of their location factors on their weights when calculating the word frequency of terms, and considering the number of occurrences or co-occurrences alone is not enough to express its actual weight.

(3) The TFIDF method needs domain-based text sets when calculating the inverse document frequency of terms, but cannot target a single text.

Contents of the invention

The purpose of the present invention is to provide a text graph structure representation model based on harmonic series in view of the deficiencies of the vector space model and the TFIDF method. The weights of keywords and keyword pairs are calculated.

In order to achieve the above-mentioned purpose, the idea of the present invention is as follows: a graph structure model is used to represent a single text, avoiding the lack of text structure information, and at the same time, the weight of keywords and keyword pairs can be adjusted according to the structural information of a single text Calculation; the graph structure model is: use the graph structure to organize the keywords of the text and the relationship between them, and then calculate the weight by the harmonic series method.

According to the above-mentioned inventive idea, the present invention adopts the following technical solutions:

A text graph structure representation model based on harmonic series, characterized in that the specific steps are as follows:

(1) Open a single text in the domain corpus;

(2) Rearrange the text content in descending order of importance;

(3) Segment the text and preserve punctuation marks;

(4) Count the number of occurrences of keywords and keyword pairs;

(5) Take the keywords as the nodes of the graph, and connect the keyword pairs whose co-occurrence times are not 0;

(6) Use the harmonic series method to calculate the weight of keywords and keyword pairs.

The harmonic series method described above is denoted as HP, and its keywords and keyword pair weights are calculated as follows:

   

为欧拉常数，

。 Among them, n is the number of occurrences of keywords and keyword pairs,

is Euler's constant,

.

Compared with the existing technology, a text graph structure representation model based on harmonic series of the present invention has the following outstanding features and advantages: in the absence of domain text sets, the ability to distinguish keywords in the text set cannot be determined, By scanning a single text, the weight of the keyword can be determined by the number of occurrences and the location of the keyword; although only the number of occurrences is used to evaluate the weight, it is easy to operate and the effect is good; due to the pairing in the harmonic series method The number is an extensible order of magnitude, so it can have the function of TFIDF and is more convenient.

Description of drawings

Fig. 1 is a flowchart of a text graph structure representation model based on harmonic series in the present invention.

Detailed ways

Embodiments of the present invention will be further described below in conjunction with accompanying drawings.

Embodiment 1: Referring to Fig. 1, this text graph structure representation model based on harmonic series is characterized in that: a graph structure model is used to represent a single text, wherein the harmonic series method is used to weight keywords and keyword pairs Calculation;

The graph structure model is to establish a connection relationship between the keywords of the text according to the co-occurrence relationship of the keyword pairs in the same sentence;

Described harmonic series method, its key word and key word pair weight computing formula are as follows:

，n为关键词和关键词对的出现次数，

为欧拉常数，

。

, n is the number of occurrences of keywords and keyword pairs,

is Euler's constant,

.

Example 2: This text graph structure representation model based on harmonic series represents the text of 70 papers from TKDE from 2011 to 2012. As shown in Fig. 1, a kind of text graph structure representation model based on harmonic series of the present embodiment, its steps are as follows:

S1. Open a single text in a field corpus, for example, open a single paper in Volume 24, Issue 1, 2011;

S2. Rearrange the content of the text in descending order of importance, for example, rearrange in order of title, abstract, introduction and summary;

S3. Segment the text and preserve punctuation marks, for example, retain periods.

S4. Count the number of occurrences of keywords and keyword pairs, denoted as n.

S5. Use keywords as nodes in the graph, and connect keyword pairs whose co-occurrence times are not 0.

S6. Use the harmonic series method to calculate the weight of keywords and keyword pairs; the formula of the harmonic series method is denoted as HP, and the calculation formula for the weight of keywords and keyword pairs is as follows:

  

为欧拉常数，

。 Among them, n is the number of occurrences of keywords and keyword pairs,

is Euler's constant,

.

Claims (2)

1. A text graph structure representation model based on harmonic series, characterized in that: a graph structure model is used to represent a single text, wherein the harmonic series method is used to calculate the weight of keywords and keyword pairs; The graph structure model of the text is to establish a connection relationship between the keywords of the text according to the co-occurrence relationship of the keyword pairs in the same sentence; the specific steps are as follows: Open a single text in a domain corpus; Rearrange the text content in descending order of importance; Tokenize text and preserve punctuation; Count the number of occurrences of keywords and keyword pairs; Use keywords as nodes in the graph, and connect keyword pairs whose co-occurrence times are not 0; The weights of keywords and keyword pairs are calculated using the harmonic series method. 2. by the described text graph structure representation model based on harmonic series according to claim 1, it is characterized in that: the harmonic series method in the described step (6), its keyword and keyword pair weight calculation formula are as follows: Among them, n is the number of occurrences of keywords and keyword pairs,

is Euler's constant,

.

Images