CN110134951B - A method and system for analyzing potential topic phrases in text data - Google Patents

Abstract

The invention discloses a method and a system for analyzing potential topic phrases of text data. The method includes: collecting a text data set, and segmenting the text data set to obtain a word expression form of the text data set; extracting words according to the text data set The effective phrases formed after word collocation are obtained, and the mixed expressions of words and phrase sets that are not collocated into effective phrases are obtained; the corresponding word vector model is obtained after word vector training is performed on the text data set of mixed expressions; DR‑Phrase LDA is constructed and Solve for each parameter; train DR‑Phrase LDA and output potential topic phrases from text data based on the training results. The present invention adopts a phrase topic model based on word vectors, which reasonably improves the statistical information of phrases in model training with the help of Chinese linguistic laws in the training of probabilistic topic models, and specifically adopts the method of word vectors to measure the relationship between phrase components. Quantitatively reflects the semantic relationship of words in the whole text and phrase parts, which makes the model more accurate.

Description

A method and system for analyzing potential topic phrases in text data

technical field

The invention relates to the field of text data mining and analysis, in particular to a method and system for analyzing potential topic phrases of text data.

Background technique

With the development of information technology, a large number of electronic texts have been accumulated in various fields, resulting in information overload. In order to help people quickly retrieve, find and effectively use this information, text semantics and structure analysis has become one of the hotspots of today's research. The analysis of potential subject information from text data is one of the key technologies in advanced application systems such as information retrieval, recommendation systems, and automatic summarization. The existing common methods use traditional probabilistic topic models based on "bag of words" such as LDA and PLDA for text topic analysis. The subject results obtained by these methods are presented in the form of subject words, while human natural language is accustomed to express semantics in the form of phrase blocks, so the subject results obtained by these methods have defects such as readability, consistency and poor visualization.

At present, there are two strategies for similar methods: the first one extracts phrases from text data and then trains the topic model. Due to the lack of statistical information of phrases, the probability of occurrence of phrases in this type of method is extremely low during the model training process and cannot be effectively reflected in the results of topic phrases. The second is to train the topic model to obtain the topic words, and then to synthesize the phrases with the topic words. Due to the flexibility of Chinese words, the quality of the topic phrases synthesized later is also poor.

SUMMARY OF THE INVENTION

Purpose of the invention: In order to overcome the deficiencies of the prior art, the present invention provides a method for analyzing potential topic phrases in text data, which overcomes the problems of readability and consistency in the topic results obtained by the topic model training based on the traditional "bag of words". The defects of poor performance and visualization are solved; and the problem that similar methods cannot obtain effective subject phrase results due to lack of statistical information of phrases is solved; the invention also provides a system for analyzing potential subject phrases of text data.

Technical solution: a method for analyzing potential topic phrases in text data according to the present invention, the method includes:

(1) Collecting a text data set, and performing word segmentation on the text data set to obtain the word representation of the text data set;

(2) According to the words in the text data set, the effective phrases formed by the collocation of the words are extracted, and the mixed expression form of the words that are not collocated into the effective phrases and the phrase set is obtained;

(3) The corresponding word vector model is obtained after word vector training is performed on the text data set of mixed expressions;

(4) Construct a phrase topic model DR-Phrase LDA based on word vector and solve each parameter;

(5) Train the DR-Phrase LDA, and output potential topic phrases of text data according to the training results.

Further, include:

In the step (2), the valid phrases include n-gram phrases, where n-grams are the number of words that form the phrase, and the valid phrases formed after the word collocation is extracted according to the words in the text data set, specifically including:

(21) Count the co-occurrence frequencies of two-word or phrase collocations in the text data set to form a binary phrase candidate set;

(22) Calculate the score(w _i , w _j ) of the candidate set of binary phrases, select the top m with high scores to form formal binary phrases, and add them to the phrase set. Update the corresponding words in the word representations of the text data set to be the obtained phrases;

(23) An n-gram phrase formed by collocation of the binary phrase and other words or phrases calculated in the iterative steps (21) and (22) is added to the phrase set in turn.

Further, include:

其中，所有文本共享K个主题；从词语的多项式分布

中采样生成词语或短语t。In the step (4), the DR-Phrase LDA model is a probability generation model, and the process of realizing text generation by DR-Phrase LDA is: first, there are M texts in the text data set D, from the Dirichlet distribution whose hyperparameter is α The topic distribution θ _d of the generated text d is sampled from ; the topic parameter z of the words or phrases in the text d is generated by sampling from the multinomial distribution θ _d of the topic, and the topic number is denoted as z _mn ; the hyperparameter is β Dirichlet distribution. Generate a distribution of words or phrases corresponding to topic z

where all texts share K topics; from the multinomial distribution of words

Sampling produces a word or phrase t.

Further, include:

The calculation of the topic parameter z of the word or phrase is completed using the Gibbs sampling approximate solution method, which is expressed as:

Among them, the word or valid phrase in the current position of document d in the sampling process is denoted as t, denoted as term, k is the assigned topic number, K is the preset number of topics, N _t is the total number of terms in the text data set, n _k/d represents the count of topic k in document d, n _t/k represents the count of t in topic k, n _r represents the number of semantically related terms of t, n _tr represents the number of related terms of t, α and β are Dirichlet hyperparameters, α, β are the vectors corresponding to α and β, respectively.

The potential topic ratio θd in a certain text _d in the text dataset is expressed as:

值表示为：The term probability contained in the topic numbered k

The value is represented as:

Further, include:

In the step (5), including:

(51) to described DR-Phrase LDA model training, the training step comprises:

Input: a text dataset of mixed expressions of words and phrase sets that are not matched into valid phrases, the trained word vector model and the Dirichlet distribution hyperparameter α of the DR-Phrase LDA model, and the Dirichlet distribution hyperparameter β , the number of topics K; the number of iterations IterNum; the first γ corresponding values of the phrase w ^p with the largest semantic similarity; the length count adjustment parameter μ of the phrase w ^p ;

Training process:

Traverse the number corresponding to the word or valid phrase t in each text in the text dataset;

If t is a valid phrase, increase the count C(w ^p ) after the phrase w ^p is drawn to topic k; at the same time, traverse the first γ word set with the largest semantic similarity with w ^p , and increase all the words in the word set accordingly. word assigned to count C( _wi ) under topic k;

在主题k下的计数

Otherwise, if t is a content word and there is a semantically related phrase, increase the count C(w) of the word t in topic k, and add the word as a valid phrase for semantic context

count under topic k

Otherwise, if t is a function word, the corresponding count is decremented by 1;

Iterate the above steps to the set number of times IterNum;

Output: A two-dimensional matrix z of topic numbers for all words and valid phrases in the text dataset;

(52) Output potential topic phrases of text data according to the training results, including:

According to the potential topic ratio θ _d in a certain text d, the probability value matrix θ of the topic ratio of each text can be obtained:

θ={θ _{(m, k)} , m∈{0,..M-1}, k∈{0,..K-1}}

Among them, M is the total number of texts in the text dataset; K is the number of topics.

值统计可得每个主题与term的概率值矩阵

According to the probability of the term contained in the topic numbered k

Value statistics can get the probability value matrix of each topic and term

where N _t is the total number of terms in the text dataset.

Further, include:

The valid phrase w ^p is drawn to the count C(w ^p ) of topic k, expressed as:

C(w ^p )=μlen(w ^p )

Wherein, μ is an integer greater than or equal to 1, and the adjustment parameter len(w ^p ) is the length of the effective phrase w ^p .

The phrase semantically related words _wi corresponding to the valid phrase w ^p are assigned to the count C( _wi ) under topic k, which is expressed as:

Among them, Int() represents an integer, Sim( _wi w ^p ) represents the similarity between word _wi and valid phrase w ^p calculated by word vector;

The count C(w) of the word w in topic k is expressed as:

表示在词语w对应的词向量与词语对应的有效短语

对应的词向量

之间的相似度；in,

Represents a valid phrase corresponding to the word vector corresponding to the word w and the word

Corresponding word vector

similarity between

在主题k下的计数

表示为：The effective phrase with the word w as the semantic background

count under topic k

Expressed as:

The invention also discloses a system for analyzing potential topic phrases of text data, the system comprising:

a preprocessing module, used to collect a text data set, and perform word segmentation on the text data set to obtain a word representation of the text data set;

The phrase extraction module is used for extracting valid phrases formed by collocation of words according to the words in the text data set, and obtaining a mixed expression form of the words and phrase sets that are not matched into valid phrases;

The word vector building module is used to obtain the corresponding word vector model after word vector training is performed on the text data set with mixed representation;

The model building module is used to build a word vector-based phrase topic model DR-Phrase LDA and solve each parameter;

A result output module is used to train the DR-Phrase LDA, and output potential topic phrases of the text data according to the training results.

Further, include:

The phrase extraction module includes:

The candidate set statistical unit is used to count the co-occurrence frequency of two-word or phrase collocations in the text data set to form a binary phrase candidate set;

The binary phrase calculation unit is used to calculate the score ( _wi , w _j ) of the candidate set of binary phrases, select the top m with high scores to form the formal binary phrases, and add them to the phrase set. Update the corresponding words in the word representations of the text data set described by the statistical unit as the obtained phrases;

The phrase set generation unit, the iterative candidate set statistics unit and the binary phrase calculation unit, and the calculated n-gram phrases composed of the binary phrase and other words or phrases are added to the phrase set in turn.

Further, include:

其中，所有文本共享K个主题；从词语的多项式分布

中采样生成词语或短语t。In the model building module, the DR-Phrase LDA model is a probability generation model, and the process of realizing text generation by DR-Phrase LDA is: first, there are M texts in the text data set D, from the Dirichlet distribution whose hyperparameter is α. Sampling to generate topic distribution θ _d of text d; sampling from multinomial distribution θ _d of topics to generate topic parameters z of words or phrases in text d, and the topic number is denoted as z _mn ; sampling from hyperparameter β Dirichlet distribution to generate distribution of words or phrases corresponding to topic z

where all texts share K topics; from the multinomial distribution of words

Sampling produces a word or phrase t.

Further, include:

The result output module, the specific training includes:

A training unit for training the DR-Phrase LDA model, specifically including:

Input: a text dataset of mixed expressions of words and phrase sets that are not matched into valid phrases, the trained word vector model and the Dirichlet distribution hyperparameter α of the DR-Phrase LDA model, and the Dirichlet distribution hyperparameter β , the number of topics K; the number of iterations IterNum; the first γ corresponding values of the phrase w ^p with the largest semantic similarity; the length count adjustment parameter μ of the phrase w ^p ;

Training process:

Traverse the number corresponding to the word or valid phrase t in each text in the text dataset;

If t is a valid phrase, increase the count C(w ^p ) after the phrase w ^p is drawn to topic k; at the same time, traverse the first γ word set with the largest semantic similarity with w ^p , and increase all the words in the word set accordingly. word assigned to count C( _wi ) under topic k;

在主题k下的计数

Otherwise, if t is a content word and there is a semantically related phrase, increase the count C(w) of the word t in topic k, and add the word as a valid phrase for semantic context

count under topic k

Otherwise, if t is a function word, the corresponding count is decremented by 1;

Iterate the above steps to the set number of times IterNum;

Output: A two-dimensional matrix z of topic numbers for all words and valid phrases in the text dataset;

Result output unit: used to output potential topic phrases of text data according to the training results, including:

According to the potential topic ratio θ _d in a certain text d, the probability value matrix θ of the topic ratio of each text can be obtained:

θ={θ _{(m, k)} , m∈{0,..M-1}, k∈{0,..K-1}}

Among them, M is the total number of texts in the text dataset; K is the number of topics.

k值统计可得每个主题与term的概率值矩阵

According to the probability of the term contained in the topic numbered k

The probability value matrix of each topic and term can be obtained by k-value statistics

where N _t is the total number of terms in the text dataset.

Beneficial effects: the present invention adopts a phrase topic model based on word vectors, which reasonably improves the statistical information of phrases in model training with the help of Chinese linguistic laws in the training of probabilistic topic models, and specifically adopts the method of word vectors to measure the difference between phrase components and words. to quantitatively reflect the semantic relationship of words in the whole text and local phrases, which makes the model more accurate; at the same time, this method considers phrase linguistic characteristics such as phrase length, context, and part of speech, and the extracted topic phrases are readable. The phrase topic model with stronger semantic expression ability constructed by this method is more in line with language usage habits, and the result has wider application value.

Description of drawings

Fig. 1 is the method flow chart of the present invention;

Fig. 2 is the DR-Phrase LDA model schematic diagram of the present invention;

3 is a schematic diagram of the system structure according to the present invention;

Figure 4 shows the consistency index values of the four models described in the present invention under different themes.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present invention provides a method for analyzing potential topic phrases of text data, as shown in Figure 1, comprising the following steps:

S1 collects a text data set, and performs word segmentation on the text data set to obtain a word representation of the text data set.

S2 extracts valid phrases formed by collocation of words according to the words in the text data set, and obtains a mixed expression form of words and phrase sets that are not matched into valid phrases.

This step specifically includes:

S21 counts the co-occurrence word frequencies of bigrams to form a candidate set of bigram phrases;

S22 calculates the score(w _i , w _j ) of the candidate set of binary phrases, selects the top m with high scores to form formal binary phrases, and adds them to the phrase set. The score( _wi , _wj ) of a bigram phrase is specifically calculated by the following formula:

In Equation 1, _wi and w _j are the binary phrase word pairs captured from the corpus, count( _wi , w _j ) is the co-occurrence frequency of words _wi and w _j , and count( _wi ) is the word w _i word frequency, count(w _j ) is the word frequency of word w _j , and the function of parameter η is to filter low-frequency phrases. For example, n=5 can filter out word pairs whose co-occurrence frequency is lower than 5 ( _wi , w _j ).

In S23, steps S21 and S22 are repeated 3-4 times, and longer phrases are synthesized by cyclically synthesizing binary phrases, which are collectively referred to as n-gram phrases. In the embodiment of the present invention, binary phrases, ternary phrases, quaternary phrases and Five-element phrases, in each synthesis process, continuously reduce the value of η to obtain longer phrases to be extracted.

S3 uses the open source package word2vec to train the word vector to obtain the word vector model of the text data set.

S4 builds a word vector-based phrase topic model DR-Phrase LDA and solves each parameter.

Specifically: S41 builds the DR-Phrase LDA model:

后，为该主题抽取一个term t；所有文本共享K个主题；document-topic and topic-term分布均为多项式分布，并且它们均服从先验为对称的狄利克雷分布(Dirichlet distribution)，αandβ为Dirichlet先验的超参数，DR-PhraseLDA模型的文本数据集生成过程可描述为如表1所示。DR-Phrase LDA is a probabilistic generative model, and its graphical model representation is shown in Figure 2. In Figure 2, the text dataset D has M texts, and N _d represents the number of words or phrases (terms) contained in the text d. The specific process of generating a term in the text includes two steps. First, from the document-topic ( A distribution θ is randomly drawn from the document-topic) distribution; a distribution is drawn from the topic-term (topic-term) distribution

After that, a term t is extracted for the topic; all texts share K topics; document-topic and topic-term distributions are multinomial distributions, and they all obey the prior symmetric Dirichlet distribution, α and β are The hyperparameters of the Dirichlet prior, the text dataset generation process of the DR-PhraseLDA model can be described as shown in Table 1.

Table 1 Text dataset generation process of DR-Phrase LDA model

S42 parameter solver

The focus of the DR-Phrase LDA model is to solve the hidden variable topic number z _mn for each term of each text d in the text dataset. Other parameters can be obtained by indirect calculation. For the hidden variable z, the approximate solution method of Gibbs sampling is used to complete. The main core of the sampling process is to deal with the low frequency problem of phrase statistics. According to the principle of the Polya jar model, the weight of phrases is increased in the sampling process, which mainly combines the following linguistic characteristics of phrases:

(1) The semantic strength of a phrase is related to the length of the phrase: in general, long phrases express stronger semantics;

(2) The overall semantics of a phrase is related to the context in which it is located: except that the semantics of some phrases are related to the constituent words of the phrase, in many cases, due to the non-semantic compositionality of language, the semantics of phrases, like ordinary words, need to be based on the context. understand its meaning;

(3) Content words play a major role in semantic expression: according to their grammatical functions and properties, they can be divided into two categories: content words and function words. Function words generally do not contain real word meanings, and their basic purpose is to express grammatical relations. Function words mainly refer to adverbs, conjunctions, prepositions, auxiliary words, interjections and onomatopoeia. Content words have real meaning, mainly referring to nouns, verbs, adjectives, numerals, quantifiers, etc. Regardless of phrases or vocabulary, content words are the core of semantic expression and play a dominant role in semantic analysis.

According to the above three characteristics, combined with the principle of the Polya jar model, the semantic status of the phrase can be more reflected in the sampling process by designing an appropriate strategy. Specifically, the following methods are used:

S421 The sampling process reaches a certain position and encounters a phrase w ^p , and at the same time increases the count of the phrase and its semantically related words. The specific method is to calculate from the trained word vector model the top γ with the largest semantic similarity to the valid phrase w ^p . The word set, as the semantic context background of the phrase w ^p , then: after the phrase w ^p is drawn to topic k, the count C(w ^p ) of w ^p in k is increased by the following formula:

C(w ^p )=μlen(w ^p ) Equation 2

Wherein, μ>=1, which is an integer, and the adjustment parameter len(w ^p ) is the length of the phrase w ^p .

At the same time, the semantically related words w _i of the phrase are added, and the count C( _wi ) under topic k adopts the following formula:

In Equation 3, Int() represents an integer, and Sim( _wi w ^p ) represents the similarity calculated by word vector between word _wi and phrase w ^p . The word vectors _of w _i and w ^p are respectively represented as w _i ⁼ (wi _,0 ,wi _,1 ,...,wi _,n ),wp ⁼ ( ^wp0 , _wp1 . ....., w ^p _n ), the similarity is calculated using the cosine formula as follows:

S422 The sampling process reaches a certain position and encounters a word w. If the word is a function word, it is counted according to normal sampling; if the word is a real word and is a topγ semantically related word of some phrases, the word and its The count of semantically related phrases, the word w is drawn to topic k's incremental count word C(w) by the following formula:

In Equation 5, the word w is the contextual semantic background of m phrases.

在主题k下的计数

采用如下公式：At the same time, add all the word w as the semantic background phrase

count under topic k

Use the following formula:

The symbols in Equation 6 have the same meanings as above. Equations 2 and 6 both act to increase the count of phrases in the Gibbs sampling process, and have similarities; however, considering that the count increase caused by the phrase itself and the count increase caused by the context word should be different, the former's count increase should be different. The contribution to the topic semantics of the phrase is more direct and the contribution is greater. Therefore, the word vector similarity value is added to the formula 6 to measure the strength to ensure that the increase in the count is associated with the semantic relevance, making it more realistic.

Through formula 2 to formula 6, the newly designed Polya jar model not only increases the probability that the registered color of each drawn ball is put back into the jar to be drawn again, but also increases the probability that the ball related to the ball is drawn again. Increase.

According to the above strategy, the DR-Phrase LDA model is obtained. The Gibbs sampling formula for solving the topic number of each term in the text dataset can be expressed as follows:

In formula 7, t represents a word or phrase, is the current position term of document d in the sampling process, k represents the assigned topic number, K is the preset number of topics, N _t is the total number of terms in the corpus, n _k/d represents the count of topic k in document d, n _t/k represents the count of t in topic k, n _r represents the number of semantically related terms of t, n _tr represents the number of related terms of t, α and β are Dirichlet hyperparameters .

The Gibbs Sampling of the DR-Phrase LDA model mainly improves the classical LDA Gibbs Sampling process through formulas 2-6 constructed under different strategies. In the process of Gibbs Sampling, the term is assigned the count of topics in different situations, which is mainly reflected in the second half of formula 7. Phrase-to-word and word-to-phrase sampling processes have four different counting promotion strategies in four cases. For convenience of expression, the simplified notation of Equation 7 is used uniformly. In the actual program implementation process, it is necessary to judge and count according to the specific situation.

S423 other parameters to solve:

The potential topic ratio θd in a text _d in the text dataset is calculated by the following formula:

值通过以下公式计算：The term probability contained in topic k

The value is calculated by the following formula:

S5 trains the DR-Phrase LDA, and outputs potential topic phrases of the text data according to the training results. Specifically include:

S51 trains the DR-Phrase LDA model according to the following algorithm

Input: the mixed representation of words and phrases of the text data set obtained by S2; the word vector model of the text data set obtained by S3; α (Dirichlet distribution hyperparameter), β (Dirichlet distribution hyperparameter), K( The number of topics); IterNum (the number of iterations); γ (the top value of the phrase w ^p with the largest semantic similarity); μ (the length count adjustment parameter of the phrase w ^p ).

Output: A two-dimensional matrix z of topic numbers for all terms in the text dataset.

Training process:

(1) Initialization: For each term in each text in the text dataset, randomly select a topic number from (1~K);

(2) Gibbs sampling process

Increase the count of t semantically related phrases assigned to topic k according to formula 6; other statistics are increased accordingly;

S52 outputs the potential topic phrases of the text data according to the training results

S521 according to formula 8 statistics can obtain the subject ratio probability value matrix θ of each text for output;

θ={θ _{(m, k)} , m∈{0,..M-1}, k∈{0,..K-1}}

Among them, M is the total number of texts in the text dataset; K is the number of topics.

并进行输出；S522 According to formula 9, the probability value matrix of each topic and term can be obtained by statistics

and output;

where N _t is the total number of terms in the text dataset.

Based on the same technical concept, the present invention provides a system implemented by a method for analyzing potential topic phrases in text data, as shown in FIG. 3 , the system includes:

A preprocessing module 1 is used to collect a text data set, and perform word segmentation on the text data set to obtain a word representation of the text data set;

The phrase extraction module 2 is used for extracting valid phrases formed by collocation of words according to the words in the text data set, and obtaining a mixed expression form of the words and phrase sets that are not matched into valid phrases.

The phrase extraction module 2 specifically includes:

The candidate set statistics unit 21 is used to count the co-occurrence word frequencies of the words collocated in the binary phrases to form a binary phrase candidate set;

The binary phrase calculation unit 22 is used to calculate the score ( _wi , w _j ) of the candidate binary phrase set, select the top m with high scores to form a formal binary phrase, and add it to the phrase set.

The score( _wi , _wj ) of a bigram phrase is specifically calculated by the following formula:

In Equation 1, _wi and w _j are the binary phrase word pairs captured from the corpus, count( _wi , w _j ) is the co-occurrence frequency of words _wi and w _j , and count( _wi ) is the word w _i word frequency, count(w _j ) is the word frequency of word w _j , and the function of parameter η is to filter low-frequency phrases. For example, n=5 can filter out word pairs whose co-occurrence frequency is lower than 5 ( _wi , w _j ).

The phrase set generating unit 23 is used for combining the binary phrases obtained by the iterative calculation with the ternary phrases formed by collocations with words, the quaternary phrases formed by the ternary phrases with words and collocations, and the quintuple phrases formed by collocations of the quaternary phrases and words Phrases are sequentially added to the phrase set; the valid phrases include binary phrases, tri-grams, quaternary phrases, and pentaphrases.

The word vector building module 3 is used for word vector training using the open source package word2vec for the text data set with mixed expressions, and the word vector model of the text data set is obtained.

The model building module 4 is used to build a word vector-based phrase topic model DR-Phrase LDA and solve each parameter.

后，为该主题抽取一个term t；所有文本共享K个主题；document-topic and topic-term分布均为多项式分布，并且它们均服从先验为对称的狄利克雷分布(Dirichlet distribution)，αandβ为Dirichlet先验的超参数，DR-PhraseLDA模型的文本数据集生成过程可描述为如表1所示。DR-Phrase LDA model building unit 41, DR-Phrase LDA is a probability generation model, the text data set D has M texts, N _d represents the number of words or phrases (terms) contained in the text d, and specifically generates a certain term in the text The process consists of two steps. First, a distribution θ is randomly selected from the document-topic (document-topic) distribution; a distribution θ is selected from the topic-term (topic-term) distribution.

After that, a term t is extracted for the topic; all texts share K topics; document-topic and topic-term distributions are multinomial distributions, and they all obey the prior symmetric Dirichlet distribution, α and β are The hyperparameters of the Dirichlet prior, the text dataset generation process of the DR-PhraseLDA model can be described as shown in Table 1.

Table 1 Text dataset generation process of DR-Phrase LDA model

Parameter solving unit 42: The DR-Phrase LDA model focuses on solving the hidden variable topic number z _mn of each term of each text d in the text data set. Other parameters can be obtained by indirect calculation. For the hidden variable z, the approximate solution method of Gibbs sampling is used to complete it. The main core of the sampling process is to deal with the low frequency problem of phrase statistics. According to the principle of the Polya jar model, the weight of phrases is increased in the sampling process, which mainly combines the following linguistic characteristics of phrases:

(1) The semantic strength of a phrase is related to the length of the phrase: in general, long phrases express stronger semantics;

(2) The overall semantics of a phrase is related to the context in which it is located: except that the semantics of some phrases are related to the constituent words of the phrase, in many cases, due to the non-semantic compositionality of language, the semantics of phrases, like ordinary words, need to be based on the context. understand its meaning;

(3) Content words play a major role in semantic expression: according to their grammatical functions and properties, they can be divided into two categories: content words and function words. Function words generally do not contain real word meanings, and their basic purpose is to express grammatical relations. Function words mainly refer to adverbs, conjunctions, prepositions, auxiliary words, interjections and onomatopoeia. Content words have real meaning, mainly referring to nouns, verbs, adjectives, numerals, quantifiers, etc. Regardless of phrases or vocabulary, content words are the core of semantic expression and play a dominant role in semantic analysis.

According to the above three characteristics, combined with the principle of the Polya jar model, the semantic status of the phrase can be more reflected in the sampling process by designing an appropriate strategy. Specifically, the following methods are used:

When the sampling process reaches a certain position and encounters a phrase w ^p , the count of the phrase and its semantically related words is increased at the same time. The specific method is to calculate from the trained word vector model the top γ with the largest semantic similarity to the valid phrase w ^p . The word set, as the semantic context background of the phrase w ^p , then: after the phrase w ^p is drawn to topic k, the count C(w ^p ) of w ^p in k is increased by the following formula:

C(w ^p )=μlen(w ^p ) Equation 2

Wherein, μ>=1, which is an integer, and the adjustment parameter len(w ^p ) is the length of the phrase w ^p .

At the same time, the semantically related words w _i of the phrase are added, and the count C( _wi ) under topic k adopts the following formula:

In Equation 3, Int() represents an integer, and Sim( _wi w ^p ) represents the similarity calculated by word vector between word _wi and phrase w ^p . The word vectors _of w _i and w ^p are respectively represented as w _i ⁼ (wi _,0 ,wi _,1 ,...,wi _,n ),wp ⁼ ( ^wp0 , _wp1 . ....., w ^p _n ), the similarity is calculated using the cosine formula as follows:

If the sampling process reaches a certain position and encounters a certain word w, if the word is a function word, it will be counted according to normal sampling; if the word is a real word and is a topγ semantically related word of some phrases, the word and its associated words will be promoted at the same time. The count of semantically related phrases, the word w is drawn to topic k's incremental count word C(w) by the following formula:

In Equation 5, the word w is the contextual semantic background of m phrases.

在主题k下的计数

采用如下公式：At the same time, add all the word w as the semantic background phrase

count under topic k

Use the following formula:

The symbols in Equation 6 have the same meanings as above. Equations 2 and 6 both act to increase the count of phrases in the Gibbs sampling process, and have similarities; however, considering that the count increase caused by the phrase itself and the count increase caused by the context word should be different, the former's count increase should be different. The contribution to the topic semantics of the phrase is more direct and the contribution is greater. Therefore, the word vector similarity value is added to the formula 6 to measure the strength to ensure that the increase in the count is associated with the semantic relevance, making it more realistic.

Through formula 2 to formula 6, the newly designed Polya jar model not only increases the probability that the registered color of each drawn ball is put back into the jar to be drawn again, but also increases the probability that the ball related to the ball is drawn again. Increase.

According to the above strategy, the DR-Phrase LDA model is obtained. The Gibbs sampling formula for solving the topic number of each term in the text dataset can be expressed as follows:

In formula 7, t represents a word or phrase, is the current position term of document d in the sampling process, k represents the assigned topic number, K is the preset number of topics, N _t is the total number of terms in the corpus, n _k/d represents the count of topic k in document d, n _t/k represents the count of t in topic k, n _r represents the number of semantically related terms of t, n _tr represents the number of related terms of t, α and β are Dirichlet hyperparameters .

The Gibbs Sampling of the DR-Phrase LDA model mainly improves the classical LDA Gibbs Sampling process through formulas 2-6 constructed under different strategies. In the process of Gibbs Sampling, the term is assigned the count of topics in different situations, which is mainly reflected in the second half of formula 7. Phrase-to-word and word-to-phrase sampling processes have four different counting promotion strategies in four cases. For convenience of expression, the simplified notation of Equation 7 is used uniformly. In the actual program implementation process, it is necessary to judge and count according to the specific situation.

The potential topic ratio θd in a text _d in the text dataset is calculated by the following formula:

值通过以下公式计算：The term probability contained in topic k

The value is calculated by the following formula:

The result output module 5 is used to train the DR-Phrase LDA, and output potential topic phrases of the text data according to the training results.

Specific training includes:

The training unit 51 is used to train the DR-Phrase LDA model, and specifically includes:

Input: mixed representation of words and phrases based on the resulting text dataset; word vector model for the text dataset; α (Dirichlet distribution hyperparameter), β (Dirichlet distribution hyperparameter), K (number of topics) ; IterNum (the number of iterations); γ (the top value of the phrase w ^p with the largest semantic similarity); μ (the length count adjustment parameter of the phrase w ^p ).

Output: A two-dimensional matrix z of topic numbers for all terms in the text dataset.

Training process: Initialization: For each term in each text in the text dataset, randomly select a topic number from (1~K);

Gibbs sampling process

Increase the count of t semantically related phrases assigned to topic k according to formula 6; other statistics are increased accordingly;

The result output unit 52 is used to output potential topic phrases of the text data according to the training results, specifically including:

According to formula 8, the subject ratio probability value matrix θ of each text can be obtained for output;

θ={θ _{(m, k)} , m∈{0,..M-1}, k∈{0,..K-1}}

Among them, M is the total number of texts in the text dataset; K is the number of topics.

并进行输出；According to formula 9, the probability value matrix of each topic and term can be obtained

and output;

where N _t is the total number of terms in the text dataset.

In order to verify the validity of the present invention, carry out simulation experiment with concrete data:

The test selected the interpretive texts of TCM diseases in the national standard - the disease section of clinical diagnosis and treatment of traditional Chinese medicine (GBT16751.1-1997), as data, a total of 20 categories and 978 explanatory texts of diseases, and the relevant information is shown in Table 2.

Table 2 TCM clinical diagnosis and treatment terms disease categories

Using python's open source word segmentation software jieba version 0.39 to segment the text data set. And extract the effective phrases in the text data set to obtain the mixed representation of words and phrases of the text data set; then, use the open source package word2vec for word vector training to obtain the word vector model of the text data set;

Specifically: The Google team launched the word vector (Distributed Representation) open source package word2ve based on neural network in 2013 to train the word vector on the text data set. During training, specify the minimum frequency of occurrence of words in the training set: min_count=5; word vector dimension: size=50. Other parameters are default parameters.

Finally, train the DR-Phrase LDA model and output the final topic phrase.

In order to evaluate the topic quality of the DR-Phrase LDA topic model provided by the present invention, the following two models are used as comparison benchmarks: 1) standard LDA model hereinafter referred to as LDA); 2) LDA based on phrase bag (hereinafter referred to as Phrase-LDA); 3) It is close to the GPU model of the present invention (Generalized Pólya Urn, hereinafter referred to as GPU-LDA). The evaluation is carried out using indicators for evaluating readability and consistency, and the calculation formula is as follows:

表示主题t的一些列概率高的前M个词。Among them, D(v) represents the document frequency in which the word v has appeared; D(v, v') represents the document frequency in which the word v and v'co-occur; M represents the top M words with the highest probability of topic t;

A list of top M words with high probability representing topic t.

In the experiment, α=50/K, and β=0.01, and only common binary and ternary phrases are extracted in order to save efficiency and computational complexity. Figure 4 shows the consistency indicators of each topic calculated by the four models of LDA, Phrase-LDA, GPU-LDA and DR-Phrase LDA proposed in this study according to the above indicators. The ordinate is Topic Coherence, the consistency index value of the four models in each category, and the abscissa 1-20 represents the 20 disease categories of the experimental data set. As can be seen from the figure, Phrase-LDA has the worst consistency index values, ranging from -2400 to -2600; LDA modes are approximately between -2200 and -2400; while GPU-LDA and DR-Phrase LDA Around -1800 to -2100; it should be noted that the Topic Coherence metric is negative, and the smaller the absolute value, the higher the topic quality.

As can be seen from Figure 4, the topic quality of GPU-LDA and DR-Phrase LDA is significantly better than that of LDA model and Phrase-LDA. Although the Phrase-LDA model replaces the traditional "Bag-of-Word" (BOW) with "Phrase-of-Word" (POW); but the traditional topic model theory assigns the same topic to the whole phrase for co-occurrence capture will cause a decrease in phrase co-occurrence information. Therefore, the common information of phrases in the Phrase-LDA model is much lower than that of a single word, resulting in a very low probability of a large number of phrases in the topic, so the topic consistency of Phrase-LDA is lower than that of LDA, and the readability and consistency of the topic didn't help.

On the contrary, GPU-LDA and the DR-Phrase LDA proposed in this study consider the strong semantic expression characteristics of phrases, and deliberately improve the statistics of phrases and their component words in the model; therefore, Figure 4 reflects that the topic quality of both has been improved. improve. The GPU-LDA model improves the statistics of phrases and their components at the same time; in contrast, the DR-Phrase LDA model takes into account the different parts of speech and the semantics of the words in the process of improving the statistics of phrases and their components. , especially the semantics of the constituent words of many phrases are completely different from the overall semantics of the phrases, and this processing strategy is more in line with the reality of natural language. It is also reflected in Figure 4 that the topic quality of the DR-Phrase LDA model is better than that of the GPU-LDA model.

Table 3 lists the Top 5 Term with the highest probability of training results for the four topics of the four models. As can be seen from the table, the training results of the four models can distinguish the central semantics of each topic as a whole. The Top 5 Term results of the LDA and Phrase LDA models are all words; the GPU-LDA model has both phrases and words, but words Relatively many points, while DR-Phrase LDA is just the opposite, with more phrases than words. LDA is a complete "word bag" model; Phrase LDA is based on phrase bag, but due to the same statistical status of phrases and general words, a large number of phrases have less co-occurrence information, so it is difficult to obtain a high probability and fails to reflect the strong phrases. The role of semantic blocks. The GPU-LDA model adopts a more general jar model (GPU) to increase the connections between constituent words and phrases.

According to the principle of GPU, observed words are put back into jars to improve the probability of phrases. This model can be understood as the rich get richer. The method adopts two strategies to construct a phrase topic model: when a word is assigned a topic, the counts of all phrases containing the word are increased; when a phrase is assigned a topic, the counts of all the constituent words of the phrase are increased. Obviously, the phenomenon of synonyms and polysemous words is very common in natural language, especially in Chinese, and the probability of enhancing phrases by simply increasing the count will cause the problem of over-enhancement that does not conform to reality. GPU-LDA not only improves the statistics of phrases, but also improves the statistics of phrase components. In the model test results of GPU-LDA, it is found that the component words of phrases are improved more than phrases. For example, in the results of topic Topic 5, "brain" "Brain disease": the probability of "disease" is higher than that of "brain disease".

Table 3 Examples of Top5 Term analysis results of some of the four models

The word vector-based DR-Phrase LDA phrase topic model proposed by this method uses the word vector method to measure the relationship between phrase components to quantitatively reflect the semantic contribution of words in the whole document and the phrase part, so as to improve the phrase topic model. the quality of the analysis results. In the results of the DR-Phrase LDA model, the overall semantics of phrases are generally stronger than their constituent words. For example, the probability of "mental stimulation" in Topic 5 is higher than that of "spirit", and the probability of "shen qi" in Topic 29 is higher than that of "kidney".

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Modifications or equivalent replacements are made to the specific embodiments of the present invention, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (4)

1. a method for analyzing text data potential topic phrases, it is characterised in that the method comprises: (1) Collecting a text data set, and performing word segmentation on the text data set to obtain the word representation of the text data set; (2) According to the words in the text data set, the effective phrases formed by the collocation of the words are extracted, and the mixed expression form of the words that are not collocated into the effective phrases and the phrase set is obtained; (3) The corresponding word vector model is obtained after word vector training is performed on the text data set of mixed expressions; (4) Construct a phrase topic model DR-Phrase LDA based on word vector and solve each parameter; In the step (4), the DR-Phrase LDA model is a probability generation model, and the process of realizing text generation by DR-Phrase LDA is: first, there are M texts in the text data set D, from the Dirichlet distribution whose hyperparameter is α The topic distribution θ _d of the generated text d is sampled from ; the topic parameter z of the words or phrases in the text d is generated by sampling from the multinomial distribution θ _d of the topic, and the topic number is denoted as z _mn ; the hyperparameter is β Dirichlet distribution. Generate a distribution of words or phrases corresponding to topic z

where all texts share K topics; from the multinomial distribution of words

medium sampling to generate a word or phrase t; The calculation of the topic parameter z of the word or phrase is completed using the Gibbs sampling approximate solution method, which is expressed as:

Among them, the word or valid phrase in the current position of document d in the sampling process is denoted as t, denoted as term, k is the assigned topic number, K is the preset number of topics, N _t is the total number of terms in the text data set, n _k/d represents the count of topic k in document d, n _t/k represents the count of t in topic k, n _r represents the number of semantically related terms of t, n _tr represents the number of related terms of t, α and β are Dirichlet hyperparameters, α, β are the vectors corresponding to α and β, respectively; The potential topic ratio θd in a certain text _d in the text dataset is expressed as:

The term probability contained in topic k

The value is represented as:

(5) the DR-Phrase LDA is trained, and the potential subject phrase of the text data is output according to the training result; In step (5), including: (51) to described DR-Phrase LDA model training, the training step comprises: Input: a text dataset of mixed expressions of words and phrase sets that are not matched into valid phrases, the trained word vector model and the Dirichlet distribution hyperparameter α of the DR-Phrase LDA model, and the Dirichlet distribution hyperparameter β , the number of topics K; the number of iterations IterNum; the first γ corresponding values of the phrase w ^p with the largest semantic similarity; the length count adjustment parameter μ of the phrase w ^p ; Training process: Traverse the number corresponding to the word or valid phrase t in each text in the text dataset; If t is a valid phrase, increase the count C(w ^p ) after the phrase w ^p is drawn to topic k; at the same time, traverse the first γ word sets with the largest semantic similarity with w ^p , and increase all the words in the word set accordingly. word assigned to count C( _wi ) under topic k; Otherwise, if t is a content word and there is a semantically related phrase, then increase the count C(w) of the word t in topic k, and add the word as a valid phrase for the semantic context

count under topic k

Otherwise, if t is a function word, the corresponding count is decremented by 1; Iterate the above steps to the set number of times IterNum; Output: A two-dimensional matrix z of topic numbers for all words and valid phrases in the text dataset; (52) Output potential topic phrases of text data according to the training results, including: According to the potential topic ratio θ _d in a certain text d, the probability value matrix θ of the topic ratio of each text can be obtained: θ={θ _(m,k) ,m∈{0,..M-1},k∈{0,..K-1}} Among them, M is the total number of texts in the text dataset; K is the number of topics; According to the probability of the term contained in the topic numbered k

Value statistics can get the probability value matrix of each topic and term

Among them, N _t is the total number of terms in the text dataset; The valid phrase w ^p is drawn to the count C(w ^p ) of topic k, expressed as: C(w ^p )=μlen(w ^p ) Among them, μ is an integer greater than or equal to 1, and the adjustment parameter len(w ^p ) is the length of the valid phrase w ^p ; The phrase semantically related words _wi corresponding to the valid phrase w ^p are assigned to the count C( _wi ) under topic k, which is expressed as:

Among them, Int() represents an integer, Sim( _wi w ^p ) represents the similarity between word _wi and valid phrase w ^p calculated by word vector; The count C(w) of the word w in topic k is expressed as:

in,

Represents a valid phrase corresponding to the word vector corresponding to the word w and the word

Corresponding word vector

similarity between Valid phrases with word w as semantic background

count under topic k

Expressed as:

2. The method for analyzing text data potential topic phrases according to claim 1, wherein in the step (2), the effective phrases include n-gram phrases, and n-grams are the number of words that form a phrase, and the Describe the effective phrases formed by extracting word collocations according to the words in the text data set, including: (21) Count the co-occurrence frequencies of two-word or phrase collocations in the text data set to form a binary phrase candidate set; (22) Calculate the score(w _i ,w _j ) score of the candidate set of binary phrases, select the top m with high scores to form formal binary phrases, and add them to the phrase set. Update the corresponding words in the word representations of the text data set to be the obtained phrases; (23) An n-gram phrase formed by collocation of the binary phrase and other words or phrases calculated in the iterative steps (21) and (22) is added to the phrase set in turn. 3. A system implemented by the method for analyzing potential topic phrases of text data according to any one of claims 1-2, wherein the system comprises: a preprocessing module, used to collect a text data set, and perform word segmentation on the text data set to obtain a word representation of the text data set; The phrase extraction module is used for extracting valid phrases formed by collocation of words according to the words in the text data set, and obtaining a mixed expression form of the words and phrase sets that are not matched into valid phrases; The word vector building module is used to obtain the corresponding word vector model after word vector training is performed on the text data set with mixed representation; The model building module is used to build a word vector-based phrase topic model DR-Phrase LDA and solve each parameter; The DR-Phrase LDA model is a probabilistic generation model. The process of DR-Phrase LDA to realize text generation is as follows: First, there are M texts in the text data set D, and the topic distribution of the text d is sampled from the Dirichlet distribution with the hyperparameter α. θ _d ; Sampling from the multinomial distribution θ _d of the topic to generate the topic parameter z of the words or phrases in the text d, and the topic number is denoted as z _mn ; Sampling from the hyperparameter β Dirichlet distribution to generate the corresponding words or phrases of the topic z distributed

where all texts share K topics; from the multinomial distribution of words

medium sampling to generate a word or phrase t; The calculation of the topic parameter z of the word or phrase is completed using the Gibbs sampling approximate solution method, which is expressed as:

Among them, the word or valid phrase in the current position of document d in the sampling process is denoted as t, denoted as term, k is the assigned topic number, K is the preset number of topics, N _t is the total number of terms in the text data set, n _k/d represents the count of topic k in document d, n _t/k represents the count of t in topic k, n _r represents the number of semantically related terms of t, n _tr represents the number of related terms of t, α and β are Dirichlet hyperparameters, α, β are the vectors corresponding to α and β, respectively; The potential topic ratio θd in a certain text _d in the text dataset is expressed as:

The term probability contained in the topic numbered k

The value is represented as:

A result output module, for training the DR-Phrase LDA, and outputting potential topic phrases of text data according to the training results; The result output module, the specific training includes: A training unit for training the DR-Phrase LDA model, specifically including: Input: a text dataset of mixed expressions of words and phrase sets that are not matched into valid phrases, the trained word vector model and the Dirichlet distribution hyperparameter α of the DR-Phrase LDA model, and the Dirichlet distribution hyperparameter β , the number of topics K; the number of iterations IterNum; the first γ corresponding values of the phrase w ^p with the largest semantic similarity; the length count adjustment parameter μ of the phrase w ^p ; Training process: Traverse the number corresponding to the word or valid phrase t in each text in the text dataset; If t is a valid phrase, increase the count C(w ^p ) after the phrase w ^p is drawn to topic k; at the same time, traverse the first γ word set with the largest semantic similarity with w ^p , and increase all the words in the word set accordingly. word assigned to count C( _wi ) under topic k; Otherwise, if t is a content word and there is a semantically related phrase, increase the count C(w) of the word t in topic k, and add the word as a valid phrase for semantic context

count under topic k

Otherwise, if t is a function word, the corresponding count is decremented by 1; Iterate the above steps to the set number of times IterNum; Output: A two-dimensional matrix z of topic numbers for all words and valid phrases in the text dataset; Result output unit: used to output potential topic phrases of text data according to the training results, including: According to the potential topic ratio θ _d in a certain text d, the probability value matrix θ of the topic ratio of each text can be obtained: θ={θ _(m,k) ,m∈{0,..M-1},k∈{0,..K-1}} Among them, M is the total number of texts in the text dataset; K is the number of topics; According to the probability of the term contained in the topic numbered k

Value statistics can get the probability value matrix of each topic and term

Among them, N _t is the total number of terms in the text dataset; The valid phrase w ^p is drawn to the count C(w ^p ) of topic k, expressed as: C(w ^p )=μlen(w ^p ) Among them, μ is an integer greater than or equal to 1, and the adjustment parameter len(w ^p ) is the length of the valid phrase w ^p ; The phrase semantically related words _wi corresponding to the valid phrase w ^p are assigned to the count C( _wi ) under topic k, which is expressed as:

Among them, Int() represents an integer, Sim( _wi w ^p ) represents the similarity between word _wi and valid phrase w ^p calculated by word vector; The count C(w) of the word w in topic k is expressed as:

in,

Represents a valid phrase corresponding to the word vector corresponding to the word w and the word

Corresponding word vector

similarity between Valid phrases with word w as semantic background

count under topic k

Expressed as:

4. The system according to claim 3, wherein, in the phrase extraction module, comprising: The candidate set statistical unit is used to count the co-occurrence frequency of two-word or phrase collocations in the text data set to form a binary phrase candidate set; The binary phrase calculation unit is used to calculate the score ( _wi , w _j ) of the candidate set of binary phrases, select the top m with high scores to form the formal binary phrases, and add them to the phrase set. Update the corresponding words in the word representations of the text data set described by the statistical unit as the obtained phrases; The phrase set generation unit, the iterative candidate set statistics unit and the binary phrase calculation unit, and the calculated n-gram phrases composed of the binary phrase and other words or phrases are added to the phrase set in turn.

Images