CN111078875B - Method for extracting question-answer pairs from semi-structured document based on machine learning - Google Patents

Abstract

The invention belongs to the technical field of natural language processing, and in particular relates to a method for extracting question-answer pairs from semi-structured documents based on machine learning. The invention applies a machine learning method to classify by using Apriori for feature selection and a naive Bayesian classification method to obtain the answer sentence in the semi-structured text. The invention combines named entity recognition and dependency syntax analysis theory to convert answer sentences into corresponding question sentences. Named entity recognition adopts the crf+BiLstm neural network model to recognize the entities in the answer sentence and add them to the entities crawled from the web. Syntax analysis reveals the interdependence relationship among the words in the sentence, so that when the question is generated, the word that depends on the entity is replaced, and a reasonable question is obtained. The invention lays a good foundation for constructing question answering systems in the future by extracting high-quality question-answer pairs from semi-structured documents.

Description

A machine learning-based method for extracting question-answer pairs from semi-structured documents

technical field

The invention belongs to the technical field of natural language processing, and in particular relates to a method for extracting question-answer pairs from semi-structured documents based on machine learning.

Background technique

The question-answer pairs of most current domain-limited question answering systems come from interactive knowledge sharing platforms. The data sources for constructing the knowledge base include Baidu Encyclopedia, question-and-answer communities, and domain websites. In the question answering system for cardiology consultation, it is often possible to get accurate answers to the knowledge inside the knowledge base, but it is often difficult to answer the knowledge not included in the knowledge base. Therefore, the construction of the knowledge base directly affects the performance of the question answering system. In medical-related question answering systems, the knowledge in the knowledge base is related to the life safety of patients. However, the knowledge in the interactive knowledge sharing platform is often derived from the answers of the majority of netizens, which lacks an authoritative guarantee. Since question-answer pairs in online communities cannot guarantee the accuracy of knowledge, more authoritative question-answer pairs are needed in the construction of knowledge bases for medical-related question-answer systems.

Obtaining high-quality question-answer pairs is one of the fundamental tasks in building question-answering systems in the medical field. The current method of using the question-answering community as a data source to obtain question-answer pairs cannot guarantee the quality of question-answer pairs. In order to obtain high-quality question-answer pairs, this paper proposes a method for extracting question-answer pairs from cardiac electronic records. Electronic medical records are filled in by professional doctors, and the accuracy of their knowledge is guaranteed, so it can be used as a reliable source of question-and-answer data. The course record, doctor's order and diagnosis and treatment plan in the electronic heart disease case include the doctor's description of the disease, diagnosis, drug introduction and use method and other knowledge. The question-answer pairs extracted from these records will have extremely high accuracy, which can greatly improve the performance of the cardiac question-answer system constructed later.

With the continuous improvement of the level of hospital informatization in our country, a large amount of clinical data has been accumulated. How to effectively use these data has become one of the focuses in the field of data science. At present, methods related to machine learning can be used to extract the required data from unstructured electronic medical records and reorganize them into structured text. Specifically, medical events and time information are extracted. At the same time, by marking the information to be extracted, according to the marking results, the extraction templates can be summarized, and the extraction rules can be generated by rewriting the extraction templates, and these rules can be used to extract the actual information, so as to obtain effective information from unstructured electronic medical records. Extract and organize into a form that can be analyzed and exploited. There is also a context-aware approach to processing electronic medical records of diabetic patients to extract information containing risk factors for heart disease.

Question Generation (QG) aims to create natural questions from a given sentence or paragraph. The key to the success of these methods lies in whether there are well-designed conversion rules from declarative sentences to interrogative sentences. Purely rule-based approaches often rely on deep linguistic knowledge. In addition to adopting various NLP techniques, including term extraction and shallow parsing, it also utilizes linguistic resources such as corpora and ontologies. The improved rule-based system uses a rule-based approach to generate multiple questions of input sentences by over-generating and ranking methods, and then uses a supervised learning-based Ranker to rank them. The method of introducing deep learning borrows from the encoder-decoder neural network model in the field of machine translation, does not rely on handcrafted rules or complex NLP pipelines, and performs end-to-end training through sequence-to-sequence learning to generate interrogative sentences for declarative sentences.

Contents of the invention

The purpose of the present invention is to provide a method for extracting question-answer pairs from semi-structured documents based on machine learning.

The purpose of the present invention is achieved through the following technical solutions: comprising the following steps:

Step 1: Input the collection of semi-structured documents to be extracted, and convert the semi-structured documents in pdf format into txt documents;

Step 2: Use regular expressions to extract the declarative sentences in the txt text, sort the obtained set of declarative sentences according to the documents, and randomly sort the declarative sentences inside the documents; randomly extract some sentences in each document, judge whether the sentences are answer sentences, and mark the answers sentence, construct the training set;

Step 3: Use the Apriori algorithm to extract the frequent itemsets in the set as features on the training set, expand the features through association rules, and obtain the feature set;

Step 4: According to the feature set, express the sentence as a feature vector, input the feature vector of the training set into the naive Bayesian classification model for training, and obtain the trained classification model;

Step 5: Use the trained classification model to classify the unlabeled sentences, extract the answer sentences, and obtain the answer sentence set;

Step 6: Crawl some domain entities through the web crawler; mark some sentences in the answer sentence set, mark the words as BIO, and establish a training set for named entity recognition;

Step 7: Build a BiLstm+crf neural network model and use the training set for training; use the trained model to sequentially label the unlabeled sentences in the answer sentence set, and convert the labels into named entities; combine some domain entities crawled by web crawlers and named entities to get the entity collection;

Step 8: Carry out word segmentation, part-of-speech tagging, and dependency syntactic analysis on the answer sentence set, analyze the relationship between words in the sentence, and obtain the question sentence corresponding to the answer sentence by replacing the entity in the sentence and the word dependent on the entity, and Output question-answer pairs to complete the extraction of question-answer pairs.

The present invention may also include:

In the described step 3, use the Apriori algorithm to extract the frequent itemsets in the set as a feature on the training set, expand the feature by association rules, and obtain the method of feature set specifically as follows:

The sentence set S contains all the extracted statement sentences {s ₁ , s ₂ ,…,s _m }, a total of m items; by using the word segmentation tool to segment the statement sentences, the word set {x ₁ , x ₂ ,…,x _n } is obtained; Traverse all statements and calculate the support of each word, the calculation method is:

Among them, num(x) is the number of sentences containing word x in the sentence set S, and m is the total number of sentences in S; set the threshold K, put the word x with sup(x) greater than K into the feature set; set the association The rule is (x, y), that is, if the word x is a feature, y is also a feature. Initially set all the binary groups as association rules, and calculate the confidence of the association rules:

Where sup(x∪y) is the probability that words x and y appear together in a sentence; set a threshold K ₂ , and use the association rules with confidence greater than the threshold as feature expansion rules, that is, put word y into the feature set.

In the step 5, the trained classification model is used to classify the unmarked sentences, and the answer sentences are extracted, and the method of obtaining the answer sentence set is specifically: according to the obtained feature combination, the sentences in the training set are represented as feature vectors As the input of the naive Bayesian classification model; the feature vector is 1×n dimensional, and n represents n features; if the feature is included in the sentence, the corresponding position is 1; if it is not included in the sentence, the corresponding position is 0; classification The model uses the naive Bayesian classification model, let x={a ₁ ,a ₂ ,…,a _n } be the item to be classified, and the category set is C={y ₁ ,y ₂ }, by calculating P(y ₁ |x ), p(y ₂ |x) to get the probability that x belongs to two categories, and select the category with the highest probability as the category of x to which p(y _k |x)=max{p(y ₁ |x), p(y ₂ |x)}; among them, P(y ₁ |x), p(y ₂ |x) are obtained according to Bayes'theorem; by calculating the probability corresponding to each category, the class with the highest probability can be found, Through the classification model, the statement sentences are classified, the answer sentences in the statement sentences are found, and finally the set of answer sentences is obtained.

通过softmax函数，可得到概率函数

在训练时，最大化似然概率p(y|X)，即损失函数为-log(p(y|X))。Constructing BiLstm+crf neural network model in the described step 7 is specifically: BiLstm+crf neural network model comprises two layers BiLstm layer and crf layer; The One-Hot encoding used by the vector outputs the probability of each category label, and a sentence finally gives P in the form of a matrix, and Pij is the probability that the i-th word is marked as the j-th label; the Crf layer is a conditional random field, The label constraints can be learned from the input sequence, so that it is more accurate in the recognition of multi-category entities. There is a transfer matrix A, and A _ij represents the probability that the i-th label is transferred to the j-th label; therefore, the input sentence sequence X , get the label y, and the final scoring function is:

Through the softmax function, the probability function can be obtained

During training, the likelihood probability p(y|X) is maximized, that is, the loss function is -log(p(y|X)).

The beneficial effects of the present invention are:

The invention provides a method for extracting question-answer pairs from semi-structured documents based on machine learning. The invention applies the method of machine learning to classify by using Apriori for feature selection and naive Bayesian classification method to obtain the answer sentence in the semi-structured text. The invention combines named entity recognition and dependency syntax analysis theory to convert answer sentences into corresponding question sentences. Named entity recognition adopts the crf+BiLstm neural network model to recognize the entities in the answer sentence and supplement them to the entities crawled from the web. Syntax analysis reveals the interdependence relationship among the words in the sentence, so that when the question is generated, the word that depends on the entity is replaced, and a reasonable question is obtained. The invention lays a good foundation for constructing a question-answer system in the future by extracting high-quality question-answer pairs from semi-structured documents.

Description of drawings

Fig. 1 is the general flowchart of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings.

The invention discloses a method for extracting question-answer pairs from semi-structured documents based on machine learning. The method includes data preprocessing module, PDF document parsing, statement sentence sampling and labeling, and obtains sentences in the document and marked training sets; answer sentence extraction module, frequent word set mining and feature expansion in short texts, based on Naive Bayes The sentence classification of the sentence extracts the answer sentence in the sentence; the question sentence generation module, based on the named entity recognition of the BiLstm-crf model, crawls the named entity from the network, and converts the answer sentence into a question sentence by relying on syntactic analysis; Extracting high-quality question-answer pairs from structured documents lays a good foundation for building question-answer systems in the future.

A method for extracting question-answer pairs from semi-structured documents based on machine learning, comprising the following steps:

Step 1: Input the collection of semi-structured documents to be extracted, and convert the semi-structured documents in pdf format into txt documents;

Step 2: Use regular expressions to extract the declarative sentences in the txt text, sort the obtained set of declarative sentences according to the documents, and randomly sort the declarative sentences inside the documents; randomly extract some sentences in each document, judge whether the sentences are answer sentences, and mark the answers sentence, construct the training set;

Step 3: Use the Apriori algorithm to extract the frequent itemsets in the set as features on the training set, expand the features through association rules, and obtain the feature set;

Step 4: According to the feature set, express the sentence as a feature vector, input the feature vector of the training set into the naive Bayesian classification model for training, and obtain the trained classification model;

Step 5: Use the trained classification model to classify the unlabeled sentences, extract the answer sentences, and obtain the answer sentence set;

Step 6: Crawl some domain entities through the web crawler; mark some sentences in the answer sentence set, mark the words as BIO, and establish a training set for named entity recognition;

Step 7: Build a BiLstm+crf neural network model and use the training set for training; use the trained model to sequentially label the unlabeled sentences in the answer sentence set, and convert the labels into named entities; combine some domain entities crawled by web crawlers and named entities to get the entity collection;

Step 8: Carry out word segmentation, part-of-speech tagging, and dependency syntactic analysis on the answer sentence set, analyze the relationship between words in the sentence, and obtain the question sentence corresponding to the answer sentence by replacing the entity in the sentence and the word dependent on the entity, and Output question-answer pairs to complete the extraction of question-answer pairs.

In the described step 3, use the Apriori algorithm to extract the frequent itemsets in the set as a feature on the training set, expand the feature by association rules, and obtain the method of feature set specifically as follows:

The sentence set S contains all the extracted statement sentences {s ₁ , s ₂ ,…,s _m }, a total of m items; by using the word segmentation tool to segment the statement sentences, the word set {x ₁ , x ₂ ,…,x _n } is obtained; Traverse all statements and calculate the support of each word, the calculation method is:

Among them, num(x) is the number of sentences containing word x in the sentence set S, and m is the total number of sentences in S; set the threshold K, put the word x with sup(x) greater than K into the feature set; set the association The rule is (x, y), that is, if the word x is a feature, y is also a feature. Initially set all the binary groups as association rules, and calculate the confidence of the association rules:

Where sup(x∪y) is the probability that words x and y appear together in a sentence; set a threshold K ₂ , and use the association rules with confidence greater than the threshold as feature expansion rules, that is, put word y into the feature set.

In the step 5, the trained classification model is used to classify the unmarked sentences, and the answer sentences are extracted, and the method of obtaining the answer sentence set is specifically: according to the obtained feature combination, the sentences in the training set are represented as feature vectors As the input of the naive Bayesian classification model; the feature vector is 1×n dimensional, and n represents n features; if the feature is included in the sentence, the corresponding position is 1; if it is not included in the sentence, the corresponding position is 0; classification The model uses the naive Bayesian classification model, let x={a ₁ ,a ₂ ,…,a _n } be the item to be classified, and the category set is C={y ₁ ,y ₂ }, by calculating P(y ₁ |x ), p(y ₂ |x) to get the probability that x belongs to two categories, and select the category with the highest probability as the category of x to which p(y _k |x)=max{p(y ₁ |x), p(y ₂ |x)}; among them, P(y ₁ |x), p(y ₂ |x) are obtained according to Bayes'theorem; by calculating the probability corresponding to each category, the class with the highest probability can be found, Through the classification model, the statement sentences are classified, the answer sentences in the statement sentences are found, and finally the set of answer sentences is obtained.

通过softmax函数，可得到概率函数

在训练时，最大化似然概率p(y|X)，即损失函数为-log(p(y|X))。Constructing BiLstm+crf neural network model in the described step 7 is specifically: BiLstm+crf neural network model comprises two layers BiLstm layer and crf layer; The One-Hot encoding used by the vector outputs the probability of each category label, and a sentence finally gives P in the form of a matrix, and Pij is the probability that the i-th word is marked as the j-th label; the Crf layer is a conditional random field, The label constraints can be learned from the input sequence, so that it is more accurate in the recognition of multi-category entities. There is a transfer matrix A, and A _ij represents the probability that the i-th label is transferred to the j-th label; therefore, the input sentence sequence X , get the label y, and the final scoring function is:

Through the softmax function, the probability function can be obtained

During training, the likelihood probability p(y|X) is maximized, that is, the loss function is -log(p(y|X)).

Example 1:

The present invention extracts question-answer pairs from semi-structured documents, and the specific flowchart is shown in FIG. 1 . Electronic medical records are documents used by doctors to record patients' conditions, including diagnostic records and some inspection reports, and are semi-structured documents. This invention will take this as an example to introduce the specific implementation of extracting question-answer pairs from semi-structured documents .

A method for extracting question-answer pairs from semi-structured documents based on machine learning, the specific steps are as follows:

1) Convert semi-structured documents in pdf format to txt documents.

Use the pdf parsing tool to parse the pdf document, decode the binary stream in it, extract the text information, and convert the document into a txt document.

2) Extract declarative sentences from semi-structured documents, and mark some declarative sentences.

Extract the text content in the txt document through regular expression matching, and segment the text. Sorting the obtained set of sentences according to the documents, randomly sorting the sentences inside the documents, sampling some sentences, judging whether the sentences are answer sentences, and marking the answer sentences.

3) Use the Apriori algorithm to mine the frequent itemsets in the sentence, and generate the extended features of the association rules.

Use all the words as the feature candidate set, and all the binary groups as the association rules, traverse the sentence set, calculate the support of each word, set the threshold, filter out the initial feature set, traverse the sentence set again, and calculate the association rule Confidence, set the threshold, and keep the valid feature set. Through the association rules, the initial feature set is expanded to obtain the feature set.

4) Use the naive Bayesian classification model to classify the declarative sentences and obtain the answer sentences.

Sentences are represented by features, and the feature vectors are input into the Naive Bayesian classification model for training, and the trained model is used to classify unlabeled sentences to obtain a set of answer sentences.

5) Crawl some domain entities from the web, and mark some answer sentences.

Part of the named entities is crawled through the web crawler, and part of the sentences in the answer sentence are marked with words, and the words are marked as BIO, and a training set for named entity recognition is established.

6) Build a BiLstm+crf neural network model, use the training set to train, and extract entities.

Construct the BiLstm+crf neural network model, input the word vector, and sequence the sentences. Sequence labeling of unlabeled sentences by the trained model, and finally convert the labeling into named entities to obtain an entity set.

7) Perform dependency syntactic analysis on the answer sentence, use the entity set, and generate a question sentence.

Perform word segmentation, part-of-speech tagging, and dependency syntactic analysis on the answer sentence to analyze the relationship between words in the sentence. By replacing the entities in the sentence and the words that depend on the entities, the corresponding question sentence of the answer sentence is obtained.

In step 1), the semi-structured document in pdf format is converted into a txt document. Electronic medical records are generally saved in pdf format, which is convenient for doctors to record and print. In order to facilitate subsequent processing, it needs to be converted into a document in txt format. The parsing process of the pdf document includes the following parts: read the pdf file stored locally, parse the file header, file body, cross-reference table and file trailer, use filters to parse the binary stream, extract the string, and save it to txt file. Commonly used parsing tools include tools such as pdfminer and pdfBOX.

In step 2), the declarative sentence is extracted from the semi-structured document, and part of the declarative sentence is marked. Taking electronic medical records as an example, declarative sentences are extracted from the two parts of "diagnosis basis" and "doctor-patient communication records". Since this part of the electronic medical record is written by a professional doctor, it often contains a lot of medical field knowledge. By writing a regular expression, that is, according to "identification basis" as the beginning and "analysis" as the end, the identification basis part is extracted. Then use serial numbers "1.", "2.", etc. as separators to separate out several declarative sentence groups with complete semantics. Since this part of the text will spread across pages, the sentences containing "**University", "Name" and "Medical Form" need to be deleted. For the doctor-patient communication part, it is necessary to use "communication records" as the beginning of the text, "communication results" as the end of the text, and use periods as separators to divide the statement sentences. The sentences containing "after admission" and "preliminary diagnosis" need to be combined into one sentence because they are logically causal. After obtaining the declarative sentences, it is necessary to label some of the declarative sentences to construct a training set. The training set consists of two parts, declarative sentences that can be answer sentences and declarative sentences that cannot be answer sentences. Among them, the declarative sentence that can be the answer sentence is marked as 1, and the declarative sentence that cannot be the answer sentence is marked as 0. The rule for extracting declarative sentences as the training set is to take each document as a unit, and each document randomly extracts several declarative sentences.

其中num(x)为句子集合S中包含词x的句子数量，m为S中句子的总数。设定阈值K，将sup(x)大于K的词x放入到特征集合中。设定关联规则为(x,y)，即词x为特征则y也为特征，初始设定所有的二元组为关联规则，计算关联规则的置信度

其中sup(x∪y)为词x，y一起出现在句子中的概率。设定阈值K₂，将置信度大于该阈值的关联规则作为特征扩充规则，即将词y放入到特征集合中。In step 3), the Apriori algorithm is used to mine the frequent itemsets in the sentence, and the extended features of the association rules are generated. The sentence set S contains all the extracted declarative sentences {s ₁ , s ₂ ,...,s _m }, a total of m items. By using a word segmentation tool, such as stammering word segmentation, to segment the declarative sentence, a word set {x ₁ , x ₂ ,…,x _n } is obtained. Traverse all declarative sentences and calculate the support of each word, the calculation method is

Where num(x) is the number of sentences containing word x in the sentence set S, and m is the total number of sentences in S. Set the threshold K, put the word x with sup(x) greater than K into the feature set. Set the association rule as (x, y), that is, if the word x is a feature, y is also a feature. Initially set all the two-tuples as association rules, and calculate the confidence of the association rules

where sup(x∪y) is the probability that words x and y appear together in a sentence. The threshold K ₂ is set, and the association rules whose confidence is greater than the threshold are used as feature expansion rules, that is, the word y is put into the feature set.

由于分母部分是常数项，这里只需将分子最大化，即

而上式中的右边每一项，都可以通过统计得到，因此就可以计算出每个类别所对应的概率，从而找到最大概率的类。通过分类模型，对陈述句分类，找到陈述句中的答案句，最终得到答案句集合。In step 4), the declarative sentence is classified using the naive Bayesian classification model to obtain the answer sentence. According to the combination of features obtained, the sentences in the training set are represented as feature vectors, which are used as the input of the naive Bayesian classification model. The feature vector is 1×n dimensional, where n represents n features. If the feature is included in the sentence, the corresponding position is 1. If not included in the sentence, the corresponding bit is 0. The classification model uses the naive Bayesian classification model, let x={a ₁ ,a ₂ ,…,a _n } be the item to be classified, and the category set is C={y ₁ ,y ₂ } by calculating P(y ₁ |x ), p(y ₂ |x) to get the probability that x belongs to two categories, and select the category with the highest probability as the category of x to which p(y _k |x)=max{p(y ₁ |x), p(y ₂ |x)}. Among them, P(y ₁ |x), p(y ₂ |x) are obtained according to Bayes' theorem. Bayesian theorem assumes that each feature attribute is independent of each other, so

Since the denominator part is a constant term, it is only necessary to maximize the numerator here, that is

And each item on the right side of the above formula can be obtained through statistics, so the probability corresponding to each category can be calculated, so as to find the category with the highest probability. Through the classification model, the statement sentences are classified, the answer sentences in the statement sentences are found, and finally the set of answer sentences is obtained.

In step 5), some domain entities are crawled from the web, and some answer sentences are marked. Use a web crawler to crawl named entities such as diseases, drugs, and organs from medical-related websites to obtain entity sets. But the entities crawled from the web are sometimes not enough, so the named entities in the sentence collection need to be identified. The training set is obtained by labeling the answer sentences. The crf-based named entity recognition regards the named entity recognition process as a sequence labeling problem, and marks the words in the sentence as BIO, B indicates the beginning of the entity, I indicates the interior of the entity, and O indicates that it does not belong to the entity. For diseases, medicines and organs, they can be marked as DISB, MEDB, ORGB to represent the beginning of the corresponding entity. Here, some sentences are extracted from the sentence collection, and the sentences are manually marked to obtain the training set for named entity recognition.

通过softmax函数，可得到概率函数

在训练时，最大化似然概率p(y|X)，即损失函数为-log(p(y|X))。通过使用python编程，利用各种开源的机器学习平台，可以搭建BiLstm+crf神经网络模型，输入训练集训练。将训练好的模型处理剩余的未标注的句子，得到答案句集合中的命名实体。In step 6), the BiLstm+crf neural network model is constructed, trained using a training set, and entities are extracted. The present invention builds a BiLstm+crf neural network model to identify entities, and the model includes two layers, a BiLstm layer and a crf layer. The BiLstm layer is a bidirectional LSTM neural network. The input is the word vector in a sentence. The word vector uses One-Hot encoding, and the output is the probability of each category label. A sentence is finally given in the form of a matrix, and Pij is the i-th The probability that a word is marked as the jth label. The Crf layer is a conditional random field, which can learn the label constraints from the input sequence, so that it is more accurate in the identification of multi-category entities. There is a transfer matrix A, and A _ij indicates that the i-th label is transferred to the j-th label. probability. Therefore, enter the sentence sequence X, get the label y, and the final scoring function is

Through the softmax function, the probability function can be obtained

During training, the likelihood probability p(y|X) is maximized, that is, the loss function is -log(p(y|X)). By using python programming and using various open source machine learning platforms, a BiLstm+crf neural network model can be built and input into the training set for training. Process the trained model on the remaining unlabeled sentences to obtain the named entities in the answer sentence set.

In step 7), the answer sentence is subjected to dependency syntax analysis, and the entity set is used to generate a question sentence. The answer sentence is first divided into words, tagged in the part of speech, and finally the dependency syntactic analysis is performed. The result obtained here is the relationship between each word and other words in the sentence. For example, SBV indicates the subject-predicate relationship. In this way, after the grammatical structure of the sentence is obtained, the position of the domain entity in the sentence and the situation of words dependent on the domain entity can be analyzed. At present, open source natural language processing platforms such as ltp can be used to perform dependency parsing on sentences. Generally, generative dependency parsing, discriminative dependency parsing, deterministic dependency parsing, and hierarchical parsing based on sequence annotations are used. method. After obtaining the syntactic analysis table of the sentence, the present invention searches the words appearing in the domain entity set by traversing the syntactic analysis table. After finding the domain entity contained in the sentence, use interrogative words to replace the entity of the disease and delete the words that depend on the entity, so as to convert the answer sentence into a fact-type question sentence.

The innovation point of the present invention is:

1. A method for extracting answer sentences is proposed. The invention applies a machine learning method to extract answer sentences from semi-structured documents. The semi-structured document is processed through pdf parsing, and the text containing candidate answer sentences is extracted. By applying Apriori for feature selection and naive Bayesian classification method for classification, answer sentences in semi-structured texts are obtained.

2. Propose a question generation method. The invention combines named entity recognition and dependency syntax analysis theory to convert answer sentences into corresponding question sentences. Named entity recognition adopts the crf+BiLstm neural network model to recognize the entities in the answer sentence and supplement them to the entities crawled from the web. Syntax analysis reveals the interdependence relationship among the words in the sentence, so that when the question is generated, the word that depends on the entity is replaced, and a reasonable question is obtained.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (3)

1. A method for extracting question and answer pairs from semi-structured documents based on machine learning, characterized in that, comprising the following steps: Step 1: Input the collection of semi-structured documents to be extracted, and convert the semi-structured documents in pdf format into txt documents; Step 2: Use regular expressions to extract the declarative sentences in the txt text, sort the obtained set of declarative sentences according to the documents, and randomly sort the declarative sentences inside the documents; randomly extract some sentences in each document, judge whether the sentences are answer sentences, and mark the answers sentence, construct the training set; Step 3: Use the Apriori algorithm to extract the frequent itemsets in the set as features on the training set, expand the features through association rules, and obtain the feature set; Step 4: According to the feature set, express the sentence as a feature vector, input the feature vector of the training set into the naive Bayesian classification model for training, and obtain the trained classification model; Step 5: Use the trained classification model to classify the unlabeled sentences, extract the answer sentences, and obtain the answer sentence set; Step 6: Crawl some domain entities through the web crawler; mark some sentences in the answer sentence set, mark the words as BIO, and establish a training set for named entity recognition; Step 7: Build a BiLstm+crf neural network model and use the training set for training; use the trained model to sequentially label the unlabeled sentences in the answer sentence set, and convert the labels into named entities; combine some domain entities crawled by web crawlers and named entities to get the entity collection; The BiLstm+crf neural network model includes two layers, the BiLstm layer and the crf layer; the BiLstm layer is a bidirectional LSTM neural network, the input is the word vector in a sentence, the One-Hot encoding used by the word vector, and the output is each category Label probability, a sentence finally gives P in the form of a matrix, Pij is the probability that the i-th word is marked as the j-th label; the Crf layer is a conditional random field, which is used to learn the constraints of the label from the input sequence, so that in The recognition of multi-category entities is more accurate. There is a transition matrix A, and A _ij represents the probability that the i-th label is transferred to the j-th label; therefore, input the sentence sequence X and get the label y, and the final scoring function is:

Through the softmax function, the probability function is obtained

During training, maximize the likelihood probability p(y|X), that is, the loss function is -log(p(y|X)); Step 8: Carry out word segmentation, part-of-speech tagging, and dependency syntactic analysis on the answer sentence set, analyze the relationship between words in the sentence, and obtain the question sentence corresponding to the answer sentence by replacing the entity in the sentence and the word dependent on the entity, and Output question-answer pairs to complete the extraction of question-answer pairs. 2. a kind of method based on machine learning according to claim 1 extracts question-and-answer pair from semi-structured document, it is characterized in that: in described step 3, use Apriori algorithm to extract the frequent item set in the collection to training set As a feature, the method of extending the feature through the association rules to obtain the feature set is as follows: The sentence set S contains all the extracted statement sentences {s ₁ , s ₂ ,…,s _m }, a total of m items; by using the word segmentation tool to segment the statement sentences, the word set {x ₁ , x ₂ ,…,x _n } is obtained; Traverse all statements and calculate the support of each word, the calculation method is:

Among them, num(x) is the number of sentences containing word x in the sentence set S, and m is the total number of sentences in S; set the threshold K, put the word x with sup(x) greater than K into the feature set; set the association The rule is (x, y), that is, if the word x is a feature, y is also a feature. Initially set all the binary groups as association rules, and calculate the confidence of the association rules:

Where sup(x∪y) is the probability that words x and y appear together in a sentence; set a threshold K ₂ , and use the association rules with confidence greater than the threshold as feature expansion rules, that is, put word y into the feature set. 3. A kind of machine learning-based method for extracting question-and-answer pairs from semi-structured documents according to claim 1 or 2, characterized in that: in the described step 5, utilize the trained classification model to unmarked sentences Carry out classification, extract the answer sentence, and the method of obtaining the answer sentence set is as follows: according to the combination of the obtained features, the sentences in the training set are expressed as feature vectors, which are used as the input of the naive Bayesian classification model; the feature vector is 1×n dimension , n represents n features; if the feature is included in the sentence, the corresponding position is 1; if it is not included in the sentence, the corresponding bit is 0; the classification model uses the naive Bayesian classification model, set x={a ₁ ,a ₂ ,...,a _n } are the items to be classified, the category set is C={y ₁ ,y ₂ }, by calculating P(y ₁ |x),p(y ₂ |x) to get x belongs to two categories Probability, select the category with the highest probability as the category of x p(y _k |x)=max{p(y ₁ |x), p(y ₂ |x)}; where P(y ₁ |x), p( y ₂ |x) is obtained according to Bayes'theorem; by calculating the probability corresponding to each category, the class with the highest probability can be found, and the statement sentences can be classified through the classification model to find the answer sentences in the statement sentences, and finally get A collection of answer sentences.

Images