CN109635108B - A remote-supervised entity relation extraction method based on human-computer interaction - Google Patents

Abstract

The invention discloses a remote supervision entity relationship extraction method based on human-computer interaction. Evaluation, according to the feedback of the model, adjust the crowdsourcing strategy in time to obtain new data and put it into the model, until all the data is cleaned or the model performance is no longer improved. Compared with the prior art, the present invention has the advantages of high quality of crowdsourcing results and low cost of crowdsourcing, and can be processed in parallel by multiple workers without relying on experts for labeling, and the obtained labeling data has higher guiding significance for model learning. It ensures the accuracy of relation extraction between entities, and better solves the problem that noise data reduces model performance in relation extraction between remote supervision entities.

Description

A remote-supervised entity relation extraction method based on human-computer interaction

technical field

The invention relates to the technical field of relationship extraction between remote supervision entities, in particular to a method for extracting relationship between remote supervision entities based on human-computer interaction.

Background technique

Information extraction (Information Extraction) is a basic task in natural language processing. It extracts structured information by processing unstructured text as the input of subsequent natural language processing tasks. In the era of knowledge explosion, people need to face massive data every day, and it is extremely important to use information extraction system to efficiently process text and extract useful information. As an extremely important part of natural language processing, information extraction itself is also composed of a series of sub-tasks, such as named entity recognition, relation extraction and event extraction.

Early inter-entity relationship extraction methods mainly rely on rules and templates written by experts to obtain correct relationships through matching. Later relation extraction tasks began to focus on statistical-based machine learning methods, which are usually divided into feature-based methods and kernel-based methods. This method obtains models by learning features or kernel functions to predict or classify the relationship between entity pairs. Since both feature extraction and kernel function determination are prone to errors, these errors may lead to errors in subsequent work, and features and kernel functions cannot guarantee to extract all relevant information in the text. In order to deal with the problems arising from the use of statistical machine learning methods, researchers use deep learning methods to automatically learn features for model training. The mainstream methods based on deep learning can be divided into two categories: supervised and remote supervision. However, the model training of supervised methods requires a large amount of manually labeled data. Therefore, when the amount of data is small and the manual participation is expected to be reduced, researchers try to use remote supervision. supervised training.

The remote supervision of the existing technology mostly combines remote supervision learning and multi-instance learning. It is hoped that meaningful data can be obtained from some unlabeled data sets and added to the training set, which can effectively expand the existing data sets. The model effect is partially improved. However, the remotely supervised dataset contains a large amount of noise data, which will have a huge impact on the model performance during the model training process. For example, for sentences containing the same entity pair, there is a certain relationship between entity pairs in some sentences Relationship types, but more are sentences that do not contain any relationship between entities, but deep learning models usually do not consider this situation during training. Although, some multi-instance methods have also been proposed to reduce the influence of noise data on the model training effect, selecting sentences with higher confidence in the instances for training, or assigning different instances to different instances according to their matching degree with the relation type during training. These methods have improved the performance of the model to a certain extent. The above are some methods based entirely on the learning ability of the deep learning model itself. Although these methods have achieved good results, they still cannot completely avoid the interference of noise data and the lack of effective labeling data. Sentences of an entity pair do not have any relationship to this entity pair, in which case these instances should all be classified as negative examples, but the current model does not do this. Therefore, it is necessary to develop a method for remotely supervised dataset cleaning and extraction of inter-entity relations using crowdsourcing technology to overcome the impact of noisy data on model performance.

SUMMARY OF THE INVENTION

The purpose of the present invention is to design a method for extracting entity relationships based on human-computer interaction for remote supervision based on the shortcomings of the prior art, using crowdsourcing technology to join the model training method of relation extraction tasks, and reducing noise data by manual labeling For the impact of model performance, the artificially labeled data is put into the deep learning model for training and evaluation, and the crowdsourcing strategy is adjusted in time according to the feedback of the model to obtain new data and input into the model, until all data is cleaned or the model performance is not satisfactory. Further improvement, the data after crowdsourcing and cleaning is used as a real training set, which better solves the problem of noise data reducing model performance in the extraction of relationships between remote supervision entities. Crowdsourcing strategies can be formulated according to users and data analysis to improve crowdsourcing. The quality of the results reduces the cost of crowdsourcing, and has strong portability. After simple training, it can be easily used in different fields to solve common-sense problems in the professional field. It does not need to rely on experts for annotation, and multiple workers can run in parallel. Processing, to a large extent, reduces the cost of time and money, and the labeled data obtained by crowdsourcing technology has higher guiding significance for model learning, ensuring the accuracy of relationship extraction between entities.

The purpose of the present invention is to achieve this: a remote supervision entity relationship extraction method based on human-computer interaction, which is characterized by using crowdsourcing technology to join the model training method of the relationship extraction task, and reducing noise data by manual labeling. Influence of performance, put the manually labeled data into the deep learning model for training and evaluation, and adjust the crowdsourcing strategy in time according to the feedback of the model to obtain new data and put it into the model, until all the data is cleaned or the model performance is no longer improved. So far, its remote-supervised entity relationship extraction process for human-computer interaction includes the following steps:

Step a: Take the dataset that can be used for entity relationship extraction as the target text, extract the entity pair E(R) containing the relationship type R, and the natural language sentence S(ep _i ) corresponding to these entities, where ep contains the header Entities and tail entities, the datasets are the "freebase" dataset and the "NYT" dataset.

Step b: Select an unprocessed entity pair ep _k and its corresponding natural sentence sequence S(ep _k ) from E(R), and use word2vec to vectorize ep _k and S(ep _k ), and we will get The real-valued vectors V _k and V _sk (including multiple words w) of , use the "transE" model to calculate the similarity, and sort the similarity score M _kj between the obtained ep _k and each sentence V _skj .

Step c: Use machine learning to select the suspected noise data that needs to be cleaned and put it into crowdsourcing for re-labeling, and then put the re-labeled data into the convolutional neural network model as training data, and adjust the selection strategy in time according to the feedback of the model for iterative cleaning. And in the iterative process, the model performance is gradually improved. The specific process includes the following steps:

(1) Setting the threshold

The threshold is set according to the learning rate of 0.05, and the threshold is in the range of [0.5, 0.8] for iterative experiments. The threshold is automatically increased by 0.05 in each iteration until the increase of the threshold no longer improves the model effect or reaches 0.8. Obtain the optimal threshold for automatic parameter tuning, and assume that this threshold represents that sentences with a similarity M higher than this threshold are default positive examples, while sentences below this threshold are suspected noise data.

(2) Crowdsourced labeling

Collect suspected noise data Z(ep _k ) according to the threshold to write a crowdsourcing question generator, and batch generate single-choice questions about whether there is a specified relationship between two entities in sentences with content Z(ep _k ), when the worker chooses yes When answering, you need to give which word w in the sentence prompted you to answer in the affirmative, and use the above content as the input data of the neural network model.

(3) Quality control of crowdsourcing results

Prepare a batch of questions with correct answers as gold data, mix them with the data that really needs to be crowdsourced, and distribute them to workers together. When the accuracy rate of the gold data is higher than 80%, it is considered that workers submitted The answer is valid labeling, otherwise there is a quality problem in the answer. The crowdsourcing is to distribute the same question to five or seven workers for labeling. The person's answer is the final result, you can immediately stop answering the question and recover the result.

(4) The labeled data is put into the deep learning model

Collect and organize the final results obtained through crowdsourcing annotation, add them to the original training set that excludes the crowdsourcing data, and put them into the convolutional neural network as the training set of the deep learning model for training to obtain the relationship between entities The input of the convolutional neural network is: the word embedding of the sentence in the training set and the vector representation of the position of the entity pair in the sentence (position embedding), and the output is: between the entity pairs contained in the sentence The probability of a certain type of relationship exists, and the parameters of the convolutional neural network are iteratively adjusted according to the output results and the standard results.

(5) Model feedback mechanism

Put the test set data into the above-trained classification model for prediction, and evaluate the model performance according to the predicted value. If the model performance is improved, it means that the data after crowdsourcing and cleaning is valid labeled data, which can be used as real data. Training set; if the model performance is not improved, it is considered that this batch of labeled data has no practical significance, then return to step c, reset the threshold according to step (1) for iterative cleaning, and repeat steps (2)~( 5) Carry out re-labeling, and decide the termination of iterative cleaning according to the improvement of model performance and the feedback of crowdsourced labeling. When one round of iterative cleaning is over, return to step b, and select the next unprocessed entity in E(R). Enter a new round of iterative cleaning for ep _k and its corresponding natural sentence sequence S(ep _k ), and repeat this iterative cleaning until all unprocessed entity pairs ep _k and their corresponding natural sentence sequences in E(R) are completed After S(ep _k ) is crowdsourced and cleaned, a data set after noise data cleaning is obtained.

Compared with the prior art, the present invention has the following advantages:

(1) The human-computer interaction method with crowdsourcing technology as the main implementation method can effectively clean the semi-supervised training set containing a large amount of noisy data, and the performance of the model can be improved;

(2) The crowdsourcing strategy formulated based on user and data analysis can improve the quality of crowdsourcing results while reducing the cost of crowdsourcing, avoiding the problem of excessively expensive annotations for large-scale crowdsourcing;

(3) It has strong portability, and can convene workers with different knowledge backgrounds to solve problems in different fields, and these workers can solve common-sense problems in the professional field after simple training, without relying on experts for annotation. Multi-worker parallel processing, greatly reducing the cost of time and money;

(4) The labeled data obtained by crowdsourcing technology has higher guiding significance for model learning, which ensures the accuracy of relation extraction between entities.

Description of drawings

Fig. 1 is the schematic flow chart of the present invention;

FIG. 2 is a schematic diagram of an example of noise data.

Detailed ways

The present invention uses the "Freebase" data set commonly used in the entity relationship extraction task as the target text, and by mapping this data set with the "NYT" data set containing the exact triple relationship, the contained entities are extracted for each sentence. For information such as the relationship type between pairs and entities, the similarity between the sentence containing a certain entity pair and the relationship type obtained by the remote supervision annotation is matched, and the sentences are sorted according to the similarity, and the similarity is lower than a certain threshold. The sentence is put into crowdsourcing for manual annotation to determine whether it is of a certain relationship type, and the annotated data is re-trained as annotated data and additional information. The experimental model uses a convolutional neural network as a relationship type classifier. The input is a real-valued vector obtained after the sentence is processed by google's word2vec, and contains the location information of the entity pair. The output of the model is this sentence. The entity pair may contain The scores of various relationship types, which specifically include the following steps:

Step a: Take the datasets that can be used for relation extraction between entities (such as "freebase" dataset and "NYT" dataset, etc.) as the target text, where the data of each row is the head entity and the tail entity, the relationship type between entities, The sentence containing this entity pair; because there are multiple different sentences containing the same entity pair (the relationship type between entities is the same), a certain type of relationship R in the relationship type is selected for data processing. And, extract the entity pairs E(R)={ep ₁ ,ep ₂ ,...,ep _n } containing the relation type R, and the natural language sentences corresponding to these entity pairs S(ep _i )={s _i1 ,s _i2 ,…,s _im }, where ep _n contains the head entity e _hn and the tail entity e _tn .

Step b: Select an unprocessed entity pair ep _k and its corresponding natural sentence sequence S(ep _k )={s _k1 ,s _k2 ,...,s _km } from E(R), and use word2vec to convert ep _k and S(ep _k ) are vectorized to obtain real-valued vectors V _k and V _sk (including multiple words w); the similarity calculation is performed according to formulas (I) and (II) to obtain ep _k and each sentence The similarity score M _kj between V _skj and sorted;

M _kj =cos(V _k ,V _skj ) (I)

Among them: the cos() function represents the Euclidean distance between two vector representations with the same dimension;

V _skj = w ₁ +w ₂ +...+w _n (II)

Among them: w is the word contained in each sentence, because the number of words in the sentence cannot be controlled, in order to ensure that each sentence vector has the same dimension, we normalize it, add the word vectors in the sentence bitwise, and the word vector The sum represents the vector representation of the entire sentence.

Step c: Use machine learning to select the suspected noise data that needs to be cleaned and put it into crowdsourcing for re-labeling, and then put the re-labeled data into the convolutional neural network model as training data, and adjust the selection strategy in time according to the feedback of the model for iterative cleaning. And in the iterative process, the model performance is gradually improved. The specific process includes the following steps:

Step 1: Set Thresholds

Assuming that this threshold represents that sentences with a similarity M higher than this threshold are positive examples by default containing such relations, while sentences below this threshold do not contain such relations are negative examples, and collect the sentences below this threshold. The sentence Z(ep _k )={z _k1 ,z _k2 ,…,z _kl }(Z(ep _k ) is included in S(ep _k )) and put into the crowdsourcing platform for annotation. At this time, the learning rate of the automatic parameter tuning process for obtaining the threshold is 0.05, and the threshold range is controlled at [0.5, 0.8], and the threshold with the best experimental effect can be obtained, which can achieve the purpose of reducing the amount of crowdsourcing data and reducing the cost of crowdsourcing. , because sentences below the threshold will not be considered in the experiment.

Step 2: Crowdsource Labeling

For the data Z(ep _k ) that needs to be crowdsourced, it is set as a single-choice question, and the content of the question is whether there is a specified sentence between the two entities (e _hl , e _tl ) contained in the given sentence z _kl . In relation R, the options include two possibilities—existence and nonexistence; at the same time, when the worker chooses an affirmative answer, additional information needs to be given, that is, which word w _pl in the sentence prompts you to make an affirmative answer.

Step 3: Results Quality Control

After preprocessing the data that needs to be manually labeled, in order to ensure the accuracy of the crowdsourcing results, we also need to carry out quality control, prepare a batch of questions with correct answers as golden data, and mix them with the data that really needs crowdsourcing. When the accuracy rate is higher than 80%, the result submitted by this worker is considered to be a valid mark, and this data is used in the subsequent process to give The worker will be paid a certain amount; otherwise, the worker will be informed that there is a problem with the quality of the answer, and it cannot be tried to work effectively.

Step 4: Put the labeled data into the deep learning model

Collect and organize the final results obtained through crowdsourcing annotation, add them to the original training set that excludes the crowdsourcing data, and put them into the convolutional neural network as the training set of the deep learning model for training to obtain the relationship between entities The input of the convolutional neural network is: the word embedding of the sentence in the training set and the vector representation of the position of the entity pair in the sentence (position embedding), and the output is: the entity pair contained in the sentence There is a certain The probability of a type of relationship, and the parameters of the convolutional neural network are iteratively adjusted according to the output results and the standard results.

In addition, the additional annotation information w _pl obtained before can be effectively used, that is, the word that prompts the annotator to make a positive choice, and input it into the model as a feature data of the convolutional neural network, that is, the labeled The number of times that the value of the word at the corresponding position in the sentence changed from 0 to being labeled. At the same time, for each sentence that needs to be trained, different weights will be given based on the similarity.

α, take

The "attention" mechanism is used to adjust the contribution of different instances to the training model, and its weight α and contribution P _k (θ) are calculated as follows (III) and (IV) respectively:

P _k (θ)=α ₁ s _k1 +α ₂ s _k2 +…+α _n s _kn (III)

Among them: α _j represents the contribution of the jth sentence in the convolutional neural network model containing a certain relation type sentence set;

Among them: P _k (θ) represents for a certain entity pair K, the comprehensive contribution of all sentences that play a positive role in the model prediction to the final prediction.

Step 5: Model Feedback Mechanism

The test set data is put into the trained classification model for prediction and the final result is obtained, and the performance of the model is evaluated according to the predicted value. Real training set; if the performance of the model is not improved, it is considered that this batch of labeled data has no practical significance and does not need to be used as training data; after reorganizing the training set, re-select the relationship type to be processed, and go back to the first step of step c to adjust the threshold value. Iteratively cleans to obtain a crowdsourced cleaned denoising dataset for the specified entity pair.

The present invention will be further described in detail through the following specific examples.

Example 1

Referring to Figure 1, large-scale free text is input, followed by preprocessing of the input free text.

See Figure 2, noisy data and correctly labeled data present in the remote supervision dataset

Proceed as follows:

Step a: Use the "Freebase" dataset aligned with the "NYT" dataset (including explicit triplet relationships) as the target text, where the data in each row is: head entity e _hn and tail entity e _tn and the relationship between entities Type, a sentence containing this entity pair. Because there are multiple different sentences containing the same entity pair (the relationship types between entities are the same), a certain type of relationship R among the sixty-three relationship types is selected for data processing, for example: /location/location/contains. Extract the entity pair E(R)={queens belle_harbor,ohiocelina,…} containing the relation type R, and the corresponding natural language sentence S(queens belle_harbor)={but instead there was a funeral, at st.francis de sales roman catholic church,inbelle_harbor,queens,the parish of his birth.###END###,…}.

Step b: Select an unprocessed entity pair (queens belle_harbor) and its corresponding natural sentence sequence from E(R), and use word2vec to vectorize ep _k and S(ep _k ) to obtain a real-valued vector _Vk and _Vsk (contains multiple words w). The similarity score M _kj between ep _k and each sentence V _skj is obtained by similarity calculation, and sorted, for example, the sentence {s ₁ ,s ₂ ,s ₃ ,s ₄ } containing the entity pair (queens belle_harbor) and the relationship The similarity of type/location/location/contains is 0.5, 0.34, 0.15, 0.42;

Step c: Use machine learning to select the suspected noise data that needs to be cleaned and put it into crowdsourcing for re-labeling, and then put the re-labeled data into the convolutional neural network model as training data, and adjust the selection strategy in time according to the feedback of the model for iterative cleaning. The specific operations are as follows:

Step 1: Set Thresholds

Suppose the initial threshold is 0.5, collect sentences {s ₂ , s ₃ } below this threshold and put them into the crowdsourcing platform for annotation, and the threshold will increase by 0.05 in the next iteration, until increasing the threshold cannot improve the model performance or the threshold has been to 0.8. According to the obtained threshold, the data set is cleaned according to steps (2) to (5), until the manual labeling stage confirms that all potential noise data has been relabeled, and the denoised data set is obtained.

Step 2: Crowdsource Labeling

For the data that needs to be crowdsourced, we set it as a single-choice question. The content of the question is whether there is a specified relationship R between the two entities contained in the given sentence. The options include two possibilities-existence and non-existence . When the worker chooses an affirmative answer, additional information is required, that is, which word in the sentence prompted you to make an affirmative answer, for example: sentence 1 is "one was for st.francis de sales roman catholic church inbelle_harbor; another board studded with electromechanical magnets will gounder the pipes of an organ at the evangelical lutheran church of christ inrosedale,queens.###END###", the user answered "no".

Sentence 2 is "but instead there was a funeral, at st. francis de sales romancatholic church, in belle_harbor, queens, the parish of his birth. ###END###", the user's answer is "exist", additional annotation information for the word "in".

Step 3: Quality Control

After preprocessing the data that needs to be manually labeled, in order to ensure the accuracy of the crowdsourcing results, we also need to carry out quality control: prepare a batch of questions with correct answers as golden data, and mix them with the data that really needs crowdsourcing. , distributed to workers together. If the correct rate of gold data is less than 80% among the recovered answers, the worker's answer will not be used. In order to ensure the accuracy of the data, the same question was distributed to five workers for annotation, and the collected answers were four "exist" and one "nonexistent", and the majority voted that the answer was "exist".

Step 4: Invest in deep learning model training

Collect and organize the final results obtained through crowdsourcing annotation, and put them into the convolutional neural network as a training set of deep learning model for training to obtain a classification model of the relationship between entities. The input of the convolutional neural network is: the word vector representation of the sentences in the training set and the vector representation of the position of the entity pair in the sentence, and additional feature information, where "in" is labeled three times and "of" is labeled once, so the feature vector is V={0,0,...,3,0 ,…,1,0,…}, that is, the number of the corresponding positions of “in” and “of” in the sentence is the number of times they are marked. At the same time, for each sentence that needs to be trained, different weights α are given on the basis of similarity, and the "attention" mechanism is adopted to adjust the contribution of different instances to the training model, where for all sentences {s ₁ , s ₂ The weights of , s ₃ , s ₄ } are {α ₁ , α ₂ , α ₃ , α ₄ } respectively.

Step 5: Adjust the crowdsourcing strategy according to the feedback mechanism of the model:

Put the processed data with the relation type of /location/location/contains into the trained classification model for prediction and get the final result, evaluate the performance of the model according to the predicted value, and find that the performance of the model has been improved, so the previous cleaning is effective Operation, go back to the first step in step c, and continue to adjust the threshold until the data processing is completed without labeling. The optimal threshold for the final effect is 0.75. So far, the relationship type selected in step b is /location/location/contains The training set containing unprocessed entity pairs (queens belle_harbor) has been denoised successfully. Then go back to step b, continue to select other unprocessed entities whose relationship type is /location/location/contains in E(R), and perform a new round of data cleaning in step c. The cleaning step is the same as the entity pair ( queensbelle_harbor) is the same as the training set, so iterative cleaning is repeated until all unprocessed entities in E(R) are crowdsourced and cleaned for ep _k and its corresponding natural sentence sequence S(ep _k ), that is, all the unprocessed entities in E(R) are cleaned by crowdsourcing. If the data sets whose relation type is /location/location/contains have been crowdsourced or the accuracy of the model remains stable and no longer improves, the crowdsourcing cleaning process can be ended, and the denoised data for the relation type to be processed can be obtained at this time. set.

The above is only a further description of the present invention, and is not intended to limit this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.

Claims (1)

1. a remote supervision entity relationship extraction method based on human-computer interaction is characterized in that adopting crowdsourcing technology to join the model training method of the relation extraction task, reducing the impact of noise data on model performance by manual labeling, and manually labeling the method. The resulting data is put into the deep learning model for training and evaluation. According to the feedback of the model, the crowdsourcing strategy is adjusted in time to obtain new data and put into the model until all data is cleaned or the model performance is no longer improved. The remotely supervised entity relation extraction process includes the following steps: Step a: Take the dataset that can be used for entity relationship extraction as the target text, extract the entity pair E(R) containing the relationship type R, and the natural language sentence S(ep _i ) corresponding to these R entities, where ep Contains the head entity and the tail entity, and the data set is the "freebase" data set and the "NYT" data set; Step b: Select an unprocessed entity pair ep _k and its corresponding natural sentence sequence S(ep _k ) from E(R), and use word2vec to vectorize ep _k and S(ep _k ), and we will get The real-valued vectors V _k and V _sk of , where each natural sentence represented by V _sk contains multiple words, the “transE” model is used for similarity calculation, and the similarity between ep _k and each sentence V _skj will be obtained Sort by degree score M _kj ; Step c: Use machine learning to select the suspected noise data that needs to be cleaned and put it into crowdsourcing for re-labeling, and then put the re-labeled data into the convolutional neural network model as training data, and adjust the selection strategy in time according to the feedback of the model for iterative cleaning. And in the iterative process, the model performance is gradually improved. The specific process includes the following steps: (1) Set the similarity threshold The similarity threshold is set according to the learning rate of 0.05, and the threshold is in the range of [0.5, 0.8] for iterative experiments. The threshold is automatically increased by 0.05 in each iteration, until the increase of the threshold no longer improves the model effect or reaches 0.8 So far, the optimal threshold for automatic parameter adjustment is obtained, and it is assumed that this threshold represents that sentences whose similarity M is higher than this threshold are positive examples by default, while sentences lower than this threshold are suspected noise data; (2) Crowdsourced labeling Collect suspected noise data Z(ep _k ) according to the threshold to write a crowdsourcing question generator, and batch generate single-choice questions about whether there is a specified relationship between two entities in sentences with content Z(ep _k ), when the worker chooses yes When answering, the keyword w related to the answer in the sentence needs to be given, and the above content will be used as the input data of the neural network model; (3) Quality control of crowdsourcing results Prepare a batch of questions with correct answers as gold data, mix them with the data that really needs to be crowdsourced, and distribute them to workers together. When the accuracy rate of the gold data is higher than 80%, it is considered that workers submitted The answer is valid labeling, otherwise there is a quality problem in the answer. The crowdsourcing is to distribute the same question to five or seven workers for labeling. The person's answer is the final result, you can immediately stop answering the question and recover the result; (4) The labeled data is put into the deep learning model Collect and organize the final results obtained through crowdsourcing annotation, add them to the original training set that excludes the crowdsourcing data, and put them into the convolutional neural network as the training set of the deep learning model for training to obtain the relationship between entities The input of the convolutional neural network is: the word vector of the sentence in the training set represents the word embedding and the vector of the position of the entity pair in the sentence represents the position embedding, and the output is: there is a certain entity between the pair of entities contained in the sentence. The probability of the type relationship, and iteratively adjust the parameters of the convolutional neural network according to the output result and the standard result; (5) Model feedback mechanism Put the test set data into the above-trained classification model for prediction, and evaluate the model performance according to the predicted value. If the model performance is improved, it means that the data after crowdsourcing and cleaning is valid labeled data, which can be used as real data. Training set; if the model performance is not improved, it is considered that this batch of labeled data has no practical significance, then return to step c, reset the threshold according to step (1) for iterative cleaning, and repeat steps (2)~( 5) Carry out re-labeling, and decide the termination of iterative cleaning according to the improvement of model performance and the feedback of crowdsourced labeling. When one round of iterative cleaning is over, return to step b and select the next unprocessed entity in E(R). Enter a new round of iterative cleaning for ep _m and its corresponding natural sentence sequence S(ep _m ), and repeat iterative cleaning until all unprocessed entity pairs ep _m and its corresponding natural sentence sequence in E(R) are completed. After S(ep _m ) is crowdsourced and cleaned, a data set after noise data cleaning is obtained.

Images