CN110059185B - An automatic tagging method for professional vocabulary in medical documents - Google Patents

Abstract

The invention relates to an automatic labeling method for professional vocabulary of medical documents, which includes: preprocessing input medical documents to obtain preprocessed medical document text; obtaining letter-level feature vectors, word-level feature vectors and language feature vectors of words And fuse it as the encoding vector of the word; classify the word labeling of the medical document text after word segmentation to obtain the labeling data set; output a multi-dimensional vector for each word as the spatial representation of the word; obtain the enhanced labeling data set; Modeling, and finally output the annotation results. The present invention has a reasonable design, uses a semi-supervised learning algorithm to label a large amount of unlabeled data, successfully overcomes the defect of too little labelled data in the existing medical industry, effectively increases the amount of data that can be used by the model, and greatly improves the algorithm for The labeling accuracy of keywords and specialized vocabulary can be widely used in medical literature processing.

Description

An automatic tagging method for professional vocabulary in medical documents

technical field

The invention belongs to the technical field of machine learning, in particular to an automatic labeling method for professional vocabulary of medical documents.

Background technique

As the medical research community grows, more and more papers are published each year. There is a growing need to find ways to improve papers and automatically understand the key ideas in those papers. However, due to the wide variety of domains and extremely limited annotation resources, there is relatively little extraction of scientific information.

At the same time, with the surge in people's demand for medical resources, corresponding medical documents and the number of cases, researchers and medical staff need to quickly organize the past medical data of patients. It is often some professional words or keywords that can quickly help medical staff make judgments from patient cases. It takes a lot of time to manually sort out these words and keywords. Due to manpower constraints, it is impossible to quickly complete a large number of cases and medical treatment. Data collation work.

To sum up, with the increasing demand for medical resources, how to automatically mark professional words or keywords to improve the processing speed of medical staff for cases and medical data and help them better treat patients is an urgent need to solve. The problem.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to propose an automatic labeling method for professional vocabulary in medical documents, which uses a semi-supervised learning algorithm to expand the data volume, and overcomes the poor model performance caused by the insufficient data volume of medical text labeling in the past. problems, and ultimately improve the accuracy of identifying specialized words and keywords in the text.

The present invention solves its technical problem by adopting the following technical solutions to realize:

An automatic labeling method for professional vocabulary of medical documents, comprising the following steps:

Step 1. Perform data preprocessing on the input medical document to obtain the preprocessed medical document text;

Step 2. Use biLSTM to model the text to obtain the letter-level feature vector of the word;

Step 3. Use word2vec to model text to obtain word-level feature vectors of words;

Step 4. Based on the pragmatic characteristics of the text language, the language feature vector of the word is obtained;

Step 5, fuse the letter-level feature vector, word-level feature vector and language feature vector of the word obtained in step 2, step 3 and step 4, as the encoding vector of the word;

Step 6. Mark the words of the segmented medical document text as the following four types of medical entities: disease name, disease symptom, treatment method and drug name. Each type of entity uses IOBES to indicate the specific position of the word in the entity, and get the labeling data set;

Step 7. Use the text obtained in step 1 and the encoding vector of the word obtained in step 5 as the input of biLSTM, and output a multi-dimensional vector as the spatial representation of the word for each word;

Step 8, using the label propagation algorithm to expand the labeling data set to obtain the enhanced labeling data set;

Step 9. Use the multi-dimensional vector of step 7 as the spatial representation of the word as the vector of the word, input the enhanced labeling data set obtained in step 8 into the conditional random field for training and modeling, and finally output the labeling result.

Further, the specific implementation method of step 1 is as follows: firstly perform word segmentation on the input medical document, form an array, store each word and punctuation in the text, then remove stop words, and finally extract stem and morphological restoration , get the basic form of the word, and form an array of unlabeled words.

Further, the specific implementation method of the step 2 is: use biLSTM to encode the letter-level features of the preprocessed medical document text, use the first five letters of each word to encode, and finally obtain a letter-level length of 5d. Feature vector.

Further, the specific implementation method of step 3 is: using Google's Word2Vec algorithm to encode the word-level features of the preprocessed medical document text, and finally obtain a word-level feature vector of length d for each word.

Further, the specific implementation method of the step 4 is: according to the pragmatic characteristics of the text language, using a manual definition method, the following characteristics are defined for the preprocessed medical document text: first letter case, all lowercase words, all uppercase words, part of speech and grammatical structures to form a feature vector of length 21, each feature represented by 0 or 1.

Further, the specific implementation method of the step 5 is: connecting the letter-level feature vector, the word-level feature vector and the language feature vector together to form a comprehensive feature vector with a length of 5d+d+21 for each word.

Further, the labeling dataset in step 6 is a combined label including 20 categories.

Further, the specific implementation method of step 7 is: using the combined feature vector formed by the three features obtained in step 5, and arranging all the feature vectors of the entire word array to form a training data matrix, the number of rows of the matrix is is the number of words in the word array, the number of columns of the matrix is 5d+d+21; using biLSTM, the hidden layer through the forward and backward computation process is passed as input to the linear layer, which projects the dimensions to the label type The size of the space is 20 and is used as the input to the CRF layer.

Further, the specific implementation method of step 8 is: first, construct a graph based on the feature vector corresponding to the word, and as a node in the graph, use the similarity between the feature vectors to define their distance and weight w _uv , in the figure The total number of nodes is equal to the sum of the unlabeled data and the labeled data; then, the label propagation algorithm is used to minimize the Kullback-Leibler distance by optimizing the objective function, so that the label distributions between adjacent nodes are as similar to each other as possible, and finally all graphs The words corresponding to the middle nodes are labeled, and the enhanced dataset is obtained.

进行最大化，以完成监督学习，同时该CRF模型作为最终的结果输出。Further, the specific implementation method of the step 9 is: the multi-dimensional word space representation obtained in step 7 is used as the word vector, and biLSTM will eventually output a labeling matrix P, the P labeling matrix includes the probability distribution for each label, the It is poured into the CRF layer, a label sequence y is obtained, the score φ(y; x, θ) of the sequence y is calculated, and the probability P _θ (y|x) of the label sequence y appearing in all the label sequences is calculated, and finally Using backpropagation for the objective function log-

Maximize to complete supervised learning, and the CRF model is output as the final result.

The advantages and positive effects of the present invention are:

1. The present invention divides the keywords in the medical literature into four categories: disease name (disease), symptom (symptom), treatment method (treatment-method) and drug name (Drug-name), and label them based on semi-supervised learning Methods Labeling medical documents or cases on professional vocabulary can help medical staff or scholars to quickly understand the content of the text and make better medical decisions or research under the condition of extremely low human and material consumption.

2. The present invention uses a semi-supervised learning algorithm to label a large amount of unlabeled data, which successfully overcomes the defect of too little labelled data in the existing medical industry, effectively increases the amount of data that can be used by the model, and greatly improves the algorithm’s ability to use keywords for keywords. The accuracy of labeling and professional vocabulary can be widely used in medical document processing.

Description of drawings

FIG. 1 is a process flow chart of the present invention.

Detailed ways

The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

The design idea of the present invention is to use machine learning algorithms and technologies, and based on the semi-supervised learning labeling method, to label medical documents or cases on professional vocabulary. The present invention constructs a three-layer hierarchical neural network to mark the text: (1) the words in the text are extracted by vectorization in three ways, BiLSTM extracts the features based on letters, and Word2Vec does word embedding for the words , and feature extraction based on grammatical structure. (2) BiLSTM extracts and encodes the context information surrounding words in the same sentence. (3) The CRF labeling layer jointly uses the CRF objective function to model the word and the label, and makes the final label judgment.

Based on the above-mentioned design idea, the automatic labeling method for medical document professional vocabulary of the present invention, as shown in FIG. 1 , includes the following steps:

Step 1: Perform data preprocessing on the input medical document to obtain the preprocessed medical document text.

In this step, the input is a medical document and the output is an array of words. The data preprocessing method is: firstly perform word segmentation on medical documents, form an array, store each word and punctuation in the text, and then remove stop words, such as is", "but", "shall", "by", Then extract the stem and morphological restoration to get the basic form of the word. For example, running, ran, runs, after extracting the stem, get the word run, the morphological restoration is basically similar, and can restore any form of vocabulary to the general form, After data preprocessing, an unlabeled word array composed of general forms is obtained.

Step 2: Use BiLSTM to model the text and get the letter-level feature vector of the word.

The input of this step is the word array after data preprocessing, and the output is a feature vector based on letter features with a length of 5d.

The present invention uses biLSTM to encode letter features, which is called Character-BasedEmbedding. The letter-level features of words are generated by the hidden layer vectors during the forward propagation and backward computation of BiLSTM. The advantage of building a character-based embedding layer is that it can handle words and formulas outside the vocabulary, which are in these Common in the data, the resulting feature vector length is set to d. However, the present invention uses the first 5-gram (that is, the first 5 letters are coded from left to right when taking words, if there are no 5 letters, the remaining length is filled with zeros) to extract features, and the final feature vector Length is 5d.

Step 3: Use word2vec to model text and get word-level feature vectors of words.

In this step, words using a fixed vocabulary (plus unknown word tokens) are mapped to a vector space, initialized using Word2vec pre-training with different corpora combinations, and words are encoded using Google's Word2Vec algorithm, called WordEmbedding, Finally, the length of the feature vector for each word is d.

Step 4: Based on the pragmatic characteristics of the text language, the language feature vector of the word is designed.

In this step, the input is an array of words that only segment the original text, and the output is a feature vector designed based on language features, with a length of 21. Features are not trained separately, but are manually defined and called FeatureEmbedding. The features defined in this part include: first letter case, all lower case words, all upper case words, part of speech and grammatical structure, a total of 21 features, the length of the formed feature vector is 21, each feature is represented by 0 or 1, indicating Do they have corresponding characteristics.

Step 5: Integrate the letter-level features, word-level features and language features of the words obtained in steps 2, 3, and 4 as the encoding vector of the word.

The input of this step is the character-level feature vector, the word-level feature vector, and the language feature vector vector of the word. The above three feature vectors are connected together to form a length of 5d+d+21 for each word. eigenvectors of .

Step 6: Labeling data: label the words in the electronic medical record text after word segmentation as four types of medical entities (disease, symptom, treatment method, drug name). Annotated as 20 categories, the labeled dataset is obtained.

In order to be able to distinguish the span of two consecutive key phrases of the same type, the present invention assigns labels to each word in the sentence, specifying their position and type in the phrase. On the basis of the preprocessed data, each word is marked with a corresponding label, indicating the position and corresponding category of the phrase they are in, and their position is first marked. The present invention uniformly uses IOBES (Inside, Outside, Begining, End and Singleton) to describe the position of a word in a specialized phrase or vocabulary, I indicates that the words are inside the phrase, B indicates that they are at the beginning of the phrase, E indicates that they are at the end of the phrase, S indicates that the word is a separate specialized vocabulary, 0 means outside the phrase, contained within the sentence. At the same time, in combination with the present invention, the categories of these professional words and phrases, including disease name (disease), symptom (symptom), treatment method (treatment-method), drug name (Drug-name), are integrated to form a complete label, For example, criticalill patients are labeled "B-symptom I-symptom E-symptom". The combined tags thus formed have a total of 20 categories. Since the number of training sets is very large, the present invention labels a part of the data, thus forming labeled data and unlabeled data in the data set.

Step 7: Use the preprocessed text in Step 1 and the encoded vector of the word in Step 5 as the input of biLSTM, set the output to 20, and output a 20-dimensional vector for each word as the spatial representation of the word.

In this step, the combined feature vector formed by the three features obtained in step 5 is used, and all feature vectors of the entire word array are arranged to form a training data matrix. The number of rows in the matrix is the number of words in the word array. , the number of columns of the matrix is 5d+d+21, the present invention uses biLSTM to pass the hidden layer of the forward and backward calculation process as input to the linear layer, which projects the dimension to the size of the label type space, which is 20 , and used as the input to the CRF layer.

Step 8: Perform data enhancement on the labeled data set obtained in step 6, and use the label propagation algorithm to expand the labeled data set to obtain the enhanced labeled data set.

This step includes two parts: the first part is to construct a graph based on the feature vector corresponding to the word, and as a node in the graph, use the similarity between feature vectors to define their distance and weight w _uv , the total number of nodes in the graph is equal to Sum of tagged data and tagged data. The second part is to use the label propagation algorithm, which aims to make the label distributions between adjacent nodes as similar to each other as possible by optimizing the objective function that minimizes the Kullback-Leibler distance. This part finally enables the words corresponding to all the nodes in the graph to be labeled, and the enhanced data set is obtained. The specific method is:

The first part first builds the relationship graph required by the label propagation algorithm. The vertices in the graph correspond to the feature vectors of words, and the edges are the distances between word features to capture semantic similarity. The total size of the graph is equal to the sum of the marked data volume _Vl and the unmarked data volume _Vu . Modeled with a set of pre-trained word embeddings (dimension d), where the 5-gram consists of the first 5 letters of the current word, the word embedding closest to the verb, and a set of part-of-speech labels and case including parts (43 and a 4-dimensional one-hot vector). The resulting eigenvectors of length 5d+d+43+4 are then projected to 100 dimensions using the PCA dimensionality reduction algorithm. The present invention defines the weight w _uv of the edge between nodes u and v as follows: w _uv =d _e (u,v)ifv∈κ(u)oru∈κ(v), where κ(u) is the k- of u The set of nearest neighbors, _de (u, v) is the Euclidean distance between any two nodes u and v in the graph.

For each node i in the graph, the present invention computes edge probabilities {q _i } using forward and backward computation processes. Let _θi represent the estimation of the CRF parameters after the nth iteration, the present invention calculates the edge probability for the IOBES tag of each tag position i in sentence j in the tagged and unlabeled data

如果节点未连接到已标记顶点，则第三项将预测分布规则化为CRF预测，确保算法至少与标准自我训练一样好。The second part uses the label propagation algorithm to enhance the data to label the unlabeled data in the dataset. The algorithm aims to make the label distributions between adjacent nodes as similar to each other as possible by optimizing the objective function that minimizes the Kullback-Leibler distance. Minimize the Kullback-Leibler distance as follows: _i ) For nodes in the graph for all words, the distribution ru of labeled data and the predicted distribution of labels _qu . ii) The distribution of all nodes u and their neighbors v, q _u and q _v in the graph. iii) _qu and CRF edge probabilities for all distributed nodes

If the node is not connected to a labeled vertex, the third term regularizes the prediction distribution to CRF predictions, ensuring that the algorithm is at least as good as standard self-training.

Step 9: The 20-dimensional word space representation in Step 7 is used as a word vector, and the enhanced labeling data set obtained in Step 8 is input into a conditional random field for training and modeling.

进行最大化，以完成监督学习，同时该CRF模型也作为最终的结果输出。具体方法如下：Taking the spatial representation of the 20-dimensional word in step 7 as the word vector, biLSTM will eventually output a label matrix P, which basically already includes the probability distribution for each label, and then pour it into the CRF layer to get a Label the sequence y, calculate the score φ(y; x, θ) of the sequence y by the method, then calculate the probability P _θ (y|x) of the label sequence y appearing in all the label sequences, and finally use back propagation for the objective function.

Maximize to complete supervised learning, and the CRF model is also output as the final result. The specific method is as follows:

Keyword classification is a task that has strong dependencies between output labels. For example, I-disease cannot follow B-treatment-method, so instead of making independent labeling decisions for each output, the present invention uses Conditional random fields model them jointly. For the input sentence x=(x1, x2, x3, . . . , xn), the present invention considers P to be the score matrix output by the biLSTM network. The size of P is n × m, where n is the number of tokens in the sentence and m is the number of distinct tokens. P _t,i corresponds to the score of the ith label of the ith word in the sentence. The present invention uses a first-order Markov model and defines a transition matrix T, where T _i,j represents the score from label i to label j. The present invention simultaneously adds y ₀ and _yn as start and end identifiers. So the dimension of the T matrix becomes m+2.

Given a possible output y and a neural network parameter θ, the present invention defines the score as

The probability of sequence y is obtained by applying softmax over all possible label sequences:

P _θ (y|x)=exp(φ(y; x, θ))/∑ _y′∈Y exp(φ(y′; x, θ))

where Y represents all possible tag sequences. The normalization term can be efficiently computed using the forward algorithm.

Finally, initial training is performed using the label data in the dataset, during which the present invention maximizes the log-probability L(Y; X, θ) of the correct label sequence for a given corpus {X, Y}. At the same time backpropagation is done based on the gradient computed using the total score of the sentence.

After getting the trained CRF algorithm, combine it with the feature extraction part constructed earlier, and you can start annotating the text. That is, inputting a sentence x=(x ₁ , x ₂ , x ₃ ..., x _n ) will get a label sequence y=(y ₁ , y ₂ , y ₃ ..., _yn ).

The meanings of the parameters in the above formula are explained as follows:

What is not described in the present invention applies to the prior art.

It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive, so the present invention includes but is not limited to the embodiments described in the specific implementation manner. Other embodiments derived from the scheme also belong to the protection scope of the present invention.

Claims (9)

1. a medical document professional vocabulary automatic labeling method, is characterized in that comprising the following steps: Step 1. Perform data preprocessing on the input medical document to obtain the preprocessed medical document text; Step 2. Use biLSTM to model the text to obtain the letter-level feature vector of the word; Step 3. Use word2vec to model text to obtain word-level feature vectors of words; Step 4. Based on the pragmatic characteristics of the text language, the language feature vector of the word is obtained; Step 5, fuse the letter-level feature vector, word-level feature vector and language feature vector of the word obtained in step 2, step 3 and step 4, as the encoding vector of the word; Step 6. Mark the words of the segmented medical document text as the following four types of medical entities: disease name, disease symptom, treatment method and drug name. Each type of entity uses IOBES to indicate the specific position of the word in the entity, and get the labeling data set; Step 7. Use the text obtained in step 1 and the encoding vector of the word obtained in step 5 as the input of biLSTM, and output a multi-dimensional vector as the spatial representation of the word for each word; Step 8, using the label propagation algorithm to expand the labeling data set to obtain the enhanced labeling data set; Step 9: Input the multi-dimensional vector obtained in step 7 and the enhanced labeling data set obtained in step 8 into the conditional random field for training and modeling, and finally output the labeling result. 2. a kind of medical document professional vocabulary automatic labeling method according to claim 1, is characterized in that: the concrete realization method of described step 1 is: firstly carry out word segmentation to the input medical document, form an array, store in the text. For each word and punctuation, then stop words are removed, and finally stemming and lemmatization are extracted to get the basic form of the word and form an array of unlabeled words. 3. a kind of medical document professional vocabulary automatic labeling method according to claim 1, is characterized in that: the concrete realization method of described step 2 is: use biLSTM to encode the letter-level feature of the preprocessed medical document text, Use the first five letters of each word to encode, resulting in a letter-level feature vector of length 5d. 4. a kind of medical document professional vocabulary automatic labeling method according to claim 1, is characterized in that: the concrete realization method of described step 3 is: use Google's Word2Vec algorithm to the word-level feature of the preprocessed medical document text Encode, and finally get a word-level feature vector of length d for each word. 5. a kind of medical document professional vocabulary automatic labeling method according to claim 1, is characterized in that: the concrete realization method of described step 4 is: according to text language pragmatic characteristic, adopt manual definition method, to preprocessed The medical document text defines the following features: first letter case, all lower case words, all capital words, part of speech and grammatical structure, forming a feature vector of length 21, each feature is represented by 0 or 1. 6. a kind of medical document professional vocabulary automatic labeling method according to claim 1, is characterized in that: the concrete realization method of described step 5 is: connect letter-level feature vector, word-level feature vector and language feature vector together , forming a comprehensive feature vector of length 5d+d+21 for each word. 7 . The automatic labeling method for professional vocabulary of medical documents according to claim 1 , wherein the labeling data set in step 6 is a combination label including 20 categories. 8 . 8. a kind of medical document professional vocabulary automatic labeling method according to claim 1 is characterized in that: the concrete realization method of described step 7 is: utilize the combined feature vector formed by three kinds of features obtained in step 5, All eigenvectors of the word array are arranged to form a training data matrix, the number of rows of the matrix is the number of words in the word array, and the number of columns of the matrix is 5d+d+21; using biLSTM, by forward and backward The hidden layer of the computational process is passed as input to the linear layer, which projects the dimension to the label type space of size 20, and is used as the input to the CRF layer. 9. a kind of medical document professional vocabulary automatic labeling method according to claim 1, it is characterized in that: the concrete realization method of described step 8 is: first, build a graph based on the feature vector corresponding to the word, and as the graph in the graph nodes, using the similarity between feature vectors to define their distance and weight w _uv , the total number of nodes in the graph is equal to the sum of the unlabeled data and the labeled data; then, the label propagation algorithm is used to minimize the Kullback-Leibler distance by optimizing The objective function is to make the label distribution between adjacent nodes as similar to each other as possible, and finally make the words corresponding to all nodes in the graph get labeled, and get an enhanced dataset.

Images