CN115563284B - A deep multi-instance weakly supervised text classification method based on semantics - Google Patents

Abstract

The invention provides a semantic-based deep multi-instance weak supervision text classification method, which comprises the following steps: s1, organizing a plurality of comment texts under the same social content into a text package, and distributing labels to the text package, so that topic related packages can be obtained; s2, extracting keywords representing topics from the topic related package, and constructing topic related vectors through the keywords; and S3, inputting topic related vectors and word vectors into the double-branch neural network as vector pairs, and predicting the text instance through the double-branch neural network to obtain the category of the text instance and the category of the package. According to the method and the device, the text information can be effectively classified on the premise that the social media text data is fast in change, difficult to annotate and seriously deficient in annotation data.

Description

A semantic-based deep multi-instance weakly supervised text classification method

Technical Field

The present invention relates to the technical field of natural language processing, and in particular to a semantic-based deep multi-instance weakly supervised text classification method.

Background Art

With the development of the Internet and social media, a huge amount of text data is generated every day, and data analysts usually only focus on data related to their own fields or a specific topic. This requires filtering out field or topic-related data from massive data. In the process of data crawling, in order to obtain as much data as possible, the crawling rules are set relatively loosely, which ensures the richness of the data, but also introduces a lot of data that is irrelevant to the topic. Before analysis, the data that is truly related to the topic must be filtered out to ensure the accuracy of the analysis. This task can be regarded as a binary classification problem of whether the text is related to the topic. If there is a corresponding binary classification annotated data set, it is just a very simple supervised text classification problem. However, in the Internet era, many new network terms are generated every year, and the speed of natural language changes has greatly increased, which also means that the timeliness of the annotated data is greatly reduced. At the same time, topics in social media change with events, and content is updated frequently. Unless there is a large amount of annotated data updated continuously, the annotated data may be very different from the data contained in the new event. However, in the current existing technology, there is no text classification method for scenarios where social media text data changes rapidly, annotation is difficult, and annotated data is severely scarce.

Summary of the invention

The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a semantic-based deep multi-instance weakly supervised text classification method.

In order to achieve the above-mentioned object of the present invention, the present invention provides a semantic-based deep multi-instance weakly supervised text classification method, comprising the following steps:

S1, organize multiple comment texts under the same social content into text packages, and use the hierarchical characteristics and topic relevance of social media data to automatically assign labels to text packages, thereby obtaining topic-related packages;

S2, extracts keywords representing topics from topic-related packages and constructs topic-related vectors through keywords; by constructing topic-related vectors, data imbalance can be avoided and collection and computational costs can be reduced.

S3, the topic-related vector and the word vector are input into the two-branch neural network as a vector pair, and the text instance is predicted by the two-branch neural network to obtain the category of the text instance and the category of the package.

Further, the S2 comprises the following steps:

S2-1, cluster topic-related packages into several topics using the LDA algorithm and extract topic keywords;

S2-2, the fasttext model is used to embed each keyword in the topic, and the average vector of the strongly related keywords is used as the topic-related vector;

的向量表示为

因此话题相关向量表示为：Keywords

The vector representation is

Therefore, the topic-related vector is expressed as:

Where V _T represents the topic-related vector;

K represents the total number of keywords strongly related to the topic.

Furthermore, the method further comprises: converting the vector pair into a dense vector and inputting the dense vector into a dual-branch neural network;

和话题相关向量V_T做内积后再叠加到词向量从而得到，公式如下：The dense vector is obtained by word vector

After doing the inner product with the topic-related vector V _T and then superimposing it on the word vector, the formula is as follows:

是叠加后的词向量，是双分支神经网络的输入；in

is the superimposed word vector, which is the input of the two-branch neural network;

[·,·] indicates the concatenation of two vectors;

表示词向量；

Represents word vectors;

× represents matrix bitwise multiplication;

V _T represents the topic-related vector;

Therefore, the input of the two-branch neural network can be expressed as:

Where _xij represents the jth text table in the ith package, which is the input of the two-branch neural network;

表示第一个叠加后的词向量，

表示第二个叠加后的词向量，

表示第L个叠加后的词向量；

represents the first superimposed word vector,

represents the second superimposed word vector,

Represents the Lth superimposed word vector;

L represents the number of words contained in the text;

[·,·,...,·,] represents a set of vectors.

和V_T做内积后，叠加到词向量上对神经网络抽取特征和分类更有帮助。Through practice, we find that the word vector in the text

After doing the inner product with _VT , superimposing it on the word vector is more helpful for neural network feature extraction and classification.

Furthermore, the following operations are performed in the dual-branch neural network:

The hidden variable Z = { _zij } is introduced to characterize the relationship between text instances and packages. _Zij represents the contribution of the jth instance of the ith package to the positive package contribution of package i, _0≤zij≤1 ; if Z follows the distribution p(z), then the probability that the ith package is a positive package can be expressed as:

p(Y _i ＝1|X _i )＝f _{j∈{1,...,N}} {p _θ (y _ij ＝1|x _ij ,z _ij )·[z _ij -γ]} (7)

Among them, _Xi represents the i-th package;

_Yi represents the label of the i-th package;

f is the mapping operator from text instances to packages;

N represents the number of packets;

p _θ (y _ij ＝1|x _ij ,z _ij ) represents the probability that instance x _ij is predicted to be 1;

y _ij represents the annotation of the jth text table in the i-th package;

x _ij represents the j-th text table in the i-th package;

z _ij represents the contribution of the jth instance of the ith package to the positive package contribution of package i.

γ is the average proportion of positive instances in a bag.

The categories of the bag are linked to the categories of the text instances, so as to achieve the goal of learning the categories of the text instances themselves through the categories of the bag.

Furthermore, f is a mean operator. In the problem scenario solved by the present invention, the positive instances contained in the positive bag are not sparse, and it is easy to predict false positive bags using the maximum value and attention mechanism as mapping operators, which reduces the accuracy, so the mean operator is used.

Furthermore, the following operations are also performed in the dual-branch neural network:

In multi-instance text classification, the learning goal is to minimize the cross entropy of the bag:

L _i =-[Y _i 'logp(Y _i |X _i )-(1-Y _i ')log(1-p(Y _i |X _i ))] (8)

Where _Li represents the cross entropy of the i-th package;

p(Y _i |X _i ) represents the probability that instance _Xi is predicted to be _Yi , which is the output of branch one;

_Xi represents the input features of the i-th text package, which is the input of branch one;

_Yi represents the predicted value of the i-th text package,

_Yi ' represents the label of the i-th text package;

For a positive packet, _Yi '=1, 1- _Yi '=0, so _Li is expressed as:

For the negative bag _Yi ' = 0, so _Li is expressed as:

为0，达到最小值；All text instances in the negative bag are negative, and when all p _θ (y _ij |x _ij ,z _ij ) and z _ij are negative,

is 0, reaching the minimum value;

等同于p(Y_i|X_i)的似然值极大化，将公式(7)代入，则Positive package, minimize

This is equivalent to maximizing the likelihood value of p(Y _i |X _i ). Substituting formula (7) into the equation, we get

Then formula (11) introduces variational inference

Where x _ij represents the jth text table in the i-th package;

y _ij represents the annotation of the jth text table in the i-th package;

z _ij represents the contribution of the jth instance of the ith package to the positive package contribution of package i;

γ is the average proportion of positive instances in the bag;

p _θ (y _ij |x _ij ) represents the probability that instance x _ij is predicted to be y _ij ;

p(z) represents the p distribution of contribution z;

p _θ (y _ij |x _ij ,z) represents the probability that instance x _ij is predicted to be y _ij when the contribution of x _ij is z;

p _θ (y _ij |x _ij ,z>γ) represents the contribution of x _ij z>γ and the probability that instance x _ij is predicted to be y _ij ;

q(z) represents the q distribution of contribution z;

E _Z～q [·] represents the mean under the condition that Z follows q distribution.

逼近q(z)。According to the idea of variational inference, the present invention uses

Approximate q(z).

Furthermore, the neural network is any one of TextCNN, LSTM, and Transformer.

Furthermore, the method further includes: S4, optimizing the network parameters of the dual-branch neural network:

目标函数为：S4-1, step E takes KL minimization as the goal and optimizes the parameters

The objective function is:

表示对

p_θ(z|x,Y)进行KL最小化；in

Express

p _θ (z|x,Y) performs KL minimization;

表示双分支神经网络中分支一的输出，为文本实例的类别；

Represents the output of branch one in a two-branch neural network, which is the category of the text instance;

p _θ (z|x,Y) represents the output of branch 2 in the two-branch neural network, which is the connection between the text instance and the bag;

z represents contribution;

Y refers to the category of the package;

x refers to a text instance in the package;

是两条分支的参数；θ and

are the parameters of the two branches;

和分布p_θ(z|x,Y)的差异，最小化L_E就是让分布q和分布p逐步逼近，达到收窄下界和证据的差值的目标。Formula (15) is used to calculate the distribution

Minimizing _LE is to make the distribution q and the distribution p gradually approach each other, so as to achieve the goal of narrowing the difference between the lower bound and _the evidence.

In a neural network determined by parameter θ, the true distribution of z is analogous to the posterior distribution p _θ (y|x).

表示对

p_θ(y|x)进行KL最小化；in

Express

p _θ (y|x) is KL minimized;

表示Y＝1条件下双分支神经网络中分支一的输出，为文本实例的类别；

represents the output of branch one in the two-branch neural network under the condition of Y=1, which is the category of the text instance;

p _θ (y|x) represents the value calculated by the neural network determined by parameter θ when θ is fixed. For negative packets, p _θ (y|x) of each instance is 0;

Formula 16 is a further evolution of Formula 15. Since the negative bag contains only irrelevant text, that is, in the negative bag, the category of all text instances is 0, and the contribution of the instance to the bag being positive is 0, it can be regarded as supervised learning. Then the only thing left is the positive bag, that is, Y = 1. Under the word condition, the true distribution p _θ (z|x,Y) obeyed by z is approximated to p _θ (y|x), and Formula 16 is obtained.

therefore

表示对

p′进行KL最小化；in

Express

p′ is KL minimized;

表示Y＝1条件下双分支神经网络中分支一的输出，为文本实例的类别；

represents the output of branch one in the two-branch neural network under the condition of Y=1, which is the category of the text instance;

_Yi = 1 means the i-th packet is positive;

x _ij represents the jth text in the i-th package;

y _ij represents the positive contribution of the jth instance in the ith package to the package;

p′＝p _θ (y|x), which represents the value calculated by the neural network determined by parameter θ when θ is fixed. For negative packets, p _θ (y|x) of each instance is 0;

In formula (17), p′ is used to replace p _θ (y|x). Since the increase and decrease of logp _θ (y _ij |x _ij ) is consistent with that of p _θ (y|x), logp _θ (y _ij |x _ij ) is used to replace p _θ (y|x) to speed up the convergence. When the package is negative, that is, when Y = 0, all probability values of the distribution are set to 0, so formula (17) is obtained.

使同样文本下

和p_θ(z|x,Y)的KL散度不变，然后通过优化参数θ，让期望最大化，对数似然值的期望表示如下S4-2, M-step fixed parameters

Make the same text

The KL divergence of and p _θ (z|x,Y) remains unchanged, and then the expectation is maximized by optimizing the parameter θ. The expectation of the log-likelihood value is expressed as follows

L _M ＝E _Z～q [logp _θ (y _ij |x _ij ,z>γ)] (18)

Where L _M represents the expectation of the log-likelihood value;

E _Z～q [·] represents the mean under the condition that Z follows q distribution;

p _θ (y _ij |x _ij ,z>γ) represents the probability that instance i in the text bag is predicted as the positive text after passing through the θ branch when z>γ;

z represents contribution;

γ is a hyperparameter that represents the average proportion of positive text instances in all positive bags;

According to the definition of formula (7), L _M can be split into two parts with z = γ as the boundary. For z>γ, it is only meaningful for _yij = 1, while for z<γ, it is only meaningful for _yij = 0. Therefore, the cost function L _M of the M-step can be further decomposed into

Where r is a hyperparameter, which represents the average proportion of positive text instances in all positive bags;

p _θ (y _ij =1|x _ij ) represents the probability that text instance j in bag i is a positive text;

p _θ (y _ij = 0|x _ij ) represents the probability that text instance j in bag i is a negative text;

y _ij = 1 means that text instance j in bag i is positive;

y _ij = 0 means that the text instance j in bag i is negative;

Formula (19) is obtained by splitting formula (18) based on z=γ.

Formula (19) can be transformed into cross entropy

L _M =y' _ij logp _θ (y _ij |x _ij )-(1-y' _ij )log(1-p _θ (y _ij |x _ij )) (20)

The meaning of p _θ (y _ij |x _ij ) in formula (20) is consistent with that of p _θ (y _ij =1|x _ij ) in formula (19), which is the probability that the text is positive. 1-p _θ (y _ij |x _ij ) represents the probability that the text is negative, which is consistent with the meaning of p _θ (y _ij =0|x _ij ). Formula (20) is the discretized form of formula (19), which is discretized into cross entropy.

Where _y'ij is the pseudo label of _yij , in the positive bag, it is determined by z, in the negative bag, all are 0;

Where mean(·) means finding the average;

γ is the average proportion of positive instances in a bag.

M步通过优化参数θ最大化期望。The parameter optimization of the present invention is different from the traditional EM algorithm. Due to the introduction of variational inference, the E step calculates the expectation by narrowing the difference between the evidence and the variational lower bound, and optimizes the parameters when narrowing the lower bound.

The M step maximizes the expectation by optimizing the parameters θ.

In summary, due to the adoption of the above technical solution, the present invention has the following advantages:

1) By introducing latent variables and variational inference, the idea of applying dual-branch deep networks to traditional multi-instance learning is effectively improved in multi-instance text classification tasks that focus on instance-level classification problems.

2) Through the proposed weakly supervised text classification method SDMIC, the characteristics of social media text data are utilized to obtain the labels established by social media users or the categories divided by the platform as weak supervision information, and the model is effectively optimized through weak supervision information, thus solving the pain points of social media text data that changes rapidly, is difficult to label, and has a serious lack of labeled data.

Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

FIG1 is a schematic diagram of a process of extracting topic keywords and calculating topic-related vectors through an LDA model in the present invention.

FIG. 2 is a schematic diagram of a topic relevance learning model and a contribution learning model according to an example of the present invention.

FIG3 is a schematic diagram of the learning speed trend of SDMIC and supervised learning of the present invention, FIG3(a) is a variation trend of the prediction accuracy Acc of the test set, and FIG3(b) is a variation trend of the recall rate F1 score.

DETAILED DESCRIPTION

Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

Based on the objective facts described in the background technology, the time and labor costs of applying purely supervised text classification to extract social media topic data are very high. Unsupervised or weakly supervised classification that does not require labeled data is more practical.

Moreover, although the text data in social media is large in volume and changes rapidly, they are generated by user behavior, and there is a certain correlation between the data and a specific hierarchical structure, which provides certain clues for data filtering. For example, in a forum, users usually go to the forum related to this topic to discuss issues. Most of the text content generated in the forum is related to the topic of the forum, but there are also many non-topic related contents. The topic of the entire forum is easy to obtain, but it is difficult to determine which individual data in the internal discussion is topic-related and which is irrelevant. If the data of a forum is regarded as a package, the topic of the package is known, and some texts in the package are related to the topic, which are positive texts, and other irrelevant texts are negative texts. There are positive texts in the package, and there may also be negative texts. This problem is very consistent with the inexact supervised learning in weakly supervised learning. According to the structure of the data, the problem can be further defined as multiple instance learning in inexact supervision, namely MIL.

Multi-instance learning is divided into two branches according to the different classification focuses: focusing on the classification of packages and focusing on the classification of entities in packages. In topic-related text data filtering, the goal of filtering data is to extract positive text from the package and filter out negative text based on all the clues that can be obtained. Therefore, the focus of the problem is how to improve the classification effect of text entities.

1 Related technologies

1.1 Multiple Instance Learning

Multiple instance learning is a typical inexact learning, that is, the granularity of the label is coarser than the granularity of the actual task. It was introduced by Dietterich et al. in the mid-1990s. Dietterich proposed the idea of multiple instances, applied this idea to drug activity prediction, and proposed the APRs learning rule. There are three optional algorithms for the APRs learning rule: the noise-tolerant standard algorithm, the "outside-in" algorithm, and the "inside-out" algorithm. They are essentially to find the hyperplane boundary for positive instances. In the early days, MIL, like other machine learning algorithms, was mainly implemented through traditional statistical learning methods. At the same time, MIL has two different prediction focuses, one is the relatively easy prediction of the package itself, and the other is the more difficult prediction of instances in the package. For example, the EE-DM algorithm proposed by Zhang, Q. et al., combined with the EM algorithm and diverse density, turned MIL to predicting instances in the package. After that, the support vector machine (SVM) method was introduced into MIL. The typical one is the MISVM algorithm proposed by S. Andrews et al., which regards the MIL problem as a maximum margin problem. The extension of the SVM learning method can obtain a mixed integer quadratic programming that can be solved heuristically. Traditional MIL also includes SbSVM and StMIL proposed by Bunescu et al., and MICA proposed by Mangasarian et al. Some of these methods focus on predicting instances within a package, but can only accurately predict a few instances with obvious features in the positive package.

With the development of deep learning, MIL has also begun to introduce deep neural networks. A typical approach is to input a package as a batch, extract features from the text in the package through the neural network, calculate the predicted probability value at the instance level, and then use an operator to combine the probability values of all the texts in the package to integrate the probability of the package, which is the same as the label of the package as supervision information to optimize the network. Ilse, M. et al. proposed to use the gated attention mechanism (attention) as an operator to predict the probability from the strength level to the package level, and applied this method to image recognition. Wang, Y., Li et al. compared the effects of five operators, namely maximum pooling, average pooling, linear softmax, exponential softmax and attention, in object positioning. Shi, X. et al. integrated attention into the loss based on Ilse, M. et al. The attention mechanism usually focuses on more obvious features, so the above methods work well for packages containing sparse positive instances, but perform poorly for packages with more positive instances. The optimization process of the present invention is based on the instance level, so the prediction effect is very good regardless of whether the positive instances in the package are sparse or dense.

Luo, Z. et al. introduced a dual-branch neural network into MIL and optimized the two neural networks in combination with the EM algorithm. However, their method was applied to action localization, and both branches were optimized using cross entropy as the loss function. This method was not ideal in text classification. Li, B. et al. applied the dual-branch neural network MIL to lesion localization in medical images. One neural network branch extracted the features of the large image (package), and the other branch extracted the features of the small image (instance), and then fused them to classify the large image. The patent of this invention also introduces a dual-branch neural network into multi-instance learning, but applies it to text classification. In addition, the patent of this invention introduces variational inference, converts the classification problem of the package into instance classification prediction and latent variable distribution prediction, and uses cross entropy and KL divergence to optimize the dual-branch network.

1.2 Weakly supervised text classification

The mainstream approach to text classification is still supervised learning, but as the Internet generates more and more data, weak supervision methods are also being tried. Hingmire S et al. proposed to pre-assign one or more labels to the topics to be extracted by LDA through prior knowledge of corpus statistics, and then classify documents according to the subsequent topic ratio. In this method, the classification process is completely limited by the scope of extraction rules and training data, and the classification effect is poor. Meng Y. et al. proposed to produce pseudo-documents with labels based on seed information, train a neural network model with pseudo-documents, and train a neural network model with real data self-supervision, and the two models share network parameters for text classification. This method is similar to the method proposed by Hingmire S et al. When pseudo-annotated data is generated through rules, noise will be introduced. According to the principle of "garbage in, garbage out" for machine learning model training, the classification effect is also limited. Li, C. et al. constructed a topic semantic space by extracting topic-related keywords, and then trained the model with existing topic annotation data to determine the relevance of text and topic space. When a topic without labeled data appears, only topic-related keywords need to be extracted, and the model predicts the topic relevance of the text. This method is more advanced than the previous two, but it only solves the problem of effectively predicting topics that may arise in the future, and it still requires a large amount of training data support. These training data contain a large number of known topics, which is completely different from the problem that the present invention wants to solve. The problem that the present invention patent wants to solve is to classify the topic relevance of the text in the absence of accurate annotation data. Therefore, the present invention patent proposes to use the labels provided by the first-posting users on the social platform or the classification labels of the community itself by the platform as weak supervision signals, and introduces the idea of multiple instances to solve the problem of accurately classifying each text in the absence of accurate classification training data.

2 Proposed method

The present invention adopts deep learning to realize end-to-end multi-instance learning text classification, and performs topic relevance binary classification on social media text data. In the definition of multi-instance learning, an instance is the smallest individual. In the present invention, an instance is a single text, and a package contains multiple instances. There are two types of packages, positive packages and negative packages. The positive package contains at least one positive text, and the negative package cannot contain any positive text. In the present invention, some texts with certain relationships in social media are integrated into packages (for example: all comments under a video in Station B can be used as a text package, and all replies to a post in Tieba can also be used as a package). When classifying a certain topic, search for content related to the topic from social media, and extract all content to form multiple positive packages. When collecting data, the label given by the social media platform is used as the topic label of the package (for example, the category of Tieba, the category label of the video uploaded by the user). According to the definition of multi-instance learning, in order to complete the task of weakly supervised instance-level classification, in addition to collecting data under the topic as a positive package, it is also necessary to collect text data of multiple other topics to form a negative package, so as to achieve the goal of comparative learning. Since the goal of the task is to perform relevance binary classification on the text instances in the positive bag, the bag label is a weak supervision signal, and the instance binary classification is completed in the form of weak supervision.

In order to better introduce the characteristics of the topic, in addition to the end-to-end deep learning multi-instance text classification, the patent of the present invention also uses a statistical learning method to extract the keywords of the topic before learning, and embed the topic keywords to obtain the key features of the topic, namely the keywords, and introduce these key features into deep learning (DL for short). In natural language, each topic will have some unique representative words. Since these representative keywords have similar contexts and semantics, their positions in the dense vector space are close. This local small space in the vector space containing all keyword positions can be used to represent the vector space of the topic. The patent of the present invention uses this feature to segment the text in all packages under the same topic, and extract the topic-strong related keywords through the seeded hidden Dirichlet allocation (hereinafter referred to as Guilded-LDA) algorithm combined with the method of manually selecting topic-related categories. As the representative words of the topic, the strongly related vector of the topic T is constructed. The construction process of the topic-related vector is shown in Figure 1.

The reference vector of topic T and the dense vector of the text instance vectorized in the same space are used as input pairs and input into a neural network model with two branches for prediction. One branch predicts whether the text instance contributes to the topic relevance, and the other branch predicts whether the text instance is related to the topic.

The forward propagation architecture of the entire SDMIC is shown in Figure 2: All the texts of a package are taken as a batch. After vectorization and inner product with the topic-related vector, the texts are integrated into input data and input into two neural network branches respectively. The neural network branches can select convolutional neural networks or other neural network layers. The patent of this invention selected convolutional neural networks in the experimental process and practical application. The outputs of different layers are fused and flattened into one-dimensional vectors, converted into category prediction values through the fully connected layer, and then the prediction values are converted into prediction probabilities through softmax. The above is the forward calculation process of the entire network. In actual classification, only the forward process needs to be calculated; in training, the forward process is calculated first, and then the prediction loss (cost) of the network is calculated. The entire training process is divided into E steps and M steps. In the E step, the p branch parameters are fixed, and the KL divergence is used as the cost function to optimize the q branch; in the M step, the q branch parameters are fixed, and the cross entropy is used as the cost function to optimize the p branch parameters. Among them, KLD is the abbreviation of Kullback–Leibler divergence, that is, KL divergence, represented by _LM ; CE is the abbreviation of cross entropy, represented by _LE .

In order to optimize the vector space parameters of word embedding and the parameters of the two neural network branches, the patent of this invention uses the Kullback-Leibler divergence (hereinafter referred to as KL divergence) and the lower bound (hereinafter referred to as LB) as the loss functions of the two branches respectively, and adjusts the network parameters by minimizing the KL divergence and maximizing the LB.

In order to better realize the topic-related binary classification of text instances, we first extract keywords representing the topic from the topic-related package, build a topic-related vector through the keywords, and then input the topic-related vector and the text instance vector as a vector pair into the neural network. The two-branch neural network makes relevant predictions on the instances, and then obtains the instance category and the package category based on the instance predictions.

2.1 Topic-related vector construction

The characteristic of the MIL task is that the supervision of annotation is very weak. It is found in practice that the training data set constructs topic-related text packages and topic-irrelevant text packages. Due to the weak supervision of annotation and the weak controllability of the neural network learning process, if the topic is not constrained, the final instance classifier will tend to predict the text that is not in the negative package but in the positive package as topic-related. If the negative package of the training data set is large enough to include all non-topic-related texts, this is not a problem, but too much negative package data will easily cause data imbalance, and the collection cost and calculation cost will be greatly increased. Therefore, in order to circumvent this problem, the patent of this invention adds the construction of topic-related vectors.

和无关话题集

为了使人工介入的工作最小化l和m应尽量小。取话题集

的关键词，根据关键词在相关话题和非相关话题中的权重对关键词进行排序，得到K个与话题T最相关的关键词。The present invention proposes to extract keywords of topics through the LDA algorithm. The present invention also adopts this method to aggregate all text instances in the positive bag in the training data into several topics through the LDA algorithm, and then screen the topic set related to topic T.

and irrelevant topics

In order to minimize the amount of manual intervention, l and m should be as small as possible.

Keywords are sorted according to their weights in relevant topics and irrelevant topics, and K keywords most relevant to topic T are obtained.

Where C represents a set of strongly related keywords of topic T;

表示排名第一的话题关键词。

Indicates the top-ranked topic keyword.

Table 1 shows the relevant keywords of the topics “Jewelry” and “Beauty” extracted from the Amazon Review dataset, and Table 2 shows the relevant keywords of the topics “Car” and “Sports” from the Toutiao News dataset.

Table 1. Keywords of the topics “Jewelry” and “Beauty” extracted from the Amazon dataset

Table 2. Keywords of the topics “Car” and “Sports” extracted from the Toutiao news dataset

通过词嵌入向量化为

话题相关向量可表示为The patent of this invention adopts the fasttext model proposed by Joulin, A. et al. to embed each keyword in the topic, and uses the average word vector of the topic-strongly related keywords as the topic-related vector.

Through word embedding vectorization

The topic-related vector can be expressed as

K represents the total number of keywords strongly related to the topic;

V _T represents the topic-related vector;

和话题相关向量V_T做内积后，叠加到词向量上对神经网络抽取特征和分类更有帮助。因此，文本实例的嵌入采用以下方式As we all know, dense embedding of words not only maps words to vector space, but also contains syntactic and semantic information. The hidden correlation between two words can be obtained by simple operations such as subtracting two word vectors, inner product, and Euclidean distance. Mikolov, T. et al. proposed that subtracting two word vectors can obtain the relationship between the two words. Li, CL, Zhou et al. proposed to use both word vector subtraction and vector inner product to represent the interaction between two words. In this task, it is found through practice that the word vector of the i-th word in the text is the text instance vector

After doing the inner product with the topic-related vector V _T , superimposing it on the word vector is more helpful for neural network feature extraction and classification. Therefore, the embedding of text instances is done in the following way

是叠加后的词向量，是双分支神经网络的输入；[·,·]表示两个向量连接，如果原来

是D维的，

则是(2D)维。

是

和V_T的按元素相乘：in

is the superimposed word vector, which is the input of the two-branch neural network; [·,·] represents the connection of two vectors.

It is D-dimensional.

It is (2D) dimensional.

yes

And element-wise multiplication of V _T :

× represents matrix bitwise multiplication;

According to the above calculation, the input of the two-branch neural network can be expressed as:

Where L represents the number of words contained in the text, and [·,·,...,·,] represents a set of vectors or a matrix.

3.2 Dual-branch prediction neural network

MIL is weakly supervised learning. The entire task is divided into two levels: package classification and instance classification. Because each package in the training data is labeled, package classification can be regarded as supervised learning, while instances are not labeled and can only rely on package labels for weak supervision. In different tasks, the emphasis on the two-level tasks is different. Maron, O. and others pointed out that if the goal of the task is to classify instances and the positive instances in the positive package are not too sparse, the training process should try to reduce its dependence on package labels. In the tasks involved in the patent of this invention, more emphasis is placed on instance classification, and the vast majority of packages have positive instances higher than 50%. Therefore, the patent of this invention focuses on characterizing the characteristics of each instance.

Assume that there are 2N text packages, the first N packages related to topic T are positive packages, represented as [X ₁ ,X ₂ ,…,X _N ], and the next N packages are collected from other topics and are not related to topic T, represented as [X _N+1 ,X _N+2 ,…,X _2N ], and each package contains n text instances. The i-th package is labeled as _Yi , and the j-th text in the i-th package is represented as x _ij as the input feature value of the neural network after the text is processed. The patent of the present invention aims to link the category of the package with the category of the text instance, so as to achieve the goal of learning the category of the text instance itself through the category of the package. The patent of the present invention introduces a hidden variable Z = {z _ij } to characterize the relationship between the text instance and the package, z _ij represents the contribution of the j-th instance of the i-th package to the positive package contribution of package i, 0≤z _ij ≤1, assuming that Z follows the distribution p(z),

z _ij ～p(z) (6)

Then the probability that the i-th packet _Yi is positive can be expressed as:

p(Y _i ＝1|X _i )＝f _{j∈{1,…,N}} {p _θ (y _ij ＝1|x _ij ,z _ij )·[z _ij -γ]} (7)

Among them, p _θ (y _ij ＝1|x _ij ,z _ij ) represents the probability that instance x _ij is predicted to be 1 (i.e. positive), γ is the average proportion of positive instances in the bag, and f is the mapping operator from text instances to bags. Common operators for f include maximum value, mean value, attention mechanism, etc. Since the positive instances contained in the positive bag are not sparse in the problem scenario solved by the present invention, it is easy to predict false positive bags using the maximum value and attention mechanism as mapping operators, which reduces the accuracy, so the mean operator is used.

3.2.1 Variational Inference

In the multi-instance text classification to be solved by this invention, the learning goal is to minimize the cross entropy of the package.

L _i =-[Y _i 'logp(Y _i |X _i )-(1-Y _i ')log(1-p(Y _i |X _i ))] (8)

Where _Li represents the cross entropy of the i-th package;

p(Y _i |X _i ) represents the probability that instance _Xi is predicted to be _Yi , which is the output of branch one;

_Xi represents the input features of the i-th text package, which is the input of branch one;

_Yi represents the predicted value of the i-th text package,

_Yi ' represents the label of the i-th text package;

For a positive packet, _Yi '=1, 1- _Yi '=0, so _Li can be expressed as:

For the negative bag _Yi ' = 0, _Li can be expressed as:

为0，达到最小值。因此，负包按照有监督的逻辑进行学习即可。By definition, all text instances in the negative bag are negative, and when all p _θ (y _ij |x _ij ,z _ij ) and z _ij are negative,

is 0, reaching the minimum value. Therefore, the negative bag can be learned according to the supervised logic.

等同于p(Y_i|X_i)的似然值极大化，将公式(7)代入，则Positive package, minimize

This is equivalent to maximizing the likelihood value of p(Y _i |X _i ). Substituting formula (7) into the equation, we get

Then formula (11) introduces variational inference

逼近q(z)，

是通过一条神经网络来表示。而p_θ(y_ij|x_ij)用另外一条神经网络来表示。因此，整个方法引入了二分支神经网络，一条网络预测文本实例的类别，另一条网络文本实例与包之间的联系。According to the idea of variational inference, the present invention uses

Approximating q(z),

is represented by a neural network. And p _θ (y _ij |x _ij ) is represented by another neural network. Therefore, the whole method introduces a two-branch neural network, one network predicts the category of the text instance, and the other network predicts the connection between the text instance and the bag.

来表示两条分支的输出，其中θ和

是两条分支的参数。There are two key points in the task: p _θ (y _ij ＝1|x _ij ) and z _ij prediction. The present invention uses a dual-branch neural network to complete the prediction of these two goals. The neural network can choose TextCNN, LSTM or even the currently popular Transformer. Many works have made good explanations of these network structures. Please refer to specific papers. The present invention will not go into details. It only uses p _θ (y _ij ＝1|x _ij ) and

To represent the output of the two branches, where θ and

are the parameters of the two branches.

3.3 Network parameter optimization

From the description of the entire network structure in the previous chapter, we can know that for all positive packets, the input is _Xi , the output p _θ ( _Yi | _Xi ) is labeled and supervised, and the learning goal of the entire network is to maximize the log-likelihood:

L＝logp(Y _i |X _i ) (13)

Combined with Jensen inequality, equation (12) can be further concluded:

According to the definition of the evidence lower bound (ELBO) in statistics, logp(Y|X) can be regarded as the evidence, E _Z～q [logp _θ (y _ij ＝1|x _ij )] is the variational lower bound, and the difference between the evidence and the variational lower bound is KL(q(z|x)||p(z|x,Y)).

M步通过优化参数θ最大化期望。The EM algorithm can be used to optimize the parameters by maximizing the log-likelihood of p(Y _i |X _i ). In the traditional EM algorithm, the E step calculates the expectation, that is, the variation lower bound, and the M step finds the parameters that maximize the expectation, thereby performing parameter estimation. The present invention introduces variational inference. Therefore, the E step calculates the expectation by narrowing the difference between the evidence and the variational lower bound, and optimizes the parameters when narrowing the lower bound.

The M step maximizes the expectation by optimizing the parameters θ.

3.3.1 E-step narrowing difference

目标函数为The goal of the E step is to narrow the difference between the lower bound and the evidence so that the lower bound is close to the expectation. According to the understanding of formula (14), the smaller the KL divergence, the closer the lower bound is to the evidence logp(Y|X). Therefore, the E step aims to minimize KL and optimize the parameters

The objective function is

The classification of packages and instances is a binary classification task. In this task, there is a certain relationship between z and _yij . For example, the contribution z of a negative text instance to a package being judged as a positive package can be regarded as 0. Therefore, the present invention assumes that in a neural network determined by parameter θ, the true distribution of z is analogous to the posterior distribution p _θ (y|x).

For the text instance in the positive bag, p _θ (y=1|x) is the value calculated by the neural network with parameter θ fixed, while for the negative bag, p _θ (y|x) of each instance should be 0, so

From the meaning of formula (17), _LE can be understood as that the contribution of a text instance to the positive prediction of the bag and the relevance of this instance to the topic try to follow the same distribution.

3.3.2 M-step maximization of the lower bound to optimize θ

M步固定参数

使同样文本下

和p_θ(z|x,Y)的KL散度不变，然后通过优化参数θ，让对数似然的期望最大化。对数似然值的期望L_M表示如下：According to the definition of formula (14), the cost function of the whole problem consists of two parts: the expectation of the log-likelihood of the predicted probability and the KL divergence between the latent variable and the true distribution. The E step minimizes the KL divergence, allowing the expectation and maximum likelihood to be passively approximated, and the parameter θ is fixed to adjust the optimization parameter

M-step fixed parameters

Make the same text

The KL divergence of and p _θ (z|x,Y) remains unchanged, and then the expectation of the log-likelihood is maximized by optimizing the parameter θ. The expectation of the log-likelihood value L _M is expressed as follows:

L _M =E _Z～q [logp _θ (y _ij |x _ij ,z>γ)] (18)

According to the definition of formula (7), L _M can be split into two parts with z = γ as the boundary. For z>γ, it is only meaningful for y _ij = 1, and for z<γ, it is only meaningful for y _ij = 0. Therefore, the cost function L _M of the M step can be further decomposed into

r is a hyperparameter used to measure how much contribution is considered effective. In actual use, it is set to the average proportion of positive text instances in all positive packages.

The first part represents the log-likelihood when z>γ, and the second part represents the log-likelihood when z<γ.

Formula (19) can be transformed into cross entropy

L _M =y' _ij logp _θ (y _ij |x _ij )-(1-y' _ij )log(1-p _θ (y _ij |x _ij )) (20)

Where _y'ij is the pseudo label of _yij , in the positive bag, it is determined by z, in the negative bag, all are 0;

γ is the average proportion of positive instances in a bag, ranging from (0,1). It is an empirical value determined by the density of positive instances in the data set. It has nothing to do with determining whether a bag is positive, but is closely related to the recall of the instances. mean(·) means averaging. According to formula (19), γ is the breakpoint for splitting y _ij . Here, the average value of the latent variable is introduced to standardize γ so that it is in the same numerical range as z _ij .

一样，当负包中的所有实例，对包话题相关没有任何贡献，伪标签设置为0，在正包中，话题相关概率低的实例，贡献度也不会高，同样将贡献度伪标签设置为0。and optimization

Similarly, when all instances in the negative bag do not contribute to the bag topic relevance, the pseudo label is set to 0. In the positive bag, instances with low probability of topic relevance will not contribute much, and the contribution pseudo label is also set to 0.

3 Experimental results analysis

3.1 Dataset

The patent of this invention uses AG news, Amazon Reviews, and Toutiao news to compare the effects of SDMIC and other methods. The topics corresponding to AG news and Toutiao news are field categories. The news texts are officially released, and the text format and content are more standardized. Amazon Reviews corresponds to products in a certain niche field and is a user's subjective evaluation of the product. The topics it contains are more granular, and the text is generated by users. The format and grammar are more arbitrary, which can be compared to user-generated content on social media. AG news and Amazon Reviews are English texts, and Toutiao news is Chinese data, thereby ensuring the adaptability of SDMIC to different languages.

Table 3. Description of experimental datasets

AG News: AG news contains four categories: Business, Sci_Tech, Sports, and World. The training set contains 30,000 texts for each category, and the test set contains 1,900 texts for each category. Each category in the training set and the test set is processed separately. The text contained in this category is used as a topic-related positive text instance, and the other three categories are used as topic-irrelevant text instances. Each package contains 50 texts, and topic-related instances and irrelevant instances are randomly selected in a ratio of 1:2 to 4:1 to form a positive text package. The negative package is all negative text instances, and the text in the negative package does not overlap with the negative text in the positive package, so as to simulate the structure of text and the topic relevance status in social media.

Amazon Reviews: Amazon Reviews has multiple versions. The patent of this invention adopts the most classic 2013 version, which contains 24 groups of products. Four products with moderate evaluation volume are selected as potential positive topics. The reviews of each product are used as positive text related to the topic, and the reviews of other products are negative text instances. They are combined into text packages in the same way as AG news. The text packages of each topic are divided into training set and test set according to 4:1.

Toutiao news: Toutiao news contains 12 topics. This experiment selects 4 topics with large text volume as potential positive topics. The text contained in each topic is the positive text related to the topic. Data from the other 11 topics are randomly selected as irrelevant text and combined into text packages in the same way as AG news. The text package of each topic is divided into training set and test set according to 4:1.

After data sorting, as shown in Table 3, the three data sets constitute a total of 12 topic text packages.

3.2 Experimental process and parameter setting

During the experiment, in order to prove that the proposed SDMIC does have advantages under the same conditions, in addition to conducting experiments based on the method proposed on the above dataset, other unsupervised and weakly supervised methods were introduced for comparison. At the same time, in order to evaluate the effect gap between SDMIC and supervised methods using manually labeled data, supervised text classification experiments were also conducted under the same neural network results and parameters.

3.2.1 Introduction to comparison algorithms

Guilded-LDA: Guilded-LDA adds seed keywords to each topic based on LDA, and uses the seed keywords to partially constrain the clustering direction. In the experiment, the topic keywords extracted from each topic are used as seeds for some clusters, which are topic-related clusters, while other clusters with empty seed words are used as topic-irrelevant clusters, so as to achieve the classification effect.

MISVM and SbMIL: The MISVM algorithm treats the MIL problem as a maximum margin problem and extends the SVM learning method to solve mixed integer quadratic programming. The algorithm solves the maximum margin of positive and negative bags, treats the edges of bags as the edges of instances, and predicts the polarity of instances. This method was originally applied to the MUSK dataset experiment, and the input feature extraction was modified to apply it to text classification. SbMIL, like MISVM, combines the SVM algorithm.

Weighted-MIL: A multi-instance regression method that vectorizes each instance in the package, estimates the weight of each instance vector corresponding to each category, and then calculates the weighted average of all instances in the package. The weighted average of the instances is used to calculate the category of the package through an operator.

Attention base and Gated attention base: A package is input as a batch, and the text in the package is extracted through a neural network. The predicted probability value at the instance level is calculated, and then the probability values of all the texts in the package are combined with an operator to integrate the probability of the package, which is the same as the label optimization network of the package. Attention base uses the attention mechanism as an operator to integrate the text probability value into the package probability value; Gated attention base adds a gating mechanism based on the attention mechanism.

CNN-supervised: A general text convolution classification algorithm with word embedding units.

3.2.2 Experimental parameter settings

During the experiment, we first used LDA to cluster the text in the positive bag into topics. Since the ratio of positive and negative instances in the combined data of all positive bags was set to 3:2 in the task, and the negative instances came from more than 10 topic categories, the number of clustered topics was set to 20. We determined the relevant topics through manual confirmation, and then took out the top 50 keywords from all topics, calculated the weight of each keyword to the topic and the non-topic weight ratio, and selected the top 20 topic keywords based on the ratio ranking.

A dictionary is generated for each dataset, the variable size of the network word embedding layer is determined, and then the embedding layer of the neural network is initialized using the fasttext English and Chinese pre-trained models.

和θ按照第2节中的方式训练，实际应用中也可引用BERT,RoBERTa等预训练大语言模型作初始化参数，然后再在训练过程中以很小的学习率微调模型。本次实验的目标是对比SDMIC和无监督方法，其他弱监督方法以及纯在表现上的优越性，与有监督分类效果的差距，而不是探索哪一种不同经典神经网络在这个任务中的表现，由于有两个神经网络分支要训练，采用大预言模型实验效率很低，因此，整个实验，在不同的方法中均使用完全一致的TextCNN网络结构来构建网络。The neural network can use TextCNN, BiLSTM and other small neural network structures as the feature extraction process, and the network parameters

and θ are trained as described in Section 2. In practical applications, BERT, RoBERTa and other pre-trained large language models can also be used as initialization parameters, and then the model can be fine-tuned with a very small learning rate during the training process. The goal of this experiment is to compare the superiority of SDMIC and unsupervised methods, other weakly supervised methods, and pure performance, and the gap with supervised classification effects, rather than exploring which different classical neural networks perform well in this task. Since there are two neural network branches to train, the efficiency of using a large oracle model experiment is very low. Therefore, throughout the experiment, the same TextCNN network structure is used to build the network in different methods.

The hyperparameter γ is used to measure the proportion of positive instances in a positive bag. In the news dataset used in this experiment, since the bag is constructed, the average proportion of positive instances is greater than that of negative instances. Based on the proportion, γ is set to 0.4. In real social media data applications, the γ value is set based on the average proportion of topic-related text.

The experimental equipment is a Tesla T4 GPU. During the training process, the learning rate of 1e-5 is used for both SDMIC and supervised binary classification. The maximum number of iterations is set to 200, overfitting detection is supported, and training is stopped early. In addition, the training is stopped if the test set loss does not decrease for 20 consecutive times, and the supervised binary classification test set loss does not decrease for 5 consecutive iterations.

For the experimental settings of other methods, the traditional multi-instance method uses CWB type features as input. In this experiment, the CountVectorizer and TfidfTransformer modules of the sklearn library are used to extract the tf-idf features of each data set, and then input into the algorithm for training to predict each text instance in the test set text package. Other deep learning methods use the same neural network structure and hyperparameter settings as SDMIC. The weak supervision method uses the same text package as input and the label of the package as the supervision information to train the model. CNN-supervised uses the label of a single text as the supervision information for training.

3.3 Experimental Results

4.3.1 Performance Analysis

(1) Evaluation indicators

Common model evaluation indicators include Accuracy (hereinafter referred to as Acc), Precision, Recall, F1 value, etc. Without loss of generality, the patent of this invention uses Acc and F1 indicators to evaluate the effect of the algorithm model.

Acc measures the average prediction accuracy of the model across all test text instances

Precision represents the proportion of true positive instances in the text instances predicted as positive, which is equivalent to the precision rate of positive text instances.

Recall represents the proportion of accurate predictions among all true positive instances, which is equivalent to the recall rate of the positive text instance.

The F1 value comprehensively considers the precision and recall of the positive text instance. F1 will be high only when both Precision and Recall are high.

Therefore, the patent of this invention uses the two indicators of Acc and F1 to simultaneously evaluate the model effect.

(2) Performance comparison analysis

By conducting experiments on three different datasets, AG News, Toutiao News, and Amazon Reviews, it can be seen from the comparative results that SDMIC has greatly improved the results of unsupervised topic clustering and other weakly supervised text classification methods in terms of Acc and F1, while the difference with the results of supervised classification that completely relies on labeled data is very small.

Table 4. Comparison of accuracy and F1 value on AG News

Table 5. Comparison of accuracy and F1 value on Toutiao News

Table 6. Comparison of accuracy and F1 value on Amazon Reviews

Tables 4 to 6 show that SDMIC's performance on the three data sets is significantly better than other unsupervised and semi-supervised methods. In principle, it transforms traditional multi-instance learning into prediction for a single text through bimolecular neural networks and EM algorithms. It is based on calculating the loss function and optimizing the network based on a single text. The experimental results prove this point. Although other methods want to make instance-level predictions, they are ultimately based on optimizing the model with packages, and the classification effect at the instance level is naturally difficult to control.

At the same time, the accuracy of each topic in the test set is only 3.219% lower than the average of supervised classification, and the F1 value is 2.602% lower. This shows that SDMIC can learn to a large extent from the annotation and text semantic features of the package.

In addition, judging from the experimental results of different data sets, SDMIC can be applied to the screening of relevant data in different languages and topics of different granularity.

3.3.2 Training speed analysis

In addition to the prediction accuracy of the method, the patent of this invention also conducts a comparative analysis of the learning speed of different methods during the training process and the changing trend of the recall value of the positive text. Figure 3 shows the changing trend of the prediction accuracy Acc and recall F1 score of the test set during the training process of SDMIC and supervised classification methods, using the Car topic of the headline short news as training and test data.

Figure 3(a) shows the trend of the prediction accuracy of the test set as the number of training iterations changes. It can be seen that SDMIC is a weakly supervised learning method, and the entire method architecture contains two deep learning networks. The parameters in the two networks are optimized in turn by the E-M method. Under the same conditions, the learning speed is indeed much slower than that of supervised learning. Figure 3(b) shows the change of recall rate. From the trend, the recall rate of the supervised method changes more sensitively, but the recall rate convergence process of SDMIC is more stable.

Although supervised methods do have advantages in terms of learning speed, in actual engineering applications, supervised methods rely entirely on manually labeled data, which is a major drawback of social media text data mining. After all, large amounts of continuous labeled data require huge costs in time and manpower. Compared with this cost, the problem of learning speed in the training process can be almost ignored.

In summary, the method of the present invention conducted a comparative analysis on the proposed SDMIC in different languages (Chinese and English), different types, and different topics. The new method was experimented and evaluated on multiple topics of AG News, Toutiao News, and Amazon Reviews datasets, and achieved the ideal effect of weakly supervised text classification.

Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims (5)

1. A semantic-based deep multi-instance weakly supervised text classification method, characterized by comprising the following steps: S1, organize multiple comment texts under the same social content into text packages, assign labels to the text packages, and thus obtain topic-related packages; S2, extract keywords representing topics from topic-related bags and construct topic-related vectors through keywords; S3, input the topic-related vector and the word vector as a vector pair into the two-branch neural network, and predict the text instance through the two-branch neural network to obtain the category of the text instance and the category of the package; The following operations are performed in the dual-branch neural network: The hidden variable Z = { _zij } is introduced to characterize the relationship between text instances and packages. _Zij represents the contribution of the jth instance of the ith package to the positive package contribution of package i, _0≤zij≤1 ; if Z follows the distribution p(z), then the probability that the ith package is a positive package can be expressed as: p(Y _i ＝1|X _i )＝f _{j∈{1,...,N}} {p _θ (y _ij ＝1|x _ij ,z _ij )·[z _ij -γ]} (7) Wherein, f is a mapping operator from text instances to packages, and f is a mean operator; N represents the number of packets; p _θ (y _ij ＝1|x _ij ,z _ij ) represents the probability that instance x _ij is predicted to be 1; The following operations are also performed in the dual-branch neural network: In multi-instance text classification, the learning goal is to minimize the cross entropy of the bag: L _i =-[Y _i 'logp(Y _i |X _i )-(1-Y _i ')log(1-p(Y _i |X _i ))] (8) Where _Li represents the cross entropy of the i-th package; p(Y _i |X _i ) represents the probability that instance _Xi is predicted to be _Yi , which is the output of branch one; _Xi represents the input features of the i-th text package, which is the input of branch one; _Yi represents the predicted value of the i-th text package, _Yi ' represents the label of the i-th text package; For a positive packet, _Yi '=1, 1- _Yi '=0, so _Li is expressed as:

For the negative bag _Yi ' = 0, so _Li is expressed as:

All text instances in the negative bag are negative, and when all p _θ (y _ij |x _ij ,z _ij ) and z _ij are negative,

is 0, reaching the minimum value; Positive package, minimize

This is equivalent to maximizing the likelihood value of p(Y _i |X _i ). Substituting formula (7) into the equation, we get

Then formula (11) introduces variational inference

Where x _ij represents the jth text table in the i-th package; y _ij represents the annotation of the jth text table in the i-th package; z _ij represents the contribution of the jth instance of the ith package to the positive package contribution of package i; γ is the average proportion of positive instances in the bag; p _θ (y _ij |x _ij ) represents the probability that instance x _ij is predicted to be y _ij ; p(z) represents the p distribution of contribution z; p _θ (y _ij |x _ij ,z) represents the probability that instance x _ij is predicted to be y _ij when the contribution of x _ij is z; p _θ (y _ij |x _ij ,z＞γ) represents the contribution of x _ij z＞γ and the probability that instance x _ij is predicted to be y _ij ; q(z) represents the q distribution of contribution z; E _Z～q [·] represents the mean under the condition that Z follows q distribution. 2. According to a semantic-based deep multi-instance weakly supervised text classification method according to claim 1, it is characterized in that S2 comprises the following steps: S2-1, cluster topic-related packages into several topics using the LDA algorithm and extract topic keywords; S2-2, the fasttext model is used to embed each keyword in the topic, and the average vector of the strongly related keywords is used as the topic-related vector; Keywords

The vector representation is

Therefore, the topic-related vector is expressed as:

Where V _T represents the topic-related vector; K represents the total number of keywords strongly related to the topic. 3. The semantic-based deep multi-instance weakly supervised text classification method according to claim 1, further comprising: converting the vector pair into a dense vector and inputting it into a dual-branch neural network; The dense vector is obtained by word vector

After doing the inner product with the topic-related vector V _T and then superimposing it on the word vector, the formula is as follows:

in

is the superimposed word vector, which is the input of the two-branch neural network; [·,·] indicates the concatenation of two vectors;

Represents word vectors;

× represents matrix bitwise multiplication; V _T represents the topic-related vector; Therefore, the input of the two-branch neural network can be expressed as:

Where _xij represents the jth text table in the ith package, which is the input of the two-branch neural network;

represents the first superimposed word vector,

represents the second superimposed word vector,

Represents the Lth superimposed word vector; L represents the number of words contained in the text; [·,·,...,·,] represents a set of vectors. 4. A semantic-based deep multi-instance weakly supervised text classification method according to claim 1, characterized in that the neural network is any one of TextCNN, LSTM, and Transformer. 5. According to the semantic-based deep multi-instance weakly supervised text classification method of claim 1, it is characterized by further comprising: S4, optimizing the network parameters of the dual-branch neural network: S4-1, step E takes KL minimization as the goal and optimizes the parameters

The objective function is:

in

Express

p′ is KL minimized;

represents the output of branch one in the two-branch neural network under the condition of Y=1, which is the category of the text instance; _Yi = 1 means the i-th packet is positive; p _θ (y|x) represents the value calculated by the neural network determined by parameter θ when θ is fixed. For negative packets, p _θ (y|x) of each instance is 0; S4-2, M-step fixed parameters

Make the same text

The KL divergence of and p _θ (z|x,Y) remains unchanged,

represents the output of branch one in the two-branch neural network, which is the category of the text instance; p _θ (z|x,Y) represents the output of branch two in the two-branch neural network, which is the connection between the text instance and the bag; Then, by optimizing the parameter θ, the expectation is maximized and the expectation of the log-likelihood value is expressed as follows L _M =E _Z～q [logp _θ (y _ij |x _ij ,z>γ)] (18) Where L _M represents the expectation of the log-likelihood value; E _Z～q [·] represents the mean under the condition that Z follows q distribution; p _θ (y _ij |x _ij ,z＞γ) represents the probability that instance i in the text bag is predicted as the positive text after passing through the θ branch when z＞γ; z represents contribution; γ is a hyperparameter; According to the definition of formula (7), L _M can be divided into two parts with z = γ as the boundary. For z>γ, it is only meaningful for y _ij = 1, while for z<γ, it is only meaningful for y _ij = 0. Therefore, the cost function L _M of the M step can be further decomposed into

Where r is a hyperparameter; p _θ (y _ij =1|x _ij ) represents the probability that text instance j in bag i is a positive text; p _θ (y _ij = 0|x _ij ) represents the probability that text instance j in bag i is a negative text; y _ij = 1 means that text instance j in bag i is positive; y _ij = 0 means that the text instance j in bag i is negative; Formula (19) can be transformed into cross entropy L _M =y′ _ij logp _θ (y _ij |x _ij )-(1-y′ _ij )log(1-p _θ (y _ij |x _ij )) (20) Where y′ _ij is the pseudo label of y _ij , which is determined by z in the positive bag and all zeros in the negative bag;

Where mean(·) means finding the average; γ is the average proportion of positive instances in a bag.

Images