CN111914054A - System and method for large scale semantic indexing - Google Patents

Abstract

Described herein are embodiments of a deep layer-by-layer Extreme Multi-Label Learning and Classification (XMLC) framework for facilitating semantic indexing of documents. In one or more embodiments, the deep layer-by-layer XMLC framework includes two sequential modules, a deep layer-by-layer multi-label learning module and a layer-by-layer pointer generation module. In one or more embodiments, the first module utilizes class-based dynamic max-pooling and macro-F-metric-based weight adjustment to decompose the terms of the domain ontology into multiple levels and build a specific convolutional neural network for each level network. In one or more embodiments, the second module combines the layer-wise output into a final summary semantic index. The effectiveness of the deep layer-by-layer XMLC framework implementation is demonstrated by comparing it with several state-of-the-art methods for automatic labeling on various datasets.

Description

System and method for large-scale semantic indexing

technical field

The present disclosure generally relates to a system and method for semantic indexing. More specifically, the present disclosure relates to a system and method for semantic indexing with deep layer-by-layer extreme multi-label learning.

Background technique

With the explosive growth of scientific literature, efficient semantic indexing methods are required to build retrieval systems. Even with efficient techniques, the semantic indexing process still involves manually curating key aspects from the scientific literature. To outline the main topic of the article, domain experts are usually invited to manually index the article using keywords selected from the domain ontology.

Therefore, there is a need for systems and methods for large-scale semantic indexing to improve automatic tagging efficiency.

SUMMARY OF THE INVENTION

In one aspect of the present disclosure, there is provided a computer-implemented method for multi-label learning and classification using one or more processors to perform steps, the method comprising:

Process the original training text into clean training text;

Based on the ontology hierarchical structure of the training labels, the training labels are parsed into layer-by-layer labels of multiple levels;

At least based on the layer-by-layer labels and the clean training text, a plurality of layer-by-layer models are trained by a layer-by-layer multi-label classification model, wherein each layer-by-layer model is associated with a corresponding level of the label;

Using one or more refinement strategies to make layer-by-layer predictions from one or more inputs through the trained plurality of layer-by-layer models; and

The layer-wise predictions are combined into a unified set of labels for the one or more input datasets using a point generative model.

In another aspect of the present disclosure, a system for multi-label learning and classification for large-scale semantic indexing is provided, the system comprising:

A layer-by-layer multi-label classification model, which decomposes the label in a high-dimensional space into layer-by-layer labels in multiple levels based on the ontology hierarchy of labels, and the layer-by-layer multi-label classification model includes Multiple Convolutional Neural Networks (CNNs), where, for each level of CNN, each CNN takes input from word embeddings for documents, word embeddings for keywords, upper-level label embeddings, and lower-level label embeddings, respectively Feature representation is extracted from the CNN including:

a max-pooling layer for dynamic max-pooling to select features from concatenated embeddings concatenated by feature representations extracted from all inputs;

one or more normalization layers and one or more fully connected layers for batch normalization and obtaining a compact representation from the selected features;

an output layer that outputs the layer-by-layer prediction for each of the layers; and

A point generation model incorporating the layer-by-layer predictions at each of the plurality of levels into a unified set of tags for the document, the point generation model comprising:

an encoder, encoding the layer-by-layer prediction into a plurality of encoder hidden state sequences corresponding to the plurality of layers respectively;

a plurality of attention generators derived from the plurality of encoder hidden state sequences to generate attention distributions and content vectors for each of the plurality of levels; and

a decoder to generate a plurality of decoder hidden state sequences based at least on the generated content vector for each of the plurality of levels, the decoder based on at least the layer-wise prediction, the encoder concealment state, the attention generator, and the decoder hidden state to generate the unified label set.

In yet another aspect of the present disclosure, there is provided a computer-implemented method for multi-label learning and classification using one or more processors to perform steps, the method comprising:

At each of the multiple hierarchical levels,

Extract feature representations from the input of word embeddings for documents, word embeddings for keywords, upper hierarchical level label embeddings, and lower hierarchical level label embeddings, respectively, using a convolutional neural network (CNN);

concatenating said feature representations extracted from all inputs into concatenated embeddings;

apply dynamic max-pooling at the max-pooling layer of the CNN to select features from the concatenated embeddings;

apply batch normalization along with one or more fully connected layers to obtain a compact representation from the selected features;

outputting, from an output layer of the CNN, a layer-by-layer prediction for the each of the plurality of hierarchical levels; and

The layer-wise predictions from the plurality of hierarchical levels are combined into a unified label set using a point generative model.

Description of drawings

Reference will be made to the embodiments of the invention, examples of which may be illustrated in the accompanying drawings. The drawings are intended to be illustrative and not restrictive. While the present invention is generally described in the context of these embodiments, it is to be understood that the scope of the present invention is not intended to be limited to these specific embodiments. Items in the drawings are not drawn to scale.

The accompanying drawing ("FIG.") 1 depicts the system architecture of a deep layer-by-layer extreme multi-label learning and classification (deep layer-by-layer XMLC) framework in accordance with an embodiment of the present disclosure.

2 depicts a markup process using deep layer-by-layer XMLC in accordance with an embodiment of the present disclosure.

3 depicts the neural structure of a deep multi-label learning model according to an embodiment of the present disclosure.

4 depicts a process for layer-by-layer prediction using a neural model at each layer, according to an embodiment of the present disclosure.

5 depicts a pointer generation model for final merging according to an embodiment of the present disclosure.

6 depicts the process of generating a model using pointers for final merging, according to an embodiment of the present disclosure.

Figure 7 graphically depicts a hierarchical relationship for tag sets in accordance with an embodiment of the present disclosure.

8 depicts macro precision, recall and F-score obtained by online macro F-metric optimization (OFO) with deep layer-by-layer extreme multi-label learning and classification (XMLC) in accordance with an embodiment of the present disclosure.

9 depicts a simplified block diagram of a computing device/information handling system in accordance with an embodiment of the present disclosure.

Detailed ways

In the following description, for purposes of explanation, specific details are set forth in order to provide an understanding of the present disclosure. However, it will be apparent to those skilled in the art that embodiments may be practiced without these details. Furthermore, those skilled in the art will appreciate that the embodiments of the present disclosure described below can be implemented on tangible computer-readable media in various ways (eg, processes, apparatus, systems, devices, or methods).

The components or modules shown in the figures are exemplary illustrations of embodiments of the invention and are intended to avoid obscuring the present disclosure. It should also be understood that throughout this discussion, components may be described as separate functional units (which may include sub-units), although those skilled in the art will recognize that various components or portions thereof may be divided into separate components, or may be integrated together (including within a single system or component). It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

Furthermore, the connections between components or systems within the figures are not intended to be limited to direct connections. Rather, data between these components may be modified, reformatted, or otherwise changed by intervening components. Additionally, additional or fewer connections may be used. It should also be noted that the terms "coupled," "connected," or "communicatively coupled" should be understood to include direct connections, indirect connections through one or more intermediary devices, and wireless connections.

References in this specification to "one embodiment," "preferred embodiment," "an embodiment," or "a plurality of embodiments" mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in the present invention in at least one embodiment of , and may be included in more than one embodiment. Additionally, the appearances of the above-mentioned phrases in various places in this specification are not necessarily all referring to the same embodiment or embodiments.

Certain terms are used in various places in this specification for the purpose of description and should not be construed as limiting. A service, function or resource is not limited to a single service, function or resource; use of these terms may refer to a distributable or aggregated grouping of related services, functions or resources. Images can be still images or from video.

The terms "comprising", "including", "including" and "comprising" are to be understood as open-ended terms and any listings thereafter are examples and are not intended to be limited to the listed items. Any headings used herein are for organizational purposes only and should not be used to limit the scope of the description or claims. Each reference mentioned in this patent document is hereby incorporated by reference in its entirety.

In addition, those skilled in the art will recognize that: (1) certain steps may be optionally performed; (2) the steps may not be limited to the specific order set forth herein; (3) certain steps may be performed in a different order; and (4) Certain steps can be performed simultaneously.

A. Introduction

With the explosive growth of scientific literature, efficient semantic indexing methods are required to build retrieval systems. Even with efficient techniques, the semantic indexing process still involves manually curating key aspects from the scientific literature. To outline the main topic of the article, domain experts are usually invited to manually index the article using keywords selected from the domain ontology.

In the medical field, MEDLINE is probably the largest database of biomedical literature in the world, and the Medical Subject Thesaurus (MeSH) is the domain ontology used to index articles in MEDLINE. The medical literature search experience is greatly improved by mapping queries to MeSH thesaurus. For example, the query juvenile drug use is mapped to the MeSH thesaurus juvenile and substance-related disorders. Currently, most mapping rules and the final index of medical literature from MEDLINE are manually generated by domain experts. This is expensive and time-consuming for the human tagging process of semantic indexing. Therefore, automatic methods are urgently needed.

However, the task of tidying up presents significant challenges. First, an article is often tagged with multiple keywords or concepts. Furthermore, domain ontology involves hundreds of thousands or even millions of labels. These labels are usually organized in a hierarchy represented in the form of a forest. Handling large numbers of labels, data samples, and complex hierarchies simultaneously is a difficult task.

In the embodiments of this patent document, the task of automatic semantic indexing is considered an extreme multi-label learning and classification (XMLC) problem. Unlike traditional multiclass, XMLC allows millions of tags to coexist per data sample. Recently, several methods have been proposed to deal with XMLC, including FASTXML, LOMTrees, SLEEC, robust bloom filters, label partitioning, fast label embedding, and several deep learning methods, using local neural networks (DXML and XML-CNN) Hierarchical multi-label classification of . Although these methods have made some progress in dealing with XMLC, the curse of dimensionality (known as huge label space) and the high requirement for hand-crafted feature engineering are two major obstacles to further improving effectiveness and efficiency.

To address these two issues, an implementation of a novel framework called Deep Layer-by-Layer Extreme Multi-Label Learning and Classification (Deep Layer-by-Layer XMLC) is disclosed in this patent document to address the problem of large-scale semantic indexing. In one or more embodiments, the deep layer-by-layer XMLC framework includes two sequential modules. In one or more embodiments, the first module is a layer-wise multi-label classification model. It effectively solves the curse of dimensionality by decomposing a large number of labels (in higher dimensional space) into multiple levels (in lower dimensional space). In one or more embodiments, for each level, a convolutional neural network with at least two innovations is constructed. The first innovation includes a class-based dynamic max-pooling method that aims to capture label co-occurrences and taxonomic relationships between labels. Dynamic max pooling methods help to tightly connect layer-by-layer classification models. The second innovation includes a prediction refinement method based on macro F-metric optimization, which enables the module to automatically select labels in an incremental manner. The second module of the deep layer-by-layer XMLC framework is a hierarchical pointer generation model, which incorporates the predicted labels at each level into the final overview semantic index through a copy-and-generate mechanism. Overall, the deep layer-wise XMLC framework avoids the high cost of human intervention by learning semantic indexing without any feature engineering. An embodiment of the overall system architecture is shown in Figure 1, where more details are discussed in Section B.

Some of the contributions of this patent document include:

Deep layer-wise XMLC is proposed to learn large-scale semantic indexing. It divides labels into multiple layers to reduce the curse of dimensionality while improving training efficiency.

A new strategy with class-dependent dynamic max-pooling is introduced to capture co-occurrence and class relationships between labels.

An implementation of a prediction refinement technique derived from macro-F-metric optimization is explored to intelligently select the best labels in an online manner.

A hierarchical pointer generation model is developed to incorporate the layer-by-layer output into the final overview semantic index.

The effectiveness of the deep layer-by-layer XMLC implementation is demonstrated by comparing it with several state-of-the-art methods for auto-tagging of MeSH from MEDLINE and AmazonCat13K (Amazon Cat13K), which is an XMLC dataset with properties similar to MeSH.

B. Method Embodiments

There are two main challenges with XMLC. First, the number of labels in a dataset may exceed 10,000, or even as high as a million. Second, a data sample may be indexed with multiple labels, the number of which typically ranges from one to dozens.

In this patent document, as shown in Figure 1, an implementation of a deep layer-by-layer XMLC framework is disclosed to address these two challenges by decomposing the labels of each data sample into multiple layers. In one or more embodiments, the deep layer-by-layer XMLC framework includes five stages: data preprocessing stage 105 , ontology parsing stage 110 , layer-by-layer model training stage 115 , layer-by-layer prediction stage 120 , and final merging stage 125 . In the data preprocessing stage, tokenization and filtering 138 are performed as usual to obtain clean training text 142 and clean validation text 144 from training text 134 and validation text 136, respectively. In one or more embodiments, training and validation data are randomly selected according to some ratio. Unlike normal natural language processing (NLP) tasks, an additional step of ontology parsing using an ontology parser 140 on the training labels 132 is used in order to divide the labels into multiple levels 146 based on the label-based ontology hierarchy. In the third stage, the layer-by-layer model 148 is trained using the neural model described in the Methods section. In one or more embodiments, the ontology parser 140 operates as a layer-wise multi-label classification model (the first of the two sequential modules described in Section A). Subsequently, in the layer-by-layer prediction stage (or test phase), after preprocessing the test data in a similar manner to obtain layer-by-layer predictions 150, the test data is fed into the trained layer-by-layer model for label prediction or labeling . In the final stage, final merging with the pointer generation model 152 is performed to output one or more predicted labels 154, and some less relevant labels are also filtered out. In one or more embodiments, the pointer generation model 152 is used as a hierarchical pointer generation model (the second of the two sequential modules described in Section A).

2 depicts a tagging process with five steps involved in deep layer-by-layer XMLC, according to an embodiment of the present disclosure. First, the input raw text for training and validation is processed (205) into clean training text and clean validation text, respectively, using an NLP preprocessor. In one or more embodiments, words are tokenized and stop words are filtered out. Second, the training labels are parsed (210) hierarchically based on their ontology into layer-wise labels using an ontology parser. Third, a plurality of layer-by-layer models are trained (215) through a layer-by-layer multi-label classification model, where each layer-by-layer model is associated with a corresponding level of labels, based at least on layer-by-layer labels and clean text. Fourth, layer-wise prediction is performed (220) on one or more input datasets using one or more refinement strategies using the trained multiple layer-by-layer models. In one or more embodiments, one or more of the input datasets may be clean validation text cleaned with the same NLP preprocessor by tokenizing words, stopword filtering, etc. In one or more embodiments, the one or more input datasets may be textual data received on the deployed framework. Finally, the point generative model is trained to merge (225) the layer-wise predictions into a unified label set.

The following subsections focus on: 1) an implementation of a deep layer-wise multi-label learning framework; and 2) an implementation of a pointer generation model for merging labels from all levels into one unified label set.

1. Implementation of deep layer-by-layer multi-label learning

深度逐层XMLC将它们分解成M个层级，并且在训练数据上训练M个神经模型。整个标签集被表示为

并且

是指

中的标签的总数。每个y_i是具有长度

的多热矢量。层级m处的每个模型在每个数据样本上预测最可能的K个标签

K是通过细化策略确定的。最后，训练指针生成模型以将每个数据样本x_i的M个层级的预测

合并到一个统一标签集y_i中。Formally, the problem can be defined as follows: given a set of input pairs

Deep layer-by-layer XMLC decomposes them into M levels and trains M neural models on the training data. The entire label set is represented as

and

Refers to

The total number of labels in . each _yi is with length

multi-heat vector. Each model at level m predicts the most likely K labels on each data sample

K is determined by the refinement strategy. Finally, the pointer generation model is trained to combine the predictions of the M levels of each data sample _xi

merged into a unified label set _yi .

1.1. Implementation of Feature Embedding Construction

In one or more embodiments, the model is constructed in a layer-by-layer fashion. As shown in FIG. 3, a neural model 300 is constructed with four parallel inputs at each level. FIG. 4 depicts a process for layer-by-layer prediction using a neural model 300 at each layer, according to an embodiment of the present disclosure.

The four inputs include word embeddings for documents 310, word embeddings for keywords 320, and hierarchy-related information (including upper-level tag embeddings 330 and lower-level tag embeddings 340). They provide a variety of information for building more discriminative features. In one or more embodiments, a convolutional neural network (CNN) 300 is employed (405) to learn a large number of feature representations 314, 324, 334, and 344 from corresponding inputs 310, 320, 330, and 340, respectively. In one or more embodiments, document embeddings 314 and keyword embeddings 324 are learned directly from the CNN. Two other embeddings, the upper-level embedding 334 and the lower-level label embedding 344, are learned from the embeddings of the prediction results of the upper and lower levels. In one or more embodiments, two steps are involved. First, similar to word embeddings for input text and keywords, in one or more embodiments, Gensim is employed to train label embeddings from annotated MeSH. Second, in training and testing, the predicted labels of certain documents at certain levels can be used as input features for the level above or below it. These two embeddings not only help to capture layer-by-layer dependencies, but also handle the tag imbalance problem in XMLC. In this way, both label co-occurrences and knowledge from their upper and lower levels can help to enhance representation learning for rare labels.

For example, in MeSH, lymphangioma is a rare label and itself is not easily represented. Cystic lymphangioma may be better represented in the embedding space using information from its upper-level MeSH lymphangioma and its lower-level MeSH lymphangioma.

After learning the four embeddings, they are concatenated (410) into concatenated embeddings 352 and passed into the max pooling layer 350.

Due to the order information, the original tokens/words may not be directly concatenated with the embeddings of keywords, upper-level tags and lower-level tags. In one or more embodiments, bidirectional long short-term memory (Bi-LSTM) is constructed for the raw tokens/words on their CNN features to preserve linguistic order information prior to concatenation.

1.2. Implementation of the objective function of the learning framework

In one or more embodiments, after the embeddings are concatenated, a max-pooling layer 350 is employed to apply (415) dynamic max-pooling to select desired features 352 from the concatenated embeddings. A compact representation 362 is obtained (420) from the selected features 352 by applying batch normalization to one or more normalization layers and one or more fully connected layers 360. Afterwards, for training purposes, at least based on the obtained compact representation 362, a (425) binary cross-entropy loss is employed on the output layer and hidden bottleneck layer 370. After training with binary cross-entropy loss, the output layer outputs layer-wise labels 380.

In one or more embodiments, the formula for the loss function L of the binary cross-entropy objective is:

和f_j(x_i)表示输出层函数。此外，f_j(x_i)＝w_og_h(w_h[P(c₁)，...P(c_ι)])。在此，w_h∈R^h×(ιp)和

是与隐藏的瓶颈层和输出层370相关联的权重矩阵，g_h是元素方式激活函数，例如，施加到瓶颈层的S型函数(sigmoid)或双曲正切(tanh)函数，并且ιp是动态最大池化层处ι和p的乘积。ι是指馈送到池化层中的特征的数量，并且p是指池化数量。两者均由x_i中的特征的数量确定。此外，c_i是在从更低层进行池化操作P(.)之后的卷积特征矢量。in,

and f _j ( _xi ) represent the output layer function. Furthermore, f _j ( _xi )=wo _{g h} ( _wh [P(c ₁ ), . . . P(c _ι )]). Here, w _h ^{∈R h×(ιp)} and

is the weight matrix associated with the hidden bottleneck layer and output layer 370, gh is an element-wise activation function, eg, a sigmoid or hyperbolic tangent ( _tanh ) function applied to the bottleneck layer, and ιp is the dynamic The product of i and p at the max pooling layer. ι refers to the number of features fed into the pooling layer, and p refers to the pooling number. Both are determined by the number of features in _xi . Furthermore, _ci is the convolutional feature vector after the pooling operation P(.) from lower layers.

1.3. Implementation of class-oriented dynamic max pooling

其中

是指从索引1开始至

的c的子矢量，p是指最大池化维数。先前的工作使用固定的p。如果p被设置过大，则可能包含冗余特征。如果设置得太小，可能丢失相关特征。In traditional CNN models for text classification, the max-over-time scheme is often adopted because intuitively, the largest element of the feature map should have the most important information, i.e. P(c)=max{c}, where c refers to the output from the CNN. However, this approach exhibits serious drawbacks. Using only one value to represent the entire feature map may lose information when the input document contains multiple topics. For multi-label learning tasks, multiple pooling can capture richer information. In this patent document, pooling is performed dynamically as

in

means starting from index 1 to

A sub-vector of c, p refers to the max pooling dimension. Previous work used a fixed p. If p is set too large, redundant features may be included. If set too small, relevant features may be lost.

In one or more embodiments, layer-wise correlation information, ie, categorical information of labels, is incorporated into the neural architecture (eg, max pooling layers) to help dynamically select p. Specifically, p is adjusted with the distribution of label levels. For example, in MeSH, all items are grouped into 16 categories such as "anatomy", "organism", "disease", etc. Each category involves a different subcategory, and each label involves a different distribution. Based on the distribution, assign different weights to determine p. The larger the weight of the category, the larger the p. In one or more embodiments, class or label weights are initialized from training data.

1.4. Implementation of Maximizing Refinement Predictions Using Macro F-Metrics

As shown in Figure 1, with embedding and dynamic max pooling, the network can perform layer-by-layer prediction. At each level, the top-K (top-K) predicted labels are selected for each data sample. However, a fixed K may yield higher recall, but lower precision. In one or more embodiments of this patent document, predictions are refined through a more flexible weight adjustment strategy.

In one or more embodiments, online F-measure optimization (OFO) is applied to weight adjustment. With OFO, a dynamic balance of precision and recall can be achieved. In one or more embodiments, the OFO algorithm optimizes the binary F-measure by performing threshold adjustments in an online manner.

并且

在此，

是第l个数据样本的第j个标签。

是在迭代t处从标签y_j上的第一个数据样本到第i个数据样本的累积F分数。in

and

here,

is the jth label of the lth data sample.

is the cumulative F-score from the first data sample on label y _j to the ith data sample at iteration t.

根据迭代间规则更新为

在不同的迭代处，它根据交叉迭代规则更新为

其中N是指一批中的数据样本的数量。在一个或多个实施方式中，当新的批次开始时，i被初始化为0，并且还不存在α或β值。在一个或多个实施方式中，最初使用来自最后一批的值。给定第i个数据样本，OFO将预测标签细化为

其中

是指在迭代t处标签y_j上的x_i的预测概率。在一个或多个实施方式中，最佳F度量

是最佳阈值

的值的两倍，为

由于提出的细化机制是动态的、逐层的和增量的，因此最佳阈值

要到训练结束后才能固定。在一个或多个实施方式中，将其保存为测试参数。Due to the incremental property, the threshold of OFO is updated by two rules. In one or more embodiments, at the same iteration (batch of data), the threshold

updated according to the inter-iteration rule to

At different iterations, it is updated according to the cross-iteration rule as

where N refers to the number of data samples in a batch. In one or more embodiments, when a new batch starts, i is initialized to 0, and there is no alpha or beta value yet. In one or more embodiments, the values from the last batch are used initially. Given the ith data sample, OFO refines the predicted labels as

in

is the predicted probability of x _i on label y _j at iteration t. In one or more embodiments, the optimal F-measure

is the optimal threshold

twice the value of , for

Since the proposed refinement mechanism is dynamic, layer-wise and incremental, the optimal threshold

It can't be fixed until after the training is over. In one or more embodiments, it is saved as a test parameter.

2. Implementation of the pointer generation model for final merge

After having layer-wise outputs, these outputs should be merged into a unified label set. However, they cannot be simply grouped together, as simple concatenation can result in a much larger number of labels compared to gold standard labels or ground truth labels. In this patent document, a filtering method is disclosed to remove some layer-wise labels to ensure that the final distribution of predicted labels is consistent with the gold standard labels. In one or more embodiments, each layer-wise prediction is treated as a sentence, inspired by text summarization, and the gold standard is regarded as the summarization output. Hierarchical relationships of labels between levels during decoding, encoding and attention states are considered.

2.1 Implementation of the Hierarchical Pointer Generation Model

In one or more embodiments, the hierarchical pointer generation model allows both copying labels from layer-by-layer predictions and generating labels from the entire label set.

5 depicts a pointer generation model for final merging according to an embodiment of the present disclosure, and FIG. 6 depicts a process for using a pointer generation model for final merging according to an embodiment of the present disclosure. The pointer generation model includes an encoder 520 , an attention generator 530 and a decoder 540 . The encoder receives ( 605 ) an input 510 of layer-wise predicted labels organized as a layer-by-layer sequence from level 1 to level M. The input is encoded (610) as M sequences of hidden states 522. In one or more embodiments, the encoder is a bidirectional LSTM encoder. The hidden state of each encoding reflects the internal relationship of the predicted labels at some level. In one or more embodiments, the encoder hidden state is denoted as e ^τ = v ^T _tanh (wh γ ^τ + w _s s ^τ + b _attn ). Here, ^sτ and ^γτ are the predicted label sequence vector and content vector around the predicted label, respectively. The terms v, _{wh, ws, and battn are weight} parameters. In one or more embodiments, the content vector is about co-occurrence of markers.

其中γ_q代表第q个标签。In one or more embodiments, a plurality of attention generators 530 are derived from the encoder hidden state to generate (615) an attention distribution a ^τ and a content vector γ ^{τ at a time step τ} . In one or more embodiments, a ^τ is calculated as a ^τ =softmax(e ^τ ). The attention distribution is the probability distribution over the predicted layer-wise labels. It is used to generate ^γτ as a hierarchical weighted sum of encoder hidden states:

where γq represents the _qth label.

In one or more embodiments, each attention generator is referred to as a coverage vector 532, which shows how much attention is given to the labels of each hierarchy. As we all know, outlining can lead to duplication. Therefore, the same label can also be generated multiple times. The well-designed overlay vector plays the role of judging whether the label is a duplicate or not. If it is not a duplicate, a label with higher attention is more likely to be decoded into a correct label. If duplicated, the duplicate avoidance mechanism (described in 2.3 in Section B) filters the tags out. Attention is drawn based on the coverage vector described in the method. Subsequently, the decoder is working to generate a reduced size output.

In one or more embodiments, to generate the decoder hidden state for the decoder, it is obtained (620) from the content vector ^γτ , the predicted label sequence vector ^sτ , and the decoder input ^yτ (gold standard or ground truth) The generation probability p _gen ∈ [0, 1] for time step τ is:

p _gen =σ(w _h γ ^τ +w _s s ^τ +w _y y ^τ +b _ptr ) (3)

(在本文参见如何在2.2中计算

)中采样来从整个标签集中生成标签或者通过从注意力分布a^τ中采样来从输入序列中复制标签之间进行选择。where w _h , _ws , w _y and b _ptr are weight parameters. Here, _pgen is used as a soft switch to distribute

(See how to calculate in 2.2 in this article

) to generate labels from the entire label set or to copy labels from the input sequence by sampling from the attention distribution a ^τ .

计算条件概率

以将概率链规则项的标签估计为：Using the above input layer-by-layer predicted labels, encoder hidden state, attention generator, and decoder hidden state, a hierarchical pointer generation model can be trained to generate (625) an output 550 of the final summarized semantic index label. When generating the output, the probability of generating the final label is learned. Given a training pair

Calculate Conditional Probabilities

To estimate the label of the probability chain rule term as:

是

矢量的序列。通过将训练集的条件概率最大化为

来学习模型的参数，其中总和超过训练示例。in

Yes

Sequence of vectors. By maximizing the conditional probability of the training set as

to learn the parameters of the model where the sum exceeds the training examples.

2.2 Implementation of sequence-to-sequence probability calculation

为：In one or more embodiments, the above process ultimately yields a label vocabulary distribution

for:

获得。在一个或多个实施方式中，损失函数是目标标签

的负对数似然。以下示例说明了在给定其他标签的情况下一个标签的概率计算过程。where v, v', b and b' are learnable parameters. For a specific tag, it can be accessed from

get. In one or more embodiments, the loss function is the target label

The negative log-likelihood of . The following example illustrates the process of calculating the probability of one label given other labels.

7 depicts a hierarchical relationship of a tag set {Nutrition and Metabolic Diseases} in accordance with one or more embodiments of the present patent disclosure. The words on the left in the figure are acronyms for labels. For example, the acronym for Wolfram Syndrome is ws. The calculation for this example is described in Section B, Section 2.2. On the left side of Figure 7, to save space and to better illustrate the process, the Sigmoid symbols are drawn here in those short forms (they are the initial letters of the MeSH term).

In one or more embodiments, given content={nmd, md, dm, dmt1}, following the hierarchical relationship between these tags, p(e _ws |context) is calculated as:

2.3 Implementation of mechanisms to avoid duplication

The problem with pointer generative models or sequence-to-sequence models is that it may copy items from the input multiple times. Duplicates are not required, as each label should be unique. Duplication can be avoided by employing an overlay mechanism. That is, if labels have been seen in the output of one level, the probability of generating them at other levels will be lower. In one or more embodiments of the present patent disclosure, this approach is taken by incorporating an overlay mechanism into the overall pointer generation model. specifically,

是指覆盖矢量，y^m是指第m层级。in

refers to the coverage vector, and y ^m refers to the mth level.

是在逐层输入上的非标准化分布，逐层输入表示迄今为止已从注意力机制接收到这些标签的覆盖程度。由于标签是按层级排序的，因此在层级的不同部分不应当重复，这种机制旨在删除在不同部分中找到的所有重复标签，并且还避免在同一层级内重复。在一个或多个实施方式中，作为对任何重复的惩罚，将

添加到注意力机制并且还将

添加到指针生成模型的总损失函数。In one or more embodiments of this patent disclosure, an overlay vector consists of a set of vectors for all levels. For each cover vector,

is the unnormalized distribution over the layer-wise input that represents how well these labels have been received so far from the attention mechanism. Since labels are ordered hierarchically, there should be no duplicates in different parts of the hierarchy, this mechanism is designed to remove all duplicate labels found in different parts, and also avoid duplication within the same level. In one or more embodiments, as a penalty for any duplication, the

added to the attention mechanism and will also

The total loss function added to the pointer generative model.

C. Some experiments

It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using one or more specific embodiments; therefore, neither these experiments nor their results should be used to limit the scope of the disclosure of this patent document.

In this section, the effectiveness of an implementation of deep layer-wise XMLC is evaluated using the MEDLINE dataset from the US National Library of Medicine labeled MeSH and Amazon-Cat13K. As described in Section A, MEDLINE is the world's largest database of biomedical literature, and the Medical Subject Thesaurus (MeSH) is a domain ontology used to label articles in MEDLINE. Another dataset, AmazonCat13K, is one of the benchmark datasets used to develop extreme classification algorithms. Similar to MeSH, it involves 13330 labels, all of which are organized hierarchically. The dataset size, expert labels, and hierarchical nature provide an ideal testbed for the presented framework.

1. Data Setup and Preprocessing

The total number of MeSH tags in MEDLINE is 26,000, of which 60% appear more than 1000 times. In one or more experimental settings, those MeSH labels that appeared less than 10 times in the experiment were removed. MEDLINE has 26 million articles with abstracts. 90% of these articles have about 20 MeSH tags. 82% of articles were assigned between 4 and 16 MeSH tags. In MeSH, there are 3.5 million abstracts with both MeSH tags and keywords. The ontology of MeSH tags can be decomposed into 7 levels, where the lowest level (level 7) includes the most specific MeSH tags, and the highest level (level 1) has the most general and abstract MeSH tags. For articles with only the lowest level of MeSH tags, it is expanded by the following method. Start with the label at the lowest level and find all labels at the level above it. In one or more experimental settings, 7 datasets are constructed for the proposed deep layer-by-layer XMLC framework.

Meanwhile, 102,167 abstracts with MeSH tags from all 7 tiers were stored for testing. The statistics of the datasets for each tier are shown in Table 1. It can be seen that the middle level has the largest number of labels, while the highest level has only 83 labels, and the lowest level has 2445 labels. Similar trends can be found in the volume of data. Two million articles have tags from tiers 2, 3, and 4, while fewer than one million articles have tags from tiers 1, 6, and 7.

For AmazonCat13K, since deep layer-by-layer XMLC requires text data, its preprocessing dataset cannot be used directly. At the same time, the data should be divided based on its layer-by-layer categories. It is found that all labels can be broken down into 9 levels. The difference is that if a document from AmazonCat13K has a lower label, it must have a higher label, which is not necessarily the case for a document from MeSH. Therefore, it is easy to find public collections for testing AmazonCat13K (using only documents with lower categories). In order to maintain reasonable pooling of test data, documents with tiers higher than 6 are ignored (only 9990, 385 and 31 documents for tiers 7, 8 and 9, respectively).

Table 1. Statistics of the dataset. For each level, there is a different amount of data. Papers in Medline do not necessarily imply that they can be labeled as higher-level MeSH terms, even if they are labeled with lower-level MeSH terms.

In the experiments, for MEDLINE articles and keywords, at each level, a single neural network is first trained according to the first component of the depth-wise layer-wise XMLC. Use the trained model to make predictions on the test data at each level. Subsequently, the predicted layer-wise labels from the training data, as well as the gold standard labels, are used by the pointer generation model for final merging. Likewise, a layer-wise model is trained for AmazonCat14K, but the latter does not have keywords.

2. Evaluation Metrics

In extreme multi-label classification datasets, the number of relevant labels per document is limited despite the often huge label space. This means that it is important to provide a short sorted list of relevant tags for each test document. The focus of the evaluation is therefore on the quality of such sorted lists, with an emphasis on the relevance of the top of each list. However, in one or more experimental settings, two evaluation metrics were used for the purpose of comparison with two dataset sources. The medical community prefers to use precision, recall, and F-score, while those in the general domain prefer to use precision at K (P@K) and normalized discounted cumulative gain (NDCG@K or G@K for short).

其中前K个项在层级m处，精度、查全率和F分数定义如下：Specifically, given a list of predicted labels

where the top K terms are at level m, and the precision, recall and F-score are defined as follows:

是排名前K个标签中的正确标签的数量；AK_i是文章i的黄金标准标签的总数；微度量与宏度量之间的差异在于预测概率的计算。对于微度量，直到将所有正确的预测加在一起后才进行概率计算，而对于宏度量，将对每篇文章进行概率计算，并且最后将平均值用作宏分数。报告这两种度量是为了查看模型对于单篇文章和整个数据集的准确性如何。where N is the number of data samples, and

is the number of correct labels among the top K labels; AK _i is the total number of gold-standard labels for article i; the difference between micrometrics and macrometrics is the computation of predicted probabilities. For micrometrics, probability calculations are not performed until all correct predictions are added together, while for macrometrics, probability calculations are performed for each article, and the average is used as the macroscore at the end. Both metrics are reported to see how accurate the model is for individual articles and the entire dataset.

Instead, P@K and NDCG@K are defined as,

表示为文档的真实标签的矢量，并且

为同一文档的系统预测分数矢量。在一个或多个实验设置中，遵循P@K和NDCG@K的约定使用k＝1，3，5。in,

is a vector represented as the true labels of the document, and

A vector of predicted scores for the system for the same document. In one or more experimental settings, k=1, 3, 5 were used following the conventions of P@K and NDCG@K.

3. Parameter setting

For deep layer-wise XMLC neural networks, rectified linear units are used. The filter window is set to 3, 4, 5. The dropout rate is set to 0.5, and the L2 constraint is set to 3. The minimum batch size is set to 256. Embedding size varies for different features. For Mesh, word embeddings for medical words involve 500,000 unique tokens, keyword embeddings involve over 100,000 phrases, and label embeddings involve 26,000 MeSH terms. Gensim is used to train embeddings of dimension 300. For AmazonCat13K, a pretrained GoogleNews-vectors-negative300.bin with 3 million tokens and 300 dimensions is utilized. Values for other hyperparameters were chosen via a lattice search on a smaller validation set from the training data.

4. Performance of Online F-Measure Optimization

As discussed in Section B.1.4, online macro-F-metric optimization (OFO) is integrated into the proposed framework. To show the effectiveness of OFO, macro precision, recall and F-score are reported for the first 6 tiers in Figure 8 for MeSH. Although Tier 7 results and AmazonCat13K results are not shown, they can achieve similar performance. It can be observed that OFO helps to achieve a balance between macro precision and recall. It is further observed that the optimal F-score is different at different tiers. If the top K (k=10 in the experiment) are always selected for layer-by-layer prediction, the best F-score cannot be obtained although the recall rate of each layer may be as high as about 80%. Accuracy can be as low as or less than 20%. The reason is that after dividing the MeSH tags of each article into 7 tiers, most articles only have 2 to 5 tags at each tier. This means that even though all labels are in the top 10, even though the recall may be 100%, the accuracy is only 20% to 50%. In this case, the F-score is also not high. OFO greatly removes less relevant labels, so the number of labels in the final prediction set at each level also varies from 2 to 5. At the same time, most of the correct predictions remain in the prediction set. Clearly, this tuning strategy greatly improves performance.

5. Layer-by-layer performance

As discussed in Section B, the proposed deep layer-by-layer XMLC framework decomposes the task of XMLC into layer-by-layer model construction. Therefore, in this section, the layer-by-layer prediction results are reported in order to understand the intermediate development and improvement of the overall model.

As shown in Figure 3, the layer-by-layer neural model learns label embeddings from MEDLINE sets, keywords, and predicted labels from upper or lower layers. Here we report the performance of layer-wise neural models with different embeddings. The effectiveness of OFO is further demonstrated by fixing the top K labels at each level and comparing with the layer-by-layer neural model. Different values of K from 1 to 10 were tested and the best performance was found to be achieved when K was 5.

Table 2 reports the microscopic performance for the layer-by-layer model with OFO and top-K fixation strategies. For best results, set K to 5 here. The performance of the macrometrics is also shown in Table 3. It can be seen that OFO always outperforms the strategy of fixing the top K, no matter on the micro-metric or the macro-metric.

Tables 2 and 3 also report layer-wise predictions for MeSH with three different embeddings. Although the evaluation on the AmazonCat13K dataset is not based on F-scores, a micrometric for AmazonCat13K is also reported to show the advantage of OFO. After all, the results of P@K and NDCG@K are computed on the filtered output using OFO. From this result, a clear incremental trend for all seven levels can be identified. That is, by adding upper and lower levels of keywords and predicted MeSH terms, the prediction will be quickly improved accordingly. It is not difficult to see that, in general, macro results are better than micro results. Among them, Tier 3 and Tier 4 of MeSH and Tier 4 and Tier 5 of AmazonCat13K yield worse results than the other datasets, while Tier 1 achieves much better results for both datasets. This is understandable considering that the third and fourth tags are more numerous (4,484 and 6,568, respectively, for MeSH and 6,181 and 5,372, respectively, for AmazonCat13K).

Table 2: Layer-wise performance for micrometrics using top-K and OFO. This table is intended to show the incremental improvement of micrometrics as new features are incrementally added for each level. Also, for each tier, the results for the top K without optimization and with OFO optimization are also shown here.

Table 3. Layer-wise performance for macrometrics using top-K and OFO. This table is intended to show the incremental improvement of micrometrics as new features are incrementally added for each level. Meanwhile, for each tier, in a similar manner to Table 2, the results for the top K without optimization and with OFO optimization are also shown here.

6. Performance of the final merge

The proposed deep layer-by-layer XMLC incorporates layer-by-layer predictions into a unified label set with a pointer-generating model. In this section, against the MeSH results, the deep layer-by-layer XMLC is further compared with five state-of-the-art methods to demonstrate the effectiveness of the pointer generation model, including: MTIDEF (Minlie Huang et al., recommended for biomedical use) MeSH Items for Article Annotations, Journal of the American Medical Information Association, Volume 18, Issue 5 (2011), pp. 660-667); MeSH Now (Yuqing Mao and Zhiyong Lu., 2017), MeSHNow: Ranking by Learning Automated MeSH Indexing at the PubMed Scale, Journal of Biomedical Semantics, Vol. 8, Issue 1 (2017), 15); MeSHLabeler, MeSHRanker (Ke Liu et al., MeSHLabeler: Improving Large-Scale MeSH by Integrating Various Evidences) Indexing Accuracy, Bioinformatics Vol. 31, No. 12 (2015) pp. i339-i347); and Deep Mesh (Shengwen Peng et al., DeepMeSH: Deep Semantic Representations for Improving Large-Scale MeSH Indexing, Bio Information, Vol. 32, Issue 12 (2016, pp. i70-i79). All of these existing systems make heavy use of feature engineering. In contrast, deep layer-by-layer XMLC uses limited external resources. For AmazonCat13K, XML-CNN results report the state of the art system on this benchmark dataset.

Starting from MeSH labeling, after obtaining layer-wise results, a hierarchical pointer generation model is trained using predictions from all layers as input and gold standard labels as output. For model training, the input can be organized with each label as a separate unit or as a label at the same level as a unit (called sentences in the overview community). Therefore, two pointer generative models are trained, where the former is called deep-level-wise XMLC _label and the latter is called deep-level-wise XMLC _level . For comparison, the results from all levels are summed, and less relevant labels are then filtered out according to their predicted probabilities and the label distribution in the gold standard (deep layer-wise XMLC _sampling ).

Table 4. Performance of deep layer-wise XMLC for the MeSH dataset. As you can see from the bold numbers, the best performance comes from deep layer-by-layer XMLC. Obviously, hierarchical-based dynamic pooling achieves better performance than label-based dynamic pooling.

As shown in Table 4, both depth-wise XMLC _label and depth-wise XMLC _level outperform other systems on the macro metrics of precision, recall and F-score. Micrometrics not reported in Table 4 have similar trends.

By involving embeddings from MEDLINE collections and keywords, deep layer-by-layer XMLC _label and deep layer-by-layer XMLC _level achieve much better performance compared to all other existing cutting-edge frameworks. It can be observed that although the F-scores are very similar, different input organizations may lead to different performances in precision and recall. The depth-wise XMLC _label achieves better precision, while the depth-wise XMLC _level achieves better recall. This seems to suggest that the proposed hierarchical pointer generation model takes into account the correlation between labels within a cell. Therefore, the depth-wise XMLC _level with longer input units achieves better recall. However, it also includes more false positives, reducing its accuracy. In contrast, the deep layer-by-layer XMLC _{lebel wins} in accuracy, probably because it considers more smaller units and therefore misses more correct positives.

Meanwhile, the results obtained by deep layer-by-layer XMLC _sampling are much worse than most existing systems. This suggests that the hierarchical pointer generation model can play an important role in ultimately achieving the best performance. In addition, deep layer-by-layer XMLC _level results using max pooling are also reported. By default, all systems work with dynamic max pooling. Clearly, the results show that dynamic max-pooling has advantages over conventional max-pooling strategies.

Table 5. Performance of deep layer-by-layer XMLC for AmazonCat13K. Use the generation network with pointers and the version based on dynamic pooling and hierarchy. As described in Section C, 3.1, to extend the proposed methodology from the medical domain to some more general domains, the proposed model implementation is also tested on AmazonCat13K. For those using AmazonCat13K, they prefer to report accuracy @K and NDCG@K. The performance of AmazonCat13K's XML-CNN is also listed for comparison.

XML-CNN: Jingzhou Liu et al., Deep Learning for Extreme Multi-Label Text Classification. In Proceedings of the 40th International ACM SIGIR Conference on Information Retrieval Research and Development (SIGIR), Tokyo, Japan, pp. 115-124.

For AmazonCat13K, the results are given in Table 5. The latest technological achievements of XML-CNN are also listed. Table 5 shows higher performance results from the work in this patent disclosure. It should be noted that the test dataset for deep layer-wise XMLC is extracted from the raw text data with labels at each level, while the work of XML-CNN is tested on the standard test dataset prepared by the data collector. .

D. Some related work

1. Tree-based approach

Due to the huge number of tags, the prediction of XMLC may involve high cost in time and space. Tree-based methods aim to reduce training and testing costs. For example, label partitioning by sublinear sorting (LPSR) methods attempts to reduce prediction time by learning a hierarchy on the base classifier. Some have proposed a method called Multi-Label Random Forest (MLRF), which aims to learn an entire random tree rather than relying on learning a base classifier. It is proposed to use FastXML to learn hierarchies not in the label space but in the feature space. It defines the label set active in a region as the union of the labels of all training points present in that region. At each node of the hierarchy, an NDCG-based objective is optimized. That is, at each node, a hyperplane is induced, and the hyperplane divides the set of documents in the current node into two subsets. Predictions are made by returning a sorted list of the most frequently occurring labels across all leaf nodes. Recently, some people have developed multi-label classification for social flow based on ensemble random forests. They integrated a base learner and a label-based learner to learn hierarchical labels. However, these methods are expensive to train due to the dimensionality of the label space and feature space.

2. Embedding method

Embedding methods attempt to overcome the intractability problem caused by a large number of labels by projecting the label vector onto a low-dimensional space and thereby reducing the number of labels. Assume that the label matrix is low-rank. Embedding methods have proven to be the most popular methods for solving XMLC problems due to their strong theoretical foundation and ability to deal with tag dependencies. Specifically, a recently proposed embedding method, Sparse Local Embedding for Extreme Multi-Label Classification (SLEEC), greatly improves accuracy after incorporating nonlinear neighborhood constraints into a low-dimensional embedding space for training, and a simple k nearest Neighborhood (k-NN) clustering was used in the embedding space for testing. In one or more embodiments of the present patent disclosure, further steps are taken to improve prediction of neural structures by exploring layer-by-layer label embeddings.

3. Maximum margin method

The maximum margin method is also used to handle multi-label classification. Someone proposed a model called PD-Sparse. Basically, a linear classifier is learned for each label with L1 and L2 norm penalties on the weight matrix associated with that label. This leads to sparse solutions in primal and dual spaces. Use the fully corrected block coordinate Frank-Wolfe training algorithm to achieve sub-linear training time with respect to the number of original variables and bivariates, while obtaining better performance than 1 for logistic regression for all SVMs and multi-label classification, greatly reducing training time and model size. However, like 1-versus-all SVMs, the PD-Sparse method is algorithmically not scalable to extreme multi-label learning.

4. Methods based on deep learning

Deep learning based methods have also been used for multi-label learning. Some people incorporate label space embeddings into feature embeddings. Specifically, an adjacency matrix is constructed for label A, and the label graph matrix is derived using the equation M=(A+A ² )/2. Subsequently, for each non-zero entry in the matrix, a tuple consisting of indices p, q, and M _pq is fed into the label embedding network to train the composite network along with the word embeddings. In the prediction stage, a k-NN search is performed in the low-dimensional feature representation to find similar samples from the training dataset. Set the average of the labels of the k-NN as the final label prediction. It has been proposed to incorporate multi-label co-occurrence patterns into neural network objectives to improve classification performance. They also propose dynamic max pooling to capture rich information from different regions of the document, and use an additional hidden bottleneck layer to reduce model size. Furthermore, the binary cross-entropy loss on the Sigmoid output is modulated as XMLC. However, these methods are not suitable for data with complex hierarchical labels, because the label-level decomposition would greatly reduce the label space. Furthermore, a hybrid learning network based on Boltzmann CNN has been proposed to handle biomedical literature classification. Their work enriches data sequence embeddings. This design is not suitable for huge tab space. Their experiments only focused on species with fewer than 2,000 MeSH tags. A Hierarchical Multi-Label Classification Network (HMCN) has been proposed, which is claimed to be able to optimize both local and global loss functions in order to discover local hierarchical classification relationships and global information from the entire classification hierarchy, while penalizing hierarchical violations. But their work has higher computational complexity due to the use of fully feed-forward layers. Even if the HMCN network is simplified with an LSTM-like model with shared weights, it still has a high computational burden. It seems that this is why datasets of up to ~4000 labels are reported for HMCN.

E. Some conclusions

Disclosed herein is an implementation of a deep learning-based layer-by-layer framework for handling extreme multi-label learning and classification, generally (for convenience and not limitation) named deep layer-by-layer XMLC. The implementation of deep layer-by-layer XMLC includes several innovations. First, in one or more embodiments, the split model training mechanism divides the labels into multiple hierarchies, thereby greatly reducing the curse of dimensionality and training costs. Second, in one or more embodiments, class-dependent dynamic max-pooling and weight adjustment using macro-F-metrics is integrated into the neural architecture, so that the final predictions are more suitable for the distribution of hierarchies and their hierarchical relationships. Third, in one or more embodiments, the layer-by-layer pointer generation model successfully incorporates layer-by-layer outputs into one unified label prediction.

The results show that the implementation of deep layer-by-layer XMLC achieves state-of-the-art results by leveraging MEDLINE sets, keywords and predicted labels from upper and lower layers. The results of AmazonCat13K also show that the implementation of deep layer-by-layer XMLC is general enough to handle various datasets.

In this patent disclosure, it is not difficult to find that deep layer-by-layer XMLC implementations can be easily transferred to tasks such as large-scale semantic indexing in order to build more efficient and accurate information retrieval engines and reduce expensive manual expert work, such as shown in.

Those skilled in the art will recognize that other implementations may include different, more robust loss functions, as well as adding more layers to handle feature refinement or weight adjustment, and at the same time improve operational efficiency.

F. System Implementation

In embodiments, aspects of this patent document may relate to, may include, or may be implemented on one or more information handling systems/computing systems. A computing system may include a computing system operable to compute, compute, determine, classify, process, transmit, receive, retrieve, originate, route, exchange, store, display, communicate, manifest, detect, record, reproduce, process or utilize any form of information, Any means or combination of means of intelligence or data. For example, a computing system can be or can include a personal computer (eg, a laptop computer), a tablet computer, a phablet, a personal digital assistant (PDA), a smart phone, a smart watch, a smart package, a server (eg, a blade server or rack servers), network storage devices, cameras, or any other suitable equipment, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources (eg, a central processing unit (CPU) or hardware or software control logic), ROM, and/or other types of memory. Additional components of the computing system may include one or more disk drives, one or more network ports for communicating with external devices, and various input and output (I/O) devices (eg, keyboard, mouse, touch screen, and/or video display). The computing system may also include one or more buses operable to transmit communications between the various hardware components.

9 depicts a simplified block diagram of a computing device/information handling system (or computing system) in accordance with an embodiment of the present disclosure. It should be understood that computing systems may be configured differently and include different components, including fewer or more components as shown in FIG. 9, but it should be understood that the functions shown for system 900 are operable to support various components of the computing system. an implementation.

As shown in FIG. 9, a computing system 900 includes one or more central processing units (CPUs) 901, which provide computing resources and control a computer. CPU 901 may be implemented with a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 919 and/or floating point coprocessors for mathematical calculations. System 900 may also include system memory 902, which may be in the form of random access memory (RAM), read only memory (ROM), or both.

As shown in Figure 9, multiple controllers and peripherals may also be provided. Input controller 903 represents an interface to various input devices 904, such as a keyboard, mouse, touch screen, and/or stylus. Computing system 900 may also include a storage controller 907 for interfacing with one or more storage devices 908, each of which includes a storage medium (such as tape or disk) or an optical medium (which may be used for Programs that record instructions for operating systems, utilities, and applications, which may include implementations of programs that implement aspects of the invention). Storage device 908 may also be used to store processed data or data to be processed in accordance with the present invention. The system 900 may also include a display controller 909 for providing an interface to a display device 911, which may be a cathode ray tube (CRT), thin film transistor (TFT) display, organic light emitting diode, electroluminescence panel, plasma panel or other type of display. Computing system 900 may also include one or more peripheral controllers or interfaces 905 for one or more peripheral devices 906 . Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, and the like. The communications controller 914 may interface with one or more communications devices 915, which enables the system 900 to communicate over various networks (including the Internet, cloud resources (eg, Ethernet cloud, Fibre Channel over Ethernet (FCoE)/Data Center Bridging (DCB) ) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN)), or by any suitable electromagnetic carrier signal (including infrared signal) to connect to the remote device.

In the illustrated system, all major system components may be connected to bus 916, which may represent more than one physical bus. However, the various system components may or may not be in physical proximity to each other. For example, input data and/or output data may be transmitted remotely from one physical location to another. Additionally, programs implementing aspects of the present invention may be accessed from a remote location (eg, a server) via a network. Such data and/or programs may be transmitted by any of a variety of machine-readable media including, but not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; such as CD-ROMs and Optical media for holographic devices; magneto-optical media; and hardware devices specifically configured to store or store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices , and ROM and RAM devices.

Aspects of the invention may utilize instructions for one or more processors or processing units to be encoded on one or more non-transitory computer readable media to cause the steps to be performed. It should be noted that the one or more non-transitory computer readable media shall include both volatile memory and nonvolatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software/hardware implementations. Hardware-implemented functions may be implemented using ASICs, programmable arrays, digital signal processing circuits, and the like. Accordingly, the term "means" in any claim is intended to cover both software and hardware implementations. Similarly, the term "computer-readable medium or medium" as used herein includes software and/or hardware or a combination thereof having a program of instructions embodied thereon. With these alternative implementations contemplated, it should be understood that the drawings and accompanying description provide those skilled in the art with what would be required to write program code (ie, software) and/or to manufacture circuits (ie, hardware) to perform the desired processes function information.

It should be noted that embodiments of the present invention may also relate to computer products having non-transitory tangible computer-readable media having computer code thereon for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present invention, or they may be known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware specially configured to store or store and execute program code Devices, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. Examples of computer code include machine code (eg, code produced by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. Embodiments of the invention may be implemented in whole or in part as machine-executable instructions in a program module executable by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in local, remote, or both settings.

Those skilled in the art will recognize that neither the computing system nor the programming language is critical to the practice of the present invention. Those skilled in the art will also appreciate that a number of the above-described elements may be physically and/or functionally divided into sub-modules or grouped together.

Those skilled in the art will appreciate that the foregoing examples and implementations are illustrative and do not limit the scope of the present disclosure. It is intended to illustrate that all, substitutions, enhancements, equivalents, combinations or improvements of the present invention that will be apparent to those skilled in the art after reading this specification and studying the accompanying drawings are included in the true spirit of the present disclosure and within the range. It should also be noted that elements of any claim may be arranged differently, including with multiple dependencies, configurations and combinations.

Claims (20)

1. A computer-implemented method for multi-label learning and classification using one or more processors to perform steps, the method comprising:

processing an original training text into a clean training text;

analyzing the training labels into layer-by-layer labels of a plurality of levels based on the body hierarchical structure of the training labels;

training a plurality of layer-by-layer models by a layer-by-layer multi-label classification model based at least on the layer-by-layer labels and the clean training text, wherein each layer-by-layer model is associated with a corresponding level of labels;

employing one or more refinement strategies to perform layer-by-layer predictions from one or more inputs through the trained plurality of layer-by-layer models; and

merging the layer-by-layer predictions into a unified labelset for the one or more input data sets using a point generation model.

2. The computer-implemented method of claim 1, wherein the one or more inputs include word embedding for documents, word embedding for keywords, upper level embedding, and lower level embedding.

3. The computer-implemented method of claim 2, wherein performing layer-by-layer prediction comprises:

receiving inputs for word embedding for documents, word embedding for keywords, upper level label embedding, and lower level label embedding at a Convolutional Neural Network (CNN) within each layer-by-layer model to extract feature representations from each input;

obtaining a series embedding using the feature representation extracted from each input;

performing dynamic max-pooling at a max-pooling layer to select a desired feature from the series embedding;

obtaining a compact representation from the desired features by applying bulk normalization and one or more fully connected layers; and

each layer-by-layer model is trained with binary cross-entropy losses on the output layer and the hidden bottleneck layer based at least on the obtained compact representation.

4. The computer-implemented method of claim 3 wherein a Bi-directional long short term memory (Bi-LSTM) is constructed on the feature representations extracted from the word embedding to preserve the linguistic order of documents prior to concatenation.

5. The computer-implemented method of claim 3, wherein in performing dynamic maximal pooling, layer-by-layer correlation information of tags is incorporated into neural structures of at least the maximal pooling layer to capture tag co-occurrences and classification relationships between tags in order to dynamically select a maximal pooling size.

6. The computer-implemented method of claim 1, wherein the one or more refinement strategies include macro F-metric optimization to enable each layer-by-layer model to incrementally automatically refine layer-by-layer predictions by threshold adjustment.

7. The computer-implemented method of claim 1, wherein merging the layer-by-layer predictions into a unified labelset using the point generation model comprises:

encoding, using an encoder within the point generative model, the layer-by-layer prediction into a plurality of encoder hidden state sequences corresponding respectively to the plurality of levels;

deriving a plurality of attention generators from the plurality of encoder concealment state sequences to generate an attention distribution and a content vector for each level of the plurality of levels;

obtaining generation probabilities from the content vector, predicted tag sequence vector and decoder input to generate a plurality of decoder hidden state sequences; and

generating a final output summarizing semantic index labels based at least on the decoder hidden state.

8. The computer-implemented method of claim 7, wherein an overlay mechanism is combined with the point generation model to remove duplicate items per level and across levels.

9. A system for multi-label learning and classification of large-scale semantic indexes, the system comprising:

a layer-by-layer multi-tag classification model that decomposes tags in a high-dimensional space into layer-by-layer tags in a plurality of levels based on an ontology hierarchy of the tags, the layer-by-layer multi-tag classification model comprising a plurality of convolutional neural networks, wherein for each level of convolutional neural networks, each convolutional neural network extracts a feature representation from inputs for word embedding of a document, word embedding for a keyword, upper-level tag embedding, and lower-level tag embedding, respectively, the convolutional neural networks comprising:

a max pooling layer for dynamic max pooling to select features from a series of embeddings that are concatenated by feature representations extracted from all inputs;

one or more normalization layers and one or more fully connected layers for batch normalization and obtaining compact representations from selected features;

an output layer outputting a layer-by-layer prediction for said each level; and

a point generative model incorporating the layer-by-layer predictions at each of the plurality of levels into a unified labelset of the document, the point generative model comprising:

an encoder that encodes the layer-by-layer prediction into a plurality of encoder hidden state sequences respectively corresponding to the plurality of levels;

a plurality of attention generators derived from the plurality of encoder concealment state sequences to generate an attention distribution and a content vector for each of the plurality of levels; and

a decoder to generate a plurality of sequences of decoder hidden states based at least on the generated content vectors for each of the plurality of levels, the decoder to generate the unified labelset based at least on the layer-by-layer prediction, the encoder hidden state, the attention force generator, and the decoder hidden state.

10. The system of claim 9, wherein a Bi-directional long short term memory (Bi-LSTM) is constructed on the feature representations extracted from the word embedding to preserve linguistic order of documents prior to concatenation.

11. The system of claim 9, wherein in performing dynamic maximal pooling, layer-by-layer relevant information of tags is incorporated into neural structures of the maximal pooling layer to dynamically select a maximal pooling size.

12. The system of claim 9, wherein the layer-by-layer multi-label classification model uses online F-metric optimization such that each convolutional neural network can incrementally and automatically refine layer-by-layer predictions by adjusting thresholds of the online F-metric optimization.

13. The system of claim 12, wherein the threshold is updated according to an inter-iteration rule within the same iteration and a cross-iteration rule between iterations.

14. The system of claim 9, wherein the point generation model incorporates an overlay mechanism to eliminate duplicate laboratories in and across each level.

15. The system of claim 9, wherein each convolutional neural network further comprises a hidden bottleneck layer with activation function, the convolutional neural network being pre-trained by employing binary cross-entropy loss on the output layer and the hidden bottleneck layer.

16. The system of claim 15, wherein the binary cross-entropy loss is a function involving a weight matrix associated with the bottleneck layer and the output layer.

17. A computer-implemented method for multi-label learning and classification using one or more processors to perform steps, the method comprising:

at each of the plurality of hierarchical levels,

extracting feature representations from the input for word embedding for documents, word embedding for keywords, upper hierarchical level label embedding, and lower hierarchical level label embedding, respectively, using a convolutional neural network;

concatenating the feature representations extracted from all inputs as a concatenation embedding;

applying dynamic max pooling at a max pooling layer of the convolutional neural network to select features from a series embedding;

applying bulk normalization and one or more fully connected layers to obtain a compact representation from the selected features;

outputting, from an output layer of the convolutional neural network, a layer-by-layer prediction for the each level of the plurality of hierarchical levels; and

merging the layer-by-layer predictions from the plurality of hierarchical levels into a unified labelset using a point generation model.

18. The computer-implemented method of claim 17, wherein merging the layer-by-layer predictions into a unified labelset comprises:

encoding, using an encoder, the layer-by-layer prediction into a plurality of encoder hidden state sequences respectively corresponding to the plurality of levels;

deriving a plurality of attention generators from the plurality of encoder concealment state sequences to generate an attention distribution and a content vector for each level of the plurality of levels; and

generating, using a decoder, a plurality of decoder hidden state sequences based at least on the generated content vectors for each of the plurality of hierarchical levels; and

generating the unified labelset from the decoder based at least on the layer-by-layer prediction, the encoder concealment state, the attention generator, and the decoder concealment state.

19. The method of claim 18, wherein in performing dynamic maximum pooling, both tag co-occurrences and classification relationships between tags are captured to dynamically select a maximum pooling size.

20. The method of claim 18, wherein the layer-by-layer multi-label classification model uses online F-metric optimization such that each convolutional neural network can incrementally and automatically refine layer-by-layer predictions by adjusting thresholds of the online F-metric optimization, the thresholds being updated according to inter-iteration rules within the same iteration and cross-iteration rules between iterations.

Images