CN115186072A - A Knowledge Graph Visual Question Answering Method Based on Dual-process Cognitive Theory - Google Patents

Abstract

The invention discloses a knowledge graph visual question-answering method based on a double-process cognitive theory, which comprises the following steps of: (1) Problem-picture joint characterization, namely, for input problems and pictures, respectively extracting the characteristics of the problems and the pictures by using a pre-training model BERT and a Faster-RCNN, and then sending the characteristics into a double-flow Transformer model to learn the problem-picture joint characterization; (2) Constructing a fact graph and a semantic graph, namely constructing the fact graph and the semantic graph respectively aiming at each question-picture pair, retrieving the fact graph from a knowledge base to construct the fact graph based on a semantic matching mode, and constructing the semantic graph in an image description mode; (3) Evidence aggregation, namely selecting evidences from the two graphs respectively by using a graph reasoning network, and then aggregating the evidences from a semantic graph to a fact graph based on a cross-modal graph reasoning network; (4) And (4) carrying out answer reasoning, and carrying out binary classification on nodes in the fact graph to obtain answers. The method has wide application prospect in the fields of education, entertainment and the like.

Description

A Knowledge Graph Visual Question Answering Method Based on Dual-process Cognitive Theory

technical field

The invention designs a knowledge map visual question answering method based on the dual-process cognitive theory, which belongs to the intersection of natural language processing and computer vision fields.

Background technique

It is a hot research topic of the combination of computer vision and natural language processing technology in recent years to make the agent have the ability to understand the world by analyzing the visual and language information. Related research has promoted the development of many applications, such as visual question answering (VQA), image indexing, and image description. Among them, VQA is a challenging task that requires the model to answer arbitrary questions based on a given image. In order to promote the development of VQA technology, relevant scholars have done a lot of preliminary research work in recent years, and have made great progress. However, existing VQA methods focus on answering questions based on image content, and cannot answer some questions that require a combination of common sense.

To advance the field, Wang et al. proposed the Fact-based Visual Question Answering (FVQA) task (P.Wang, Q.Wu, C.Shen, A.Dick and A.van denHengel," FVQA: Fact-Based Visual Question Answering,"in IEEE Transactions onPattern Analysis and Machine Intelligence,vol.40,no.10,pp.2413-2427,1Oct.2018,doi:10.1109/TPAMI.2017.2754246). At the same time, they also released a new dataset that provides additional supporting facts for each question-answer pair and asks the model to answer the question through a joint analysis of images and external knowledge. The study by Wang et al. first parses sentences and maps them into a knowledge graph, and then utilizes keyword matching to find the correct answer. This approach has obvious flaws and becomes ineffective when no obvious visual concept is mentioned in the question or when there are synonyms and homographs. Therefore, some scholars subsequently proposed a retrieval method based on semantic learning, projecting image-question-visual concepts and candidate facts into a learned embedding space, and finding supporting facts by calculating the corresponding distance. However, this method evaluates one fact node at a time, so it is inefficient, and furthermore, the method cannot utilize the structural information of external knowledge bases. To address this issue, Narasimhan et al. propose a graph inference-based method to select answers by reasoning over the entire graph (Narasimhan M, Schwing AG. Straight to the facts: Learning knowledge baseretrieval for factual visual question answering [C ] // Proceedings of the European conference on computer vision (ECCV. 2018:451-468.). This method constructs an entity graph, each node in the graph is represented by the connection represented by the entity, image and question, and then uses the graph convolutional network to aggregate the message to obtain the corresponding node update feature, and finally based on the updated node feature. Prediction of the answer. Since the problem only focuses on part of the visual content, this approach inevitably introduces noisy information.

For humans, when a picture and a question are given, inferring the answer can be divided into two steps: (1) The brain quickly obtains the content presented in the image and the visual information that the question is concerned about by analyzing the image and question; (2) According to the output of the first step, combined with the knowledge stored in the brain, conduct in-depth analysis to find the correct answer. This process is known in cognitive science as two-process theory. The dual-process cognitive theory states that the human brain first perceives external input through an implicit, unconscious process performed by a system called System 1. This information is then fed into a system called System2 to perform an explicit, conscious, controlled reasoning process. The reasoning process is sequential inference in working memory, which is slower but is a property of humans as advanced agents. From this perspective, the FVQA problem can be solved using the dual-process cognitive theory - System 1 quickly retrieves information from images and questions, and System 2 combines external knowledge to perform deep reasoning to find the correct answer.

Inspired by the dual-process cognitive theory, the present invention proposes a new framework for solving the FVQA problem based on a dual-process cognitive system. Specifically, System 1 in the framework of the present invention is implemented by a multimodal Transformer network that utilizes a cross-attention mechanism to capture the complex relationship between questions and images. System 1 outputs a joint representation of the image and the question. For System 2, the present invention uses a Graph Neural Network (GNN) to reason on two external knowledge graphs (fact graph and semantic graph) to find the correct answer. System 2 first performs intra-modal evidence selection and aggregation, and aggregates question-related evidence information from two knowledge graphs, respectively; then performs cross-modal selection, and aggregates evidence from fact graphs into semantic graphs to assist Better inferences about the answer. In the reasoning process, the present invention proposes a dual attention mechanism of node-level attention and path-level attention, which captures valuable information from key nodes and paths, makes the reasoning process more reasonable, and further improves the performance of visual question answering based on knowledge graphs .

SUMMARY OF THE INVENTION

Inspired by the dual-process cognitive theory, the present invention proposes a new framework for solving the FVQA problem with a dual-process cognitive system. Specifically, System 1 in the framework of the present invention is implemented by a multimodal Transformer network, which utilizes a cross-attention mechanism to capture the complex relationship between questions and images and outputs a joint question-image representation. For System 2, a GCN network is used to reason over two external knowledge graphs (fact graph and semantic graph) to find the correct answer.

The present invention realizes above-mentioned purpose through following technical scheme:

1. The knowledge graph visual question answering framework based on the dual-process cognitive theory described in the invention is shown in Figure 1, which includes two parts: a coordinated perception module (system 1) and an explicit reasoning module (system 2). The framework has a specific inference process. Include the following steps:

(1) Use the text pre-training model BERT and the image pre-training model Faster-RCNN to extract features from the input text and images, respectively. For each question, add [CLS] and [SEP] flags at the beginning and end of the question, respectively , and then sent to the BERT model to extract features; for each image, 36 target areas are extracted, each target area contains a target's visual features and the target's spatiotemporal location feature information.

(2) The image and text features extracted in step (1) are fed into a dual-stream Transformer network to learn the joint representation of images and texts, where a single-stream Transformer network is used to learn image-guided problem representations, while the other The single-stream Transformer is used to learn the image representation guided by the problem. Finally, the outputs of the two dual-stream Transformer networks are average pooled and further multiplied to obtain a joint representation.

(3) Construction of fact graph and semantic graph. For each question-image pair, a fact graph and a semantic graph are constructed respectively, where the fact graph is selected from an external knowledge base by means of sentence-level semantic matching, while the semantic graph is It is obtained by first describing the image semantically, and then performing semantic analysis on the generated sentence.

(4) Evidence aggregation based on graph reasoning, first for the fact graph and semantic graph, based on the attention mechanism, two graph reasoning networks are used to aggregate evidence information from the knowledge graph and fact graph respectively, and then the cross-modal reasoning network is used to aggregate evidence information from the semantic graph. Aggregate question-related evidence information into a knowledge graph.

(5) Answer prediction: Do the dot product calculation between the joint representation of the question-picture and the feature vector of each node in the fact graph to obtain the semantic relevance score of each node and the question, and finally send the relevance score to a Sigmoid layer predicts the corresponding answer.

目标的空间位置特征

和对应的标签。为了同时捕获视觉特征和空间特征，本发明首先将外观特征和空间位置特征投影到同一维度中(本发明为768维)，然后对外观视觉特征和空间位置特征计算平均得到每个目标对象的特征表示。Specifically, in step (1), first use the BERT pre-training model to initialize the word of the input question, where BERT uses the bert-base-uncased version, and after the vector initialization, the feature vector of the word C=[c ₀ ,c ₁ ,...,c _n ], the feature vector dimension of each word is 768 dimensions. Then use the Faster-RCNN pre-training model to extract 36 target regions of the input picture, each region contains the appearance feature of a target

Spatial location features of the target

and corresponding labels. In order to capture visual features and spatial features at the same time, the present invention first projects appearance features and spatial position features into the same dimension (768 dimensions in the present invention), and then calculates and averages the appearance visual features and spatial position features to obtain the features of each target object express.

In step (2), a cross-modal Transformer is used to align the relationship between these two modalities to learn the question representation of image participation and the image representation of question participation, as shown in Figure 1.

Through step (1), the problem feature vector C and the image feature vector V are obtained, and the problem feature vector C and the image feature vector V are sent into a dual-stream Transformer network to learn the complex interaction between the problem and the image. A single-stream Transformer network uses In order to learn the problem representation under the guidance of pictures, another single-stream Transformer network is used to learn the picture representation under the guidance of the picture; in the learning of the question representation under the guidance of pictures, the question feature vector is used as the query vector, and the picture feature vector is used as the key and The formula for calculating the dependency between the value vector, the picture and the question is as follows:

W′ _* in formula (1) represents the parameter matrix to be learned by the model; in the image representation learning guided by the problem, the image feature vector is used as the query vector, and the problem feature vector is used as the key and value vectors. The formula for calculating the relationship is as follows:

W″ _* in formula (1) represents the parameter matrix to be learned by the model; after the joint representation of multi-layer (in the present invention, specifically 9 layers) image and text, the [CLS] bit of the text sequence is Features serve as joint representation features for the entire question-image.

The construction of the fact map and the semantic map in step (3), the first step, for each question-picture pair, the input question and the labels of the detected objects in the picture are sequentially spliced to obtain the question-picture instance set; Each triple in the library is converted (the conversion method is to splicing the head entity, the relation and the tail entity in turn) into a natural language processing sentence, and the corresponding fact instance set is obtained. In the second step, the question-picture instance set and the fact instance set are encoded with the pre-trained sentence encoder Universal-sentence-encoder to obtain the corresponding instance representation, and then the question-picture instance set of each instance object in the set is encoded. The feature representation calculates the cosine similarity with the feature representation of the fact instance object in turn to obtain the corresponding correlation score. Finally, after sorting all the instance objects according to the cosine similarity score, the top 10 instances with the highest scores are obtained as alternative supporting facts. The third step is to construct a fact graph based on the candidate supporting facts retrieved. The nodes in the graph are entities in the knowledge base, and the edges are the relationships between two entities. The corresponding initialization representation, where the initialization representation of nodes and edges is the average of the word embeddings of all words in the nodes and edges.

The evidence aggregation based on graph reasoning in step (4) includes evidence aggregation within a modality and evidence aggregation between modalities:

计算过程如下：During the aggregation of evidence within the modality, the attention mechanism proposed by the present invention containing two levels is firstly used: node-level attention (Node-level) and path-level attention (Path-level) for feature selection and aggregation; In the process of level attention calculation, first calculate the attention score of each node in the graph and the joint question-picture representation

The calculation process is as follows:

Then the attention score is multiplied by the initial feature vector of each node in the graph to obtain the node feature vector based on the image-question guidance; in the process of path node attention calculation, which path is more important to the inference process, where Each path is defined as a path composed of all nodes and edges directly connected to the target node, which is defined as follows:

φ _{ij =} (vi , r _ij , v _j ) (4)

Among them, _vi , ri , and v _{j are} divided into tables to represent the representation of the head node, the representation of the relationship and the feature representation of the tail node in the fact graph; after obtaining the path representation, the path-level attention calculation process is as follows:

Then, according to the message propagation network, the features are aggregated from the neighbor nodes, and the calculation formula of the feature aggregation process of the neighbor nodes is as follows:

Finally, the feature of the neighbor node is fused with the feature of the target node, and then the feature of the target node is further updated. In order to prevent the feature of the neighbor node from over-updating the initial feature of the node, a gating mechanism is designed to control the feature of the neighbor node and the original feature of the target node. The proportion of features, the feature update process of the target node is calculated as follows:

The process of evidence selection in the semantic graph is the same as the steps in the fact graph, and will not be repeated here;

Inter-modal evidence aggregation, when performing inter-modal evidence aggregation, first, under the guidance of the question, calculate the attention weight coefficient of each node in the fact graph and each node in the semantic graph, and finally calculate the attention weight coefficient according to the attention weight coefficient. Each node in the semantic graph is weighted and summed to obtain the relevant features in the semantic graph. The relevant process calculation process is as follows:

Finally, the feature vector in the semantic graph is fused with the feature vector of the original node in the fact graph to obtain the updated feature after cross-modal fusion. The gating mechanism is used to control the proportion of the two different modal features. The specific process calculation formula is as follows:

Finally, the updated features are used for answer inference.

Description of drawings

Fig. 1 is the main frame of the network model proposed by the present invention.

Figure 2 is the structure of the cross-modal Transformer network.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings:

Figure 1 is the structure of the entire network, which consists of two parts, the coordinated perception module (system 1) and the explicit reasoning module (system 2). System 1 in the framework is implemented by a multimodal Transformer network that utilizes a cross-attention mechanism to capture the complex relationship between questions and images. System 1 outputs a joint representation of the image and question. For System 2, the present invention uses a Graph Neural Network (GNN) to reason on two external knowledge graphs (fact graph and semantic graph) to find the correct answer.

Figure 2 is a cross-modal Transformer framework, which feeds the extracted image and text features into a dual-stream Transformer network to learn the joint representation of images and text, and a single-stream Transformer network is used to learn image-guided problems. representation, and another single-stream Transformer model is used to learn the image representation guided by the problem. Finally, the outputs of the two dual-stream Transformer networks are subjected to average pooling (Average pooling) and further multiplied to obtain a joint representation.

Tables ₁ and 2 are the experimental results of the present invention on the public datasets FVQA and OK-VQA. The experiments show that the proposed model has the best comprehensive evaluation index F1 value compared with the existing best model. result.

Table 1 The experimental comparison results of the network model of the present invention on the FVQA data set and other existing models

Table 2 The experimental comparison results of the network model of the present invention on the OK-VQA data set and other existing models

The above-described embodiments are only preferred embodiments of the present invention, and are not intended to limit the technical solutions of the present invention. As long as the technical solutions that can be realized on the basis of the above-described embodiments without creative work, all should be regarded as falling into the patent of the present invention. within the scope of protection of rights.

Claims (4)

1. A knowledge graph visual question-answer based on a double-process cognitive theory is characterized by comprising the following steps:

(1) Respectively using a text pre-training model BERT and a target detection model Faster-RCNN to extract the characteristics of an input text and an image, respectively adding [ CLS ] and [ SEP ] marks at the beginning and the end of each sentence aiming at the text, and then sending the sentence into the BERT model to extract the characteristics; for each picture, extracting 36 target regions, wherein each target region comprises the appearance visual characteristics of an object and the spatial position characteristic information of the object in the picture;

(2) Sending the extracted picture and text characteristics into a double-flow Transformer network to learn joint representation of the picture and the text, wherein one single-flow Transformer network is used for learning problem representation under picture guidance, the other single-flow Transformer model is used for learning picture representation under problem guidance, and finally multiplying the output of the two double-flow Transformer networks after average pooling to obtain problem-image joint representation;

(3) The method comprises the steps of constructing a fact graph and a semantic graph respectively aiming at each question-picture pair, wherein the fact graph is constructed by retrieving alternative supporting facts from an external knowledge base based on a sentence-level semantic matching mode, the semantic graph is constructed by performing semantic description on pictures firstly and then performing semantic analysis on generated sentences;

(4) The evidence aggregation based on graph reasoning comprises the steps of firstly, aggregating evidence information from a fact graph and a semantic graph by using two graph reasoning networks with attention mechanisms according to the fact graph and the semantic graph, and then aggregating the evidence information related to problems from the semantic graph into a knowledge graph by using a cross-modal reasoning network;

(5) And (3) answer prediction, namely performing point multiplication calculation on the joint representation of the question and the picture and the feature vector of each node in the fact graph to obtain a semantic matching degree score of each node and the question, and finally sending the matching degree score into a Sigmoid layer to predict a corresponding answer.

2. The method according to claim 1, wherein the problem-picture joint characterization learning method in (2) comprises the following steps:

giving a problem feature vector C and a picture feature vector V, and sending the problem feature vector C and the picture feature vector V into a double-flow Transformer network to learn the complex interaction of the problem and the picture, wherein one single-flow Transformer network is used for learning problem representation under picture guidance, and the other single-flow Transformer network is used for learning picture representation under problem guidance; in the problem representation learning under the guidance of pictures, a problem feature vector is used as a query vector, a picture feature vector is used as a key vector and a value vector, and the calculation formula of the dependency relationship between pictures and problems is as follows:

w 'in the formula (1)' _* Representing a parameter matrix to be learned by the model; in the picture representation learning under the guidance of the question, the picture feature vector is used as a query vector, the question feature vector is used as a key vector and a value vector, and the calculation formula of the dependency relationship between the two vectors is as follows:

w in formula (2) _* The parameter matrix to be learned of the model is also represented; after a multi-layered (in the present invention, specifically 9 layers) joint characterization of the image and the text, [ CLS ] of the text sequence]The bit features act as joint representation features for the entire problem-picture.

3. The method according to claim 1, wherein the fact graph constructing method in step (3) comprises the following steps:

(1) For each problem-picture pair, firstly, sequentially splicing input problems and tags of objects detected in pictures to obtain a problem-picture example set, then converting each triple in an external knowledge base into a natural language sentence to obtain a corresponding fact example set, wherein the conversion method is obtained by sequentially splicing a head entity, a relation and a tail entity;

(2) Coding the problem-picture instance set and the fact instance set by using a pre-trained sentence coder Universal-content-encoder to obtain corresponding instance representation;

(3) Finally, calculating cosine similarity between the feature representation of each instance object in the problem-picture instance set and the feature representation of the fact instance object in sequence to obtain corresponding association scores, and finally sequencing all instance objects according to the cosine similarity scores to obtain the top 10 instances with the highest scores as alternative supporting facts;

(4) And constructing a fact graph according to the retrieved alternative support facts, wherein nodes in the graph are entities in a knowledge base, edges are relations between the two entities, and after the fact graph is constructed, performing corresponding initialized representation on the nodes and the edges by using a BERT model, wherein the initialized representation of the nodes and the edges is the average of word embedding of all words in the nodes and the edges.

4. The method according to claim 1, wherein the evidence aggregation step in (4) is divided into intra-modality evidence aggregation and inter-modality evidence aggregation:

(1) In the case of intra-modal evidence of aggregation, the attention proposed by the present invention is first utilized, including a double level: the Node-level attention (Node-level) and Path-level attention (Path-level) networks are used for feature selection and aggregation; in the node level attention calculation process, the attention score of each node and the problem-picture joint representation in the graph is calculated firstly

The calculation process is as follows:

w and b in the above formula represent the parameter matrix to be learned by the model and the offset, and W and b in the following formula represent the same meaning and will not be described repeatedly; obtaining an attention weight coefficient, and multiplying the attention score by the initial feature vector of each node in the graph to obtain a node feature representation guided based on the picture-question; in the path node attention calculation process, which path is more important to the inference process is mainly concerned, wherein each path is defined as a path formed by nodes and edges which are directly connected with a target node, and the path is defined as follows:

φ _ij ＝(v _i ,r _ij ,v _j ) (4)

wherein v is _i ，r _ij ，v _j The sublist represents the feature representation of the head node, the feature representation of the relationship and the feature representation of the tail node in the fact graph, and after the path representation is obtained, the attention calculation process of the path level is as follows:

and then, aggregating the features from the neighbor nodes by using a message propagation mechanism, wherein the feature aggregation process of the neighbor nodes is shown as the following formula:

finally, fusing the characteristics of the neighbor nodes and the characteristics of the target node and then further updating the characteristics of the target node, in order to prevent the characteristics of the neighbor nodes from excessively updating the characteristics of the nodes, designing a gating mechanism to control the ratio of the characteristics of the neighbor nodes to the original characteristics of the target node, wherein the characteristic updating process of the whole target node is shown as the following formula:

the process of evidence aggregation in the semantic graph is the same as the steps in the fact graph, and the description is not repeated here;

(2) And (2) aggregating the inter-modal evidences, wherein when the inter-modal evidences are aggregated, attention weight coefficients of each node in the fact graph and each node in the semantic graph are calculated under the guidance of a problem, and finally, the feature of each node in the semantic graph is weighted and summed according to the attention weight coefficients to obtain the related features in the semantic graph, wherein the related process calculation process comprises the following steps:

finally, fusing the feature vectors aggregated from the semantic graph with the feature vectors of the nodes in the fact graph to obtain new features subjected to cross-mode fusion, and designing a corresponding gating mechanism to control the proportion of two different mode features in order to prevent the features from the semantic graph from excessively updating the features of the nodes in the fact graph, wherein the specific process calculation formula is as follows:

and finally, using the updated features for the inference of the answer.

Images