CN116739056A - Visual dialogue generation method and device based on counterfactual common sense causal reasoning - Google Patents

Abstract

The invention discloses a visual dialogue generating method and device based on the causal reasoning of the counterfacts common sense, wherein the method comprises the following steps: introducing common sense into a visual dialogue task, constructing a visual dialogue causal graph based on common sense fusion, and calculating the total effect of input features on answer prediction; constructing a corresponding anti-facts causal graph based on the visual dialogue causal graph; for the anti-facts causal graph, estimating the influence of harmful deviation generated by the introduced common sense according to the natural direct effect on answer prediction, and removing the influence from the total effect; training a visual dialogue model by adopting model integration, and obtaining an answer prediction result by taking cross entropy and KL divergence loss as training targets; integrating the image features, the dialogue history features, the common sense features and the current problem features, sending the integrated image features, the dialogue history features, the common sense features and the current problem features into a decoder, minimizing a loss function, optimizing network parameters, and finally providing real disaster environment information for disaster rescue sites. The device comprises: a processor and a memory.

Description

Visual dialogue generation method and device based on counterfactual common sense causal reasoning

Technical field

The present invention relates to the field of visual dialogue generation, and in particular to a visual dialogue generation method and device based on counterfactual common sense causal reasoning.

Background technique

With the rapid development of computer vision technology and natural language processing technology, the multimodal field of visual and language interaction has received widespread attention. From image description ^[1] , scene graph generation ^[2] , visual question answering ^[3] , to visual dialogue ^[4] , researchers are committed to improving the ability of computers to continuously interact with humans. Among them, the visual dialogue task has always been the focus of research in the multi-modal field. It requires the agent to continuously infer the answer to the current question based on the existing image information and the text information contained in historical question and answer. The relationship between humans and agents The dialogue will last for multiple rounds. Therefore, visual dialogue tasks require agents to have strong human-computer interaction capabilities. Based on this, visual dialogue has great application value in fields such as helping the visually impaired and disaster rescue missions.

In recent years, many excellent works have emerged in the field of visual dialogue. For example: methods based on recurrent neural networks ^[4] use recurrent neural networks and their variants to encode visual-linguistic multi-modal features to obtain answers, and methods based on attention mechanisms ^[5][6] mainly use attention mechanisms. To more precisely extract the image information needed to answer current questions, contextual information in dialogue history, etc., graph structure-based methods ^[7][8] mainly use graph structures to encode images, dialogue history or current questions, giving intelligence stronger reasoning ability to generate answers. They all infer answers based on existing image information and text information contained in historical questions and answers. However, in some more complex dialogue scenarios, it is not enough to just use this information. The agent also needs to use external common sense knowledge like humans to assist in answer generation. This is often ignored by researchers, thus limiting Improvement of intelligent human-computer interaction capabilities. For example: at a fire rescue scene, the agent can enter the fire location to take photos and answer questions from rescuers in real time. When the rescuers ask the question "Is there a gas tank at the fire scene?", if the intelligent agent does not have relevant common sense such as "The gas tank is in the kitchen", it will not pay attention to the kitchen area, and thus cannot answer the rescuers' questions correctly.

At present, only the knowledge-based structured network method (SKANet) ^[9] and the multi-structure-based common sense knowledge reasoning method (RMK) ^[10] have introduced external common sense knowledge into visual dialogue tasks, and have made a series of progress. They all extract common sense knowledge from external common sense databases, and then integrate multi-modal information after encoding to obtain answers. But the above frameworks are all based on the underlying assumption that this common sense knowledge will always have a positive impact on answer generation. Although they filter out common sense that is not relevant to the current conversation by calculating the semantic similarity between image descriptions and common sense knowledge, constructing a common sense knowledge graph through the graph embedding algorithm (TransE algorithm), and then calculating the cosine distance between nodes, etc., however, in common sense Some of the "harmful bias" contained in the implication has still not been removed and will have a negative impact on answer generation. For example, image tags or keywords in image descriptions used to retrieve common sense knowledge appear more frequently in common sense knowledge, which may interfere with the agent's ability to generate answers or even cause the agent to generate incorrect answers containing these high-frequency words. For example, if the agent has common sense related to "gas tank", when answering the rescuer's question "Is there a gas tank at the fire scene?", the agent may pay too much attention to the wrong answer containing the high-frequency word "gas tank", thus causing the problem. Failure to provide correct information to rescuers increases the difficulty of rescue.

Based on this research status, the challenges currently faced mainly include the following three aspects: (1) How to more effectively select and use common sense knowledge related to images and current conversations to assist in answer generation; (2) How to quantify the "common sense" contained in common sense The negative impact of "harmful bias" on answer generation; (3) How to remove the negative impact of common sense on answer generation as a whole and retain only the positive impact of common sense on answer generation, thereby improving the answer prediction accuracy of the agent at the disaster rescue site, To assist rescue personnel to better understand the disaster scene environment, formulate rescue plans, and carry out rescue work, etc.

Contents of the invention

The present invention provides a visual dialogue generation method and device based on counterfactual common sense causal reasoning. The invention constructs a visual dialogue factual causal graph based on common sense fusion, and subtracts it from the answer prediction score generated by the visual dialogue based on the causal graph deduction. Common sense has a natural direct effect on answer generation. This process retains the positive impact of common sense on answer generation while removing the negative impact of the "harmful bias" contained in common sense on answer generation, thereby improving the performance of the agent at the disaster rescue site. Computer interaction capabilities provide rescuers with more realistic and detailed disaster environment information, as described below:

A visual dialogue generation method based on counterfactual common sense causal reasoning, which method includes:

For the extracted common sense triplet, a common sense subgraph is constructed, and the attention graph convolution network is used to encode the subgraph to obtain common sense features;

Introduce common sense into the visual dialogue task, construct a visual dialogue causal graph based on common sense fusion, and calculate the total effect of input features on answer prediction; based on the visual dialogue causal graph, construct its corresponding counterfactual causal graph;

For counterfactual causal diagrams, the impact of harmful biases introduced by common sense on answer predictions is estimated based on natural direct effects and removed from the total effect;

Model ensemble is used to train the visual dialogue model, and cross-entropy and KL divergence loss are used as training targets to obtain answer prediction results;

Integrate image features, conversation history features, common sense features and current problem features and send them to the decoder to minimize the loss function, optimize network parameters, and finally provide real disaster environment information for disaster relief sites.

Among them, the common sense triplet is: extract the common sense related to each visual dialogue unit in the training set, verification set and test set samples of the database; the specific operation of the extracted common sense triplet is:

Use the Faster R-CNN framework to detect object labels, and select the top several object labels with the highest scores based on the confidence scores corresponding to each label; similarly, select the top few keywords with the highest scores for image descriptions;

Among them, based on the visual dialogue causal diagram, the corresponding counterfactual causal diagram constructed is:

By assigning null values to I, Q, H, and C respectively, that is, I=i*, Q=q*, H=h*, and C=c*, and then K=k*, blocking I, Q, H, and The impact of K on the prediction of answer A;

Common sense C can be assigned two values at the same time in the counterfactual world, namely C=c* and C=c; the former is used to obtain K=k*, and the latter is directly connected to the answer A to evaluate the answer prediction of common sense C A natural direct effect.

The second aspect is a visual dialogue generation device based on counterfactual common sense causal reasoning. The device includes: a processor and a memory. Program instructions are stored in the memory. The processor calls the program instructions stored in the memory to cause The apparatus performs the method steps of any one of the first aspects.

In a third aspect, a computer-readable storage medium stores a computer program. The computer program includes program instructions. When executed by a processor, the program instructions cause the processor to execute the first aspect. The method steps described in any one of.

The beneficial effects of the technical solution provided by the present invention are:

1. The present invention introduces external common sense knowledge into existing visual dialogue tasks. The agent can not only reason based on existing image information and conversation history context information, but also use information from common sense knowledge to generate answers; existing Visual dialogue methods often ignore the important role of common sense knowledge in the process of visual dialogue generation; this invention focuses on the sources of information required by the agent when reasoning about answers. The introduction of common sense knowledge enables the agent's human-computer interaction capabilities to continuously improve and effectively improve Improved answer generation accuracy;

2. While introducing common sense knowledge into visual dialogue tasks, the present invention considers the negative impact of "harmful bias" contained in common sense on answer generation, constructs a counterfactual causal diagram based on common sense fusion, and quantifies the negative impact into common sense It has a natural direct effect on answer generation; while the existing visual dialogue generation method based on common sense fusion only introduces common sense into the visual dialogue task, making the agent susceptible to the interference of "harmful bias" when generating answers; the present invention focuses on the use of common sense knowledge in Regarding the impact on answer generation in the visual dialogue task, we use the natural direct effect in the causal theory to quantify the negative impact of the "harmful bias" contained in common sense on answer generation, and then remove it to further improve the accuracy of answer generation;

3. The present invention constructs a visual dialogue fact-cause and effect diagram based on common sense fusion. Existing visual dialogue frameworks based on common sense fusion are often deduced based on this diagram. Based on this diagram, the present invention obtains the total effect of common sense on answer generation. From the total effect The natural direct effect of common sense on answer generation is removed. At the same time, the present invention also minimizes the KL-divergence loss function to make the answer probability distribution of the natural direct effect and the total effect similar, thereby better removing the "harmful bias" contained in common sense. Existing methods often ignore the negative impact on answer generation and retain the positive impact of common sense on answer generation; the present invention focuses on removing the negative impact of "harmful bias" contained in common sense on answer generation, and improves the human intelligence of the agent. Computer interaction capabilities;

4. The solution of the present invention uses the counterfactual analysis method in the causal theory to remove the negative impact of "harmful bias" contained in common sense on answer generation. It has a certain degree of generalization and can be applied to most visual dialogue models;

5. Apply the generated results of the present invention to disaster rescue tasks, which improves the answer prediction accuracy of the intelligent agent when answering rescuers' questions in real time, allowing rescuers to fully understand the environmental information of the disaster site, thereby better formulating rescue plans. The plan reduces the difficulty of rescue and avoids rescue casualties.

Description of drawings

Figure 1 is a flow chart of a visual dialogue generation method based on counterfactual common sense causal reasoning;

Figure 2 is a schematic diagram of causal theory and counterfactual analysis;

Figure 3 is a visual dialogue fact-effect diagram based on common sense fusion;

Figure 4 shows the counterfactual causal diagram of visual dialogue based on common sense fusion.

Figure 5 is a model diagram of a visual dialogue generation method based on counterfactual common sense causal reasoning.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

Example 1

In order to solve the problem that current visual dialogue ignores the harmful bias caused by common sense when integrating common sense knowledge, thereby improving the accuracy of visual dialogue answer prediction, commonly used indicators are recall rate and normalized loss cumulative gain, see Figure 1 , embodiments of the present invention provide a visual dialogue generation method based on counterfactual common sense causal reasoning. The method includes the following steps:

101: Construct a common sense subgraph for the extracted common sense triplet, and use the attention graph convolution network to encode the subgraph to obtain common sense features;

102: Introduce common sense into the visual dialogue task, construct a visual dialogue causal graph based on common sense fusion, and calculate the total effect of input features on answer prediction; based on the visual dialogue causal graph, construct its corresponding counterfactual causal graph;

103: For counterfactual causal diagrams, estimate the impact of harmful biases introduced by common sense on answer predictions based on natural direct effects and remove them from the total effect;

104: Use model integration to train the visual dialogue model, and use cross-entropy and KL divergence loss as training targets to obtain answer prediction results;

105: Integrate image features, conversation history features, common sense features and current problem features, and send them to the decoder to minimize the loss function, optimize network parameters, and finally provide real disaster environment information for disaster relief sites.

To sum up, the embodiment of the present invention removes the negative impact of "harmful deviations" contained in common sense on answer generation through the above steps 101-105, thereby improving the human-computer interaction ability of the agent at the disaster rescue site and providing rescuers with Provide more realistic and detailed disaster environment information.

Example 2

The solution in Embodiment 1 will be further introduced below with specific calculation formulas and examples. For details, see the following description:

201: For the database used, extract the common sense related to each visual dialogue unit in its training set, verification set and test set samples;

For the dataset used, it consists of visual dialogue units. Each visual dialogue unit includes: image and corresponding picture description, historical question and answer pairs and current questions that need to be answered, and each question corresponds to 100 candidate answers. The more commonly used VisDial v0.9 and VisDial v1.0 databases are used here, but the embodiments of the present invention are not limited to this database, and can be any database containing required visual dialogue units.

For the required common sense sources, the embodiment of the present invention uses the more commonly used ConceptNet common sense knowledge base. The common sense knowledge base is a common sense knowledge base based on the most basic things a person knows, and is composed of basic common sense triples. The specific form of a triple is <start node, relationship label, end node>, where both the start node and the end node refer to natural language words or phrases, such as <bus, capable of, transport people>, and each three-tuple The tuple will be given a credibility score. The larger the score, the more reliable the relationship between the start node and the end node of the triple.

First, in order to retrieve common sense related to each dialogue unit from the common sense knowledge base, search terms need to be generated first. In this embodiment of the present invention, the search terms are derived from labels and image descriptions of objects in the image. Specifically, for the labels of objects in the image, the Faster R-CNN (fast regional convolutional neural network) framework is used to detect the object labels, and the top five object labels with the highest scores are selected based on the confidence scores corresponding to each label; For image descriptions and keywords, similarly, based on the confidence score corresponding to each keyword, the top 5 keywords with the highest scores are selected. From this, the five extracted object labels and five keywords together constitute the search terms used to retrieve common sense.

Secondly, use the search terms to search for relevant common sense in the ConceptNet common sense knowledge base. Specifically, the search terms are used sequentially to match the starting nodes of the triples in the common sense database, and the triples whose search terms are the same as the starting nodes are selected. Based on the credibility score given by the common sense knowledge base for each triple, the triplet is selected. The top 20 triples with the highest scores.

202: For the common sense triplet extracted in step 201, first construct a common sense subgraph, and then use aGCN (attention graph convolution network) to encode the subgraph to obtain the common sense feature c;

First, use the 20 common sense triples corresponding to each dialogue unit extracted in step 201 to construct the common sense subgraph G _c . The node features included in G _c are the node features corresponding to the start nodes and end nodes of all common sense triples. Specifically, Glove word vector embedding is performed on the start node and end node in each common sense triplet to obtain the node features. Among them, d refers to the embedding dimension, and i refers to the i-th node.

Afterwards, aGCN (Attention Graph Convolutional Network) is used to further encode the common sense subgraph G _c to update the current node feature z _i . specifically:

Among them, W is a learnable linear transformation matrix; N(i) is the set of adjacent nodes of the i-th node; s is the nonlinear activation function sigmoid; L is the number of convolutional layers of the aGCN network; a _ij is the i-th node The predefined weight between node feature z _i and j-th node feature z _j ;

specifically:

a _ij =softmax(u _ij )

Among them, W _h and All are learnable parameters; [·,·] refers to splicing node features; softmax is a nonlinear activation function.

Finally, each node feature in the common sense subgraph G _c is updated through aGCN (attention graph convolution network), and has the feature information of its adjacent nodes. All updated node features are average pooled and the common sense feature c is output.

203: Concepts of causal diagrams and counterfactual analysis;

In the embodiment of the present invention, a powerful graphical tool—causality diagram—is used to explore the causal relationship behind variables. It is a directed acyclic graph. Each node in a cause-and-effect diagram represents a variable, and the arrows between variables represent the causal connection between them. Usually the starting point of the arrow is the "cause" and the end point is the "result." As shown in Figure 2(a), X→Y means that X has a direct impact on Y, where X is the "cause" and Y is the "result". X→M→Y means that X indirectly affects Y through the intermediary M. When a variable is assigned a ^certain value, such as X=x ^and The calculation method of the total effect (TE) of X on Y is:

in, and/> Respectively represent the influence of variable X on variable Y through X→Y and X→M→Y when X=x and X=x ^* .

Furthermore, the total effect TE can be decomposed into natural direct effect (NDE) and total indirect effect (TIE), namely:

TE＝NDE+TIE

Specifically, in order to calculate the natural direct effect NDE, the embodiment of the present invention constructs a counterfactual world, as shown in Figure 2(d). In the counterfactual world, a variable can be assigned two values at the same time (such as X=x or X=x ^* ), which is obviously impossible to happen in the factual world. In Figure 2(c)), a variable can only be assigned one value at a time (such as X=x or X=x ^* ). When variable X directly affects variable ^Y through X→Y, assign variable X to X=x; when variable X indirectly affects Y through X→M→Y, assign variable The total impact of Y is The calculation method of natural direct effect NDE is:

Intuitively speaking, NDE indicates how much direct influence X has on Y (X→Y) after excluding the indirect influence of X on Y (X→M→Y).

Furthermore, the calculation method of total indirect effect TIE is:

204: Introduce common sense into visual dialogue tasks and construct a visual dialogue causal graph based on common sense fusion;

For most of the existing visual dialogue models based on common sense fusion, a visual dialogue causal diagram based on common sense fusion is constructed, as shown in Figure 3.

First, a visual encoder (most models use Faster-RCNN) is used to extract visual features from the image, and the visual feature is recorded as node I in the causal diagram (Figure 3); the current question and conversation history usually use long short-term memory networks (LSTM) is encoded to obtain the corresponding problem features and historical features, which are represented as node Q and node H respectively in the causal graph. The common sense feature output in step 202 is recorded as node C in the causal graph.

After that, the subgraph (I→K, Q→K, H→K, H→Q→K, C→K) represents the encoder input H, Q, I, C in the visual dialogue model and outputs multi-modal features K , where, since the current question that needs to be answered originates from the previous conversation history, there is H→Q.

Finally, the question feature Q and the multimodal feature K are input into the discriminative decoder or the generative decoder to generate the answer A to the current question, and the corresponding subgraphs are Q→A and K→A.

In addition, as mentioned in step 201, although the search terms used to retrieve common sense are derived from image descriptions in the image and conversation history, that is, I→C, H→C, the subgraph I→C, H→C does not affect the search results based on The causality between the visual dialogue model input (H, Q, I, C) and the output (A) represented by the common sense fusion visual dialogue causality diagram (Figure 3) is not marked on Figure 3 for the sake of simplicity. Although existing visual dialogue models based on common sense fusion may use common sense knowledge to improve answer prediction accuracy in different ways, without loss of generality, subgraph C→K can be used to represent common sense features and multi-modality Causality between features. At the same time, the embodiment of the present invention cuts off the direct connection between H and A, that is, removes the subgraph H→A to prevent the visual dialogue model from learning the language bias contained in H to have a negative impact on answer prediction.

205: According to the visual dialogue causal graph based on common sense fusion in step 204, calculate the total effect TE of the input features (H, Q, I, C) on the prediction of the answer (A).

First, for the visual dialogue causal graph based on common sense fusion in step 204, according to the calculation method of the total effect TE in step 203, the input variables H, Q, I, and C are assigned specific observation values and null values respectively, that is, I=i , Q=q, H=h, C=c and I=i ^* , Q=q ^* , H=h ^* , C=c ^* ; further, for the visual dialogue model based on common sense fusion, the observation value i refers to It is the image feature output by a specific image after using the visual encoder in step 204. The observation values q and h respectively refer to the current question and dialogue history corresponding to the image, which are processed using the long short-term memory network (LSTM) in step 204. The corresponding problem features and historical features are obtained by coding; the observation value c refers to the common sense feature extracted by the current visual dialogue unit through steps 201 and 202; the null values i ^* , q ^* , h ^* , c ^* respectively refer to the visual dialogue The model does not input features corresponding to images, questions, conversation history, and common sense.

Afterwards, when the input variables H, Q, I, C are assigned specific observed values, their impact on the prediction of answer (A) is recorded as A (I=i, Q=q, H=h, C=c), When the input variable is assigned a null value, its impact on the prediction of answer (A) is recorded as A (I=i ^* , Q=q ^* , H=h ^* , C=c ^* ). According to the total effect TE in step 103 The calculation method is:

TE=A(I=i,Q=q,H=h,C=c)-A(I=i ^* ,Q=q ^* ,H=h ^* ,C=c ^* )

In order to simplify the expression, in the embodiment of the present invention, A (I=i, Q=q, H=h, C=c) is abbreviated as A _i,q,h,c , and A (I=i ^* , Q=q ^* , H＝h ^* , C＝c ^* ) are abbreviated as A _{i*, q*, h*, c*} .

As shown in Figure 3, there are two direct paths connecting the answer (A), namely K→A and Q→A. Furthermore, the embodiment of the present invention uses P(·) to represent the effect of K and Q on A. Comprehensive effect, then A _{i, q, h, c} can be recorded as P _{k, q} , where P _{k, q} = P (K = k, Q = q), and the observation value k refers to the visual dialogue model in step 204 The multi-modal feature k output by the encoder in after inputting I=i, Q=q, H=h, C=c; Similarly, A _{i*, q*, h*, c*} can be recorded as P _{k *,q*} , where P _k*,q* =P(K=k*,Q=q*), the null value k* refers to the encoder in the visual dialogue model in step 204 when inputting the null value I=i ^* ,Q=q ^* ,H=h ^* ,C=c ^* The multi-modal null value k* output after; then the total effect TE is expressed as:

Most of the existing visual dialogue methods based on common sense fusion predict answers by deducing the cause-and-effect diagram shown in Figure 3, ignoring the impact of harmful biases caused by introduced common sense on answer prediction; that is, the total effect TE not only includes the introduced The "beneficial" impact of common sense on answer predictions also includes the negative impact of harmful biases on answer predictions.

206: In order to remove the harmful bias caused by the introduced common sense in the visual dialogue task, based on the visual dialogue causal graph based on common sense fusion constructed in step 204, construct its corresponding counterfactual causal graph to estimate the negative impact of harmful bias on answer prediction. Influence.

Regarding the visual dialogue causal graph based on common sense fusion constructed in step 204, common sense C has an indirect impact on answer prediction through multi-modal knowledge K, that is, subgraph C→K→A. However, in practice, the model not only learns "beneficial" common sense knowledge that helps improve answer prediction accuracy, but also learns "harmful" biases that reduce answer prediction accuracy. For example, for the current question "What is the bus used for" that needs to be answered, the common sense triplet extracted in step 201 not only contains the "useful" common sense knowledge <bus, capable of, transport people>, but also helps The model predicts the correct answer "Transport people"; at the same time, the extracted common sense triplet also contains <bus, is a, public transport>, <bus, is a, car> and <bus, at location, road> etc., which forms a "harmful" bias that reduces the accuracy of answer prediction. For example, since the starting node "bus" often appears in the extracted common sense triplet, it will mislead the model to favor candidate answers containing "bus", such as " City bus”, “On a bus”, and even makes its predicted score higher than the correct answer “Transport people”.

However, most of the existing visual dialogue models based on common sense fusion ignore the "harmful" bias caused by the introduced common sense. In the embodiment of the present invention, in order to estimate and remove the negative impact of "harmful" bias on answer prediction, According to the visual dialogue causal graph based on common sense fusion constructed in step 204, its corresponding counterfactual causal graph is constructed. The core idea is to assume that "if only the common sense variable C is input to the visual dialogue model, and the input of the other variables I, Q, H, and K is blocked, what impact will this have on answer prediction?" Specifically, the embodiment of the present invention Evaluate the natural direct effect (NDE) of common sense (C) on the prediction of the answer (A) by blocking the effects of I, Q, H, and K on the prediction of the answer (A).

The counterfactual causal diagram based on common sense fusion constructed by the embodiment of the present invention is shown in Figure 4.

First, based on the causal diagram (Figure 3) based on common sense fusion, the embodiment of the present invention assigns null values to I, Q, H, and C respectively, that is, I=i*, Q=q*, H=h* and C=c*, and then K=k*, blocking the influence of I, Q, H and K on the prediction of answer (A).

After that, as described in step 203, common sense C can be assigned two values at the same time in the counterfactual world, namely C=c* and C=c; among them, the former is used to obtain K=k*, and the latter is related to the answer A direct connection evaluates the natural direct effect (NDE) of common sense (C) on answer prediction (A). Since the model can only predict the answer through common sense (C) in the counterfactual world, the natural direct effect (NDE) preserves the impact of the harmful bias introduced by common sense on the answer prediction.

207: For the counterfactual causal diagram based on common sense fusion constructed in step 206, according to the calculation method of natural direct effect NDE in step 203, estimate the impact of the harmful bias caused by the introduced common sense on the answer prediction, and use it from step 205 removed from the total effect TE described in;

First, for the counterfactual causal diagram based on common sense fusion constructed in step 206, according to the calculation method of natural direct effect NDE in step 203, the natural direct effect NDE of common sense on answer prediction is obtained:

in, Refers to the impact on answer prediction after blocking the input of question Q and multi-modal knowledge K and only retaining common sense C,/> Refers to the impact on answer prediction after all input variables of the visual dialogue model are assigned null values.

After that, in order to remove the harmful bias caused by the introduced common sense on answer prediction, the natural direct effect NDE is removed from the total effect TE, and the total indirect effect TIE is obtained:

Finally, the influence of “beneficial” commonsense knowledge (i.e., TIE) is retained, while the influence of “harmful” common sense deviations (i.e., NDE) is eliminated. What needs to be made clear here is that Equivalent to/> Because they all refer to the impact on answer prediction after all inputs to the model are blocked. In addition, the counterfactual causal reasoning method proposed in the embodiment of the present invention has strong generalization ability and can be applied to various visual dialogue models based on common sense fusion.

208: Based on the total indirect effect TIE of step 207, use the model integration method to train the visual dialogue model, and use cross-entropy loss and KL divergence loss as training targets to obtain the answer prediction results;

Regarding the total indirect effect TIE in step 207, in practical applications, the embodiment of the present invention uses a model integration method for training, as shown in Figure 5. Specifically, the dialogue method of P _q,k can be decomposed into two components, P _q and P _k , which are calculated by neural networks N _Q and N _K respectively, that is, P _q =N _Q (q) and P _k =N _K (i ,q,h,c). N _Q (q) represents the subgraph Q→A, in which the features of the question encoded by LSTM are input into the softmax decoder to obtain the prediction score of the candidate answer. N _K (i,q,h,c) covers the subgraph {I→K,Q→K,H→K,H→Q→K,C→K}, where the relationship between problems, history, images and common sense Multimodal interaction is accomplished using attention mechanisms.

The embodiment of the present invention combines the output results from the two networks to generate the final answer score of P _q,k in the real world:

Among them, s is the nonlinear activation function sigmoid.

Likewise, in the counterfactual world, is decomposed into/> and P _c . As described in step 206, in order to evaluate the direct impact of C on A, all signals including Q and K are blocked, that is, the inputs of N _Q and N _K are invalid. A reasonable strategy to follow in the counterfactual scenario is to assume that both networks will guess and pick out the answer without any specific processing. Specifically, if one or more of i, q, h, or c is not given, then/> and/> is assigned to a learnable parameter m with a uniform distribution.

At the same time, P _c is calculated by the neural network N _C , that is, P _c = N _C (c), corresponding to the subgraph C A. Specifically, common sense features are encoded with LSTM, and then candidate answers are ranked using 2-layer MLP. Finally, all the output results of the network are fused to obtain /> in the counterfactual world. Answer points for:

In the inference stage, counterfactual reasoning is used for unbiased commonsense learning to facilitate VD prediction:

The network is trained by minimizing the following loss function:

Among them, a is the ground truth answer; γ is the loss weight; CE represents the cross entropy loss between the real answer and its predicted answer. Specifically, an ensemble model is used to perform counterfactual calculations, where the three branches K→A, Q→A, and C→A are above P _k , P _q , and P _c , respectively. At the same time, in order to control The sharpness of the distribution, using KL divergence D _kl to optimize the learnable parameter m:

Among them, p(a|q,k)=softmax(P _q,k ), Softmax is a nonlinear activation function. When calculating D _kl , only the learnable parameter m is optimized.

209: Integrate image features, conversation history features, common sense features and current question features and send them to the decoder. At the same time, the loss function is minimized, the network parameters are optimized, and finally the correct answers to questions raised by humans for the current scene are generated.

Specifically, based on the factual causal graph fused with common sense, the visual dialogue network inputs real image features, conversation history features, common sense features and current problem features, and then fed into the decoder after fusion; Based on the counterfactual causal graph fused with common sense, the visual dialogue network The dialogue network still inputs real common sense features, and at the same time uses learnable parameters to replace the input of real image features, conversation history features and current question features, and then sends them to the same decoder after fusion; Calculation method of total indirect effect TIE based on step 207 As well as the output results of the decoder in these two cases, the ranking prediction results of the candidate answers are obtained.

At the same time, minimizing the loss function described in step 208 is used as the training goal, and the visual dialogue network parameters are continuously optimized to achieve the optimal performance of the answer prediction results, and finally obtain accurate answers to questions raised by humans for the current scene. The solution of the present invention has certain generalizability and can be applied to most visual dialogue models. It improves the answer prediction accuracy of the agent when answering rescuers' questions in real time, so that rescuers can fully understand the environmental information of the disaster site, thereby Better rescue plans can be made to reduce the difficulty of rescue and avoid rescue casualties.

Example 3

The feasibility verification of the solutions in Examples and 2 is carried out below with specific calculation examples and Table 1 and Table 2. See the following description for details:

In order to verify the effect of the present invention, experiments were conducted using the method of the present invention on the Visdial-v1.0 and Visdial-v0.9 data sets respectively. Evaluation indicators for answer prediction include recall rate (R@1, R@5, R@10), normalized loss cumulative gain (NDCG), mean reciprocal ranking of correct answers (MRR) and mean ranking of correct answers (Mean ). The higher the recall rate (R@1, R@5, R@10), normalized loss cumulative gain (NDCG) and the average reciprocal ranking (MRR) of the correct answer, the higher the average ranking (Mean) of the correct answer. Low means that the visual dialogue generation method is better. The experimental results are shown in Table 1 (using the Visdial-v1.0 data set) and Table 2 (using the Visdial-v0.9 data set) respectively.

Table 1 Results of the method of the present invention on the Visdial-v1.0 test set

Table 2 Results of the method of the present invention on the Visdial-v0.9 verification set

It can be seen from the results in Table 1 and Table 2 that the present invention introduces common sense into the visual dialogue method, verifying that common sense knowledge can effectively improve the accuracy of answer prediction; at the same time, after introducing common sense knowledge, the present invention uses counterfactual analysis method to remove common sense. The contained "harmful bias" has a negative impact on answer generation, while retaining common sense has a positive impact on answer generation. The experimental results verify the effectiveness of the present invention.

The solution of the invention can be applied in disaster rescue missions. When a disaster occurs, rescuers can make the intelligent agent arrive at the disaster scene first and input the disaster scene images, relevant common sense knowledge and question and answer history into the intelligent agent. The intelligent agent can answer the rescuer's questions in real time based on this information, so that the rescuer can fully Understand the disaster scene environment, formulate a reasonable rescue plan, and reduce rescue casualties.

Example 4

A visual dialogue generation device based on counterfactual common sense causal reasoning. The device includes: a processor and a memory. Program instructions are stored in the memory. The processor calls the program instructions stored in the memory to cause the device to perform any of the method steps:

For the extracted common sense triplet, a common sense subgraph is constructed, and the attention graph convolution network is used to encode the subgraph to obtain common sense features;

Introduce common sense into the visual dialogue task, construct a visual dialogue causal graph based on common sense fusion, and calculate the total effect of input features on answer prediction; based on the visual dialogue causal graph, construct its corresponding counterfactual causal graph;

For counterfactual causal diagrams, the impact of harmful biases introduced by common sense on answer predictions is estimated based on natural direct effects and removed from the total effect;

Model ensemble is used to train the visual dialogue model, and cross-entropy and KL divergence loss are used as training targets to obtain answer prediction results;

Integrate image features, conversation history features, common sense features and current problem features and send them to the decoder to minimize the loss function, optimize network parameters, and finally provide real disaster environment information for disaster relief sites.

Among them, the common sense triplet is: extract the common sense related to each visual dialogue unit in the training set, verification set and test set samples of the database; the specific operation of the extracted common sense triplet is:

Use the Faster R-CNN framework to detect object labels, and select the top several object labels with the highest scores based on the confidence scores corresponding to each label; similarly, select the top few keywords with the highest scores for image descriptions;

Further, using the attention graph convolutional network to encode the subgraph to obtain common sense features is: using aGCN to encode the common sense subgraph G _c to update the current node feature z _i ;

Among them, W is a learnable linear transformation matrix; N(i) is the set of adjacent nodes of the i-th node; s is the nonlinear activation function sigmoid; L is the number of convolutional layers of the aGCN network; a _ij is the i-th node The predefined weight between node feature z _i and j-th node feature z _j ; common sense feature c is output after average pooling of all updated node features.

Among them, common sense is introduced into the visual dialogue task and a visual dialogue causal graph based on common sense fusion is constructed as follows:

Use the visual encoder to extract visual features from the image. The visual features are recorded as node I in the causal diagram; they are represented as node Q and node H respectively in the causal diagram, and the common sense features are recorded as node C in the causal diagram;

After that, the subgraph (I→K, Q→K, H→K, H→Q→K, C→K) represents the encoder input H, Q, I, C in the visual dialogue model and outputs multi-modal features K ;

The question feature Q and the multimodal feature K are input into the discriminative decoder or the generative decoder to generate the answer A to the current question, and the corresponding subgraphs are Q→A and K→A.

Furthermore, the total effect of input features on answer prediction is calculated as:

First, the input variables H, Q, I, and C are assigned specific observed values and null values respectively, that is, I=i, Q=q, H=h, C=c and I=i ^* , Q=q ^* , H =h ^* ,C=c ^* ;

When the input variables H, Q, I, and C are assigned specific observed values, the impact on the prediction of answer A is recorded as A (I=i, Q=q, H=h, C=c), and the input variables are assigned empty values. When the value is , the impact on the prediction of answer A is recorded as A (I=i ^* , Q=q ^* , H=h ^* , C=c ^* ). The calculation method of the total effect TE is:

Among them, based on the visual dialogue causal diagram, the corresponding counterfactual causal diagram is constructed as:

By assigning null values to I, Q, H, and C respectively, that is, I=i*, Q=q*, H=h*, and C=c*, and then K=k*, blocking I, Q, H, and The impact of K on the prediction of answer A;

Common sense C can be assigned two values at the same time in the counterfactual world, namely C=c* and C=c; the former is used to obtain K=k*, and the latter is directly connected to the answer A to evaluate the answer prediction of common sense C A natural direct effect.

Furthermore, for the counterfactual causal diagram, the impact of the harmful bias introduced by common sense on answer prediction is estimated based on the natural direct effect, and is removed from the total effect as:

in, Refers to the impact on answer prediction after blocking the input of question Q and multi-modal knowledge K and only retaining common sense C,/> Refers to the impact on answer prediction after all input variables of the visual dialogue model are assigned null values;

The total indirect effect TIE is obtained as:

Among them, model ensemble is used to train the visual dialogue model, and cross-entropy and KL divergence loss are used as training targets to obtain the answer prediction result:

Among them, s is the nonlinear activation function sigmoid, is decomposed into/> and P _c are calculated by neural networks N _Q , N _K and N _C respectively. P _q,k is decomposed into two components P _q and P _k , which are calculated by neural networks N _Q and N _K respectively.

It should be pointed out here that the device description in the above embodiments corresponds to the method description in the embodiments, and the embodiments of the present invention will not be described again here.

The execution subjects of the above-mentioned processors and memories can be computers, microcontrollers, microcontrollers and other devices with computing functions. During specific implementation, the embodiments of the present invention do not limit the execution subjects and can be selected according to the needs of actual applications.

Data signals are transmitted between the memory and the processor through a bus, which will not be described in detail in the embodiment of the present invention.

Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium. The storage medium includes a stored program. When the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiments.

The computer-readable storage medium includes but is not limited to flash memory, hard disk, solid state drive, etc.

It should be pointed out here that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be described again here.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, processes or functions according to embodiments of the present invention are generated in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. Computer instructions may be stored in or transmitted over a computer-readable storage medium. Computer-readable storage media can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or other integrated media that contains one or more available media. Available media may be magnetic media or semiconductor media, etc.

references:

[1]X.Yang, H.Zhang, and J.Cai, "Deconfounded image captioning: A causalretrospect," IEEE Transactions on Pattern Analysis and Machine Intelligence, pp.1–1, 2021.

[2]X.Han, .Video Technol.,vol.32,pp.8611–8622,2022.

[3] Y. Niu, K. Tang, H. Zhang, Z. Lu, X. Hua, and J. Wen, "Counterfactual VQA: A cause-effect look at language bias," in CVPR, 2021, pp. 12 700 –12 710.

[4] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. F. Moura, D. Parikh, and D. Batra, "Visual dialog," in CVPR, 2017, pp. 1080–1089.

[5] J.Lu, A.Kannan, J.Yang, D.Parikh, and D.Batra, "Best of both worlds: Transferring knowledge from discriminative learning to a generative visualdialog model," in NIPS, 2017, pp.314 –324.

[6] Y.Niu, H.Zhang, M.Zhang, J.Zhang, Z.Lu, and J.Wen, “Recursive visualattention in visual dialog,” in CVPR, 2019, pp.6679–6688.

[7] –1273.

[8] D.Guo, H.Wang, and M.Wang, "Context-aware graph inference with knowledge distillation for visual dialog," IEEE Trans.PatternAnal.Mach.Intell., vol.44, no.10, pp.6056 –6073, 2022. [9] L. Zhao, H. Zhang, X. Li, S. Yang, and Y. Song, “You should know more: Learning external knowledge for visual dialog,” Neurocomputing, vol.488, pp. 54–65, 2022.

[10] S. Zhang,

[11] R.Speer, J.Chin, and C.Havasi, “Conceptnet 5.5: An open multilingualgraph of general knowledge,” in AAAI, 2017, pp.4444–4451.

The embodiments of the present invention do not limit the models of each device unless otherwise specified, as long as the devices can complete the above functions.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the above-mentioned serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.

Claims (10)

1. A visual dialogue generation method based on counterfactual common sense causal reasoning, characterized in that the method includes: For the extracted common sense triplet, a common sense subgraph is constructed, and the attention graph convolution network is used to encode the subgraph to obtain common sense features; Introduce common sense into the visual dialogue task, construct a visual dialogue causal graph based on common sense fusion, and calculate the total effect of input features on answer prediction; based on the visual dialogue causal graph, construct its corresponding counterfactual causal graph; For counterfactual causal diagrams, the impact of harmful biases introduced by common sense on answer predictions is estimated based on natural direct effects and removed from the total effect; Model ensemble is used to train the visual dialogue model, and cross-entropy and KL divergence loss are used as training targets to obtain answer prediction results; Integrate image features, conversation history features, common sense features and current problem features and send them to the decoder to minimize the loss function, optimize network parameters, and finally provide real disaster environment information for disaster relief sites. 2. A kind of visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that the common sense triplet is: extracting the training set, verification set and test set sample of the database with each Common sense related to each visual dialogue unit; the specific operation of the extracted common sense triplet is: Use the Faster R-CNN framework to detect object labels, and select the top several object labels with the highest scores based on the confidence scores corresponding to each label; similarly, select the top few keywords with the highest scores for image descriptions. 3. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that the common sense feature obtained by encoding the subgraph using an attention graph convolutional network is: using aGCN to encode the common sense subgraph Graph G _c updates the current node feature z _i ; Among them, W is a learnable linear transformation matrix; N(i) is the set of adjacent nodes of the i-th node; s is the nonlinear activation function sigmoid; L is the number of convolutional layers of the aGCN network; a _ij is the i-th node The predefined weight between node feature z _i and j-th node feature z _j ; common sense feature c is output after average pooling of all updated node features. 4. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that the introduction of common sense into the visual dialogue task and the construction of a visual dialogue causal graph based on common sense fusion are specifically: Use the visual encoder to extract visual features from the image. The visual features are recorded as node I in the causal diagram; they are represented as node Q and node H respectively in the causal diagram, and the common sense features are recorded as node C in the causal diagram; After that, the subgraph (I→K, Q→K, H→K, H→Q→K, C→K) represents the encoder input H, Q, I, C in the visual dialogue model and outputs multi-modal features K ; The question feature Q and the multimodal feature K are input into the discriminative decoder or the generative decoder to generate the answer A to the current question, and the corresponding subgraphs are Q→A and K→A. 5. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that the total effect of the calculated input features on answer prediction is: First, the input variables H, Q, I, and C are assigned specific observed values and null values respectively, that is, I=i, Q=q, H=h, C=c and I=i ^* , Q=q ^* , H =h ^* ,C=c ^* ; When the input variables H, Q, I, and C are assigned specific observed values, the impact on the prediction of answer A is recorded as A (I=i, Q=q, H=h, C=c), and the input variables are assigned empty values. When the value is , the impact on the prediction of answer A is recorded as A (I=i ^* , Q=q ^* , H=h ^* , C=c ^* ). The calculation method of the total effect TE is: TE=A(I=i,Q=q,H=h,C=c)-A(I=i ^* ,Q=q ^* ,H=h ^* ,C=c ^* ) =P _k,q -P _k*,q* . 6. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, based on the visual dialogue causal diagram, constructing its corresponding counterfactual causal diagram is: By assigning null values to I, Q, H, and C respectively, that is, I=i*, Q=q*, H=h*, and C=c*, and then K=k*, blocking I, Q, H, and The impact of K on the prediction of answer A; Common sense C can be assigned two values at the same time in the counterfactual world, namely C=c* and C=c; the former is used to obtain K=k*, and the latter is directly connected to the answer A to evaluate the answer prediction of common sense C A natural direct effect. 7. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that, for the counterfactual causal diagram, the answer prediction is based on harmful biases caused by common sense introduced by natural direct effect estimation. and removing it from the total effect is: in, Refers to the impact on answer prediction after blocking the input of question Q and multi-modal knowledge K and only retaining common sense C,/> Refers to the impact on answer prediction after all input variables of the visual dialogue model are assigned null values; The total indirect effect TIE is obtained as: 8. A visual dialogue generation method based on counterfactual common sense causal reasoning according to claim 1, characterized in that the visual dialogue model is trained using model integration, and the answer is obtained with cross entropy and KL divergence loss as training targets. The predicted result is: Among them, s is the nonlinear activation function sigmoid, is decomposed into/> and P _c are calculated by neural networks N _Q , N _K and N _C respectively. P _q,k is decomposed into two components P _q and P _k , which are calculated by neural networks N _Q and N _K respectively. 9. A visual dialogue generation device based on counterfactual common sense causal reasoning, characterized in that the device includes: a processor and a memory, program instructions are stored in the memory, and the processor calls the program instructions stored in the memory To cause the device to perform the method steps described in any one of claims 1-8. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program includes program instructions, and when executed by a processor, the program instructions cause the processor to execute the right The method steps described in any one of claims 1-8.