CN104021224A - Image labeling method based on layer-by-layer label fusing deep network - Google Patents

Abstract

The invention discloses an image labeling method based on a layer-by-layer label fusing deep network. The method comprises the following steps: extracting a bottom layer vision characteristic for a training image with centralized training; layering the label of the training image to construct a hierarchical structure of the label; fusing the bottom vision characteristic information and label information layer by layer for the training image and obtaining the layered characteristic representation of the training image through parameter learning of the deep network; extracting a bottom layer vision characteristic for a testing image with centralized test, thereby obtaining the layered characteristic representation through deep network learning; and finally, forecasting the labeling information of the image according to layered characteristic representation of the testing image. The image labeling method disclosed by the invention belongs to layered labeling and is more precise than conventional labeling methods.

Description

Image annotation method based on layer-by-layer label fusion deep network

technical field

The invention relates to the technical field of social network image tagging, in particular to an image tagging method based on layer-by-layer tag fusion deep network.

Background technique

In recent years, with the continuous development of social media, the number of images on social platforms has exploded. How to label massive social images has become an important research content in the field of network multimedia.

The current mainstream image annotation methods mainly focus on methods based on visual information. This type of method first extracts the underlying features, and then uses machine learning models to classify images based on feature representation. This type of method has achieved good results to a certain extent, but its effect is still not ideal because it only uses visual information and ignores its contextual text information.

The core of image annotation is to use image-related information (including vision, contextual text label information, etc.) to understand image content, integrate image label information and visual information, and obtain more expressive image features. Social imagery has an important facilitative effect. However, the heterogeneity of visual features and text label information brings challenges to the fusion of the two types of information. The image annotation method based on the layer-by-layer label fusion deep network proposed in the present invention fuses the two types of information layer by layer and solves the problem of heterogeneity. The problem of structural information fusion plays an important role in social image annotation.

Contents of the invention

In order to solve the above-mentioned problems existing in the prior art, the present invention proposes an image labeling method based on layer-by-layer label fusion deep network.

A kind of image labeling method based on layer-by-layer label fusion depth network proposed by the present invention comprises the following steps:

Step 1. For the training image in the training set, extract its underlying visual feature X;

Step 2. Hierarchize the labels of the training images to construct a hierarchical structure of the labels;

Step 3, for the training image, fuse its underlying visual feature information and label information layer by layer, and learn the hierarchical feature representation of the training image through deep network parameter learning;

Step 4. For the test images in the test set, extract their underlying visual features, then obtain their hierarchical feature representations through the deep network learning, and finally predict their annotation information according to the hierarchical feature representations of the test images.

Internet image annotation has been widely used in many important related fields. Vision-based image annotation is a challenging problem due to the existence of a semantic gap between visual top-level information and high-level semantics. The above-mentioned image tagging method based on layer-by-layer tag fusion deep network proposed by the present invention can automatically tag social images, and in addition, the layered tagging method of the present invention is more accurate than the traditional tagging method.

Description of drawings

Fig. 1 is the flowchart of the image tagging method based on layer-by-layer tag fusion depth network according to an embodiment of the present invention;

Figure 2 is an example diagram of label hierarchy;

Fig. 3 is a model structure diagram of a layer-by-layer feature fusion deep network according to an embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

The relevant data sets involved in the method proposed by the present invention include: 1) training set, which includes images and social labels corresponding to the images; 2) test set, which only includes test images to be labeled without label information.

Considering the heterogeneity of the underlying visual information and social tag information of the image, the present invention proposes an image tagging method based on a layer-by-layer tag fusion deep network. The core idea of this method is to fuse label information and visual information layer by layer under the framework of deep network, so as to learn the hierarchical features of images and provide feature representation for image annotation.

Fig. 1 shows the flow chart of the image labeling method based on the layer-by-layer label fusion depth network proposed by the present invention. As shown in Fig. 1, the method includes:

Step 1, for the training image in the training set, extract its underlying visual features;

Step 2. Hierarchize the labels of the training images to construct a hierarchical structure of the labels;

Step 3, for the training image, fuse its underlying visual feature information and label information layer by layer, and learn the hierarchical feature representation of the training image through deep network parameter learning;

Step 4. For the test images in the test set, extract their underlying visual features, then obtain their hierarchical feature representations through the deep network learning, and finally predict their annotation information according to the hierarchical feature representations of the test images.

The specific execution process of the above four steps will be described in detail below.

In step 1, the underlying visual feature extraction of the object is to obtain the initial representation of the object. For image information, the present invention preferably adopts the scale-invariant feature transform feature (SIFT) (such as 1000 dimensions) as the underlying visual feature of the image, and the underlying visual feature of the image Features are denoted by X.

In step 2, using some available tools, WordNet is preferred in the present invention, and a label hierarchy with a layer number of K is constructed for the social label of the image. For example: if an image has tags animal, plant, cat, dog, flower, the corresponding tag hierarchy is shown in Figure 2 (here the number of layers is 2).

The step 3 is to fuse the underlying visual feature information and label information of the training image layer by layer, and obtain the hierarchical features of the training image through deep network parameter learning.

In step 3, a deep network with layers L (L>K) is constructed, and the K layer of the label hierarchy corresponds to the highest layer of the deep network. Let the variables of each layer of the deep network be expressed as h={h ⁽⁰⁾ , ..., h ^(L) }, wherein, h ⁽⁰⁾ represents the underlying visual feature X of the image; each of the label hierarchy structure corresponding to the K layer The variables of a layer are denoted as y={y ^(L-K+1) ,...,y ^(L) }.

This step is an important part of the present invention. FIG. 3 is a model structure diagram of a layer-by-layer feature fusion depth network according to an embodiment of the present invention. Referring to FIG. 3, the step 3 can be divided into the following sub-steps:

Step 3.1: Preliminarily adjust the parameters of the deep network from layer h ⁽⁰⁾ to layer h ^(L-K+1) based on the reconstruction error by constructing an auto-encoder;

Said step 3.1 further comprises the following steps:

Step 3.1.1: From layer h ⁽⁰⁾ up to layer h ^(L-K+1) , build an autoencoder between every two adjacent layers, through which the autoencoder can be obtained from the representation of the next layer The mapping represented by the previous layer;

For example, based on the autoencoder between the h ^(l-1) and h ^(l) layers, the representation of the h ^(l-1) layer can be mapped to the h ^(l) layer representation:

${\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

in, Represents the weight parameter between h ^(l-1) and h ^(l) layers, b ^(l) represents the bias (bias) parameter of h ^(l) layer, s() represents the logistic function:

In this way, the representation of h ^{(l) layer can be obtained through mapping from the representation of h (l-1 )} layer.

Step 3.1.2: Map back from the representation of the previous layer to obtain the reconstructed representation of the next layer;

For example, the reconstructed representation z of h ^{(l-1) can} be obtained by mapping the representation of h (l):

$\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; \&prime;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

in, for The transpose representation of , b' represents the bias (bias) parameter of h ^(l-1) .

Step 3.1.3: Adjust the parameters of the deep network according to the error between the correct representation and the reconstructed representation.

For example, the initial adjustment of the deep network parameters can be achieved by minimizing the reconstruction error between z and h ^(l-1) layer representations. In an embodiment of the present invention, it is preferable to minimize the reconstruction cross entropy to Make preliminary adjustments to the above parameters:

Among them, k represents the subscript of the component of z, and D ^(l-1) represents the dimension of z.

Proceed in this way until the h ^(L-K+1) layer is adjusted.

Step 3.2: For the h ^(L-K+1 ) layer to the highest h ^(L) layer in the deep network, combine a certain layer in the deep network, such as the h ^(l) layer and the corresponding layer in the label hierarchy , such as the u ^(l) layer, performing feature fusion and adjusting corresponding parameters in the deep network;

This step can be divided into two sub-steps: (take h ^(l) as an example)

Step 3.2.1: Utilize the y ⁽¹⁾ layer label in the label hierarchy structure to adjust the parameters from h ⁽⁰⁾ to h ⁽¹⁾ layer in the deep network;

In this step, first calculate the cross-entropy loss:

$\mathrm{<span\; class="notranslate">\; loss</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; \{</span>\mathrm{<span\; class="notranslate">\; W</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{\mathrm{<span\; class="notranslate">\; t</span>}}_{\mathrm{<span\; class="notranslate">\; nk</span>}}\mathrm{<span\; class="notranslate">\; ln</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; nk</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Among them, N represents the number of samples, K represents the number of labels in this layer, y _nt represents the value of the k-th dimension of the model's prediction of the n-th sample, and t _nk represents the k-th dimension of the n-th sample in the training sample the true value of .

Then the loss is reversed to adjust the parameters of the deep network from h ⁽⁰⁾ to h ^(l) layers. In an embodiment of the present invention, the well-known backpropagation algorithm is used to adjust the global parameters.

Step 3.2.2: Obtain the feature representation of the h ( ^l+1) layer by combining the representations of the h ^(l) layer and the y ^(l) layer;

In this step, the representations of layer h ^(l) and layer y ^(l) are combined to form an auto-encoder with the representation of layer h ^(l+1) :

${\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; h</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; b</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Likewise, the parameters between h ^(l) , y ^(l) and h ^(l+1) are optimized by minimizing the reconstruction cross-entropy.

And so on, until the h ^(L) layer.

Through the above-mentioned layer-by-layer feature fusion, the label information of the image can be fused into the visual information, and the parameters of the deep network are also optimized.

In step 4, use the deep network whose parameters have been optimized to annotate the test images in the test set.

The step 4 is further divided into the following sub-steps:

Step 4.1: Extracting the underlying visual features X _test of the test image, this step is similar to the method of extracting the underlying visual features of the training images in the training set in step 1;

Step 4.2: Using the deep network after optimizing the parameters, obtain the hierarchical feature representation {h ^(L-K+1) ,..., h ^(L) } of the underlying visual feature X _test of the test image;

Step 4.3: Use the hierarchical feature representation to predict the label information {h ^(L-K+1) , ..., h ^(L) } of the test image:

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

Among them, W _i represents the label and the weight between feature h ^(l) .

The specific embodiments described above have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above descriptions are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. An image annotation method based on a layer-by-layer label fusion depth network is characterized by comprising the following steps:

step 1, extracting a bottom layer visual characteristic X of a training image in a training set;

step 2, carrying out hierarchy on the labels of the training images to construct a hierarchical structure of the labels;

step 3, fusing bottom layer visual characteristic information and label information of the training image layer by layer, and obtaining the hierarchical characteristic representation of the training image through deep network parameter learning;

and 4, extracting bottom layer visual features of the test images in the test set, then obtaining hierarchical feature representation of the test images through deep network learning, and finally predicting the labeling information of the test images according to the hierarchical feature representation of the test images.

2. The method of claim 1, wherein the underlying visual features of the training image are its scale-invariant feature transform features.

3. The method of claim 1, wherein the deep network has a number of layers L and the label hierarchy has a number of layers K, wherein L is>K, and the variable of each layer of the deep network is represented as h = { h = { h⁽⁰⁾，...，h^(L)In which h⁽⁰⁾An underlying visual feature X representing an image; the variable of each layer corresponding to the label hierarchy is represented as y = { y = { y =^(L-K+1)，...，y^(L)}。

4. The method according to claim 3, wherein the step 3 comprises the steps of:

step 3.1: by constructing an auto-encoder, the reconstruction error is based on the slave h in the depth network⁽⁰⁾Layer to h^(L-K+1)The parameters of the layer are preliminarily adjusted;

step 3.2: for h in the deep network^(L-K+1)Layer to maximum h^(L)Layer, incorporating a layer in a deep network, such as h^(l)Layers and corresponding layers in the hierarchy of labels, e.g. y^(l)And the layer is used for carrying out feature fusion and adjusting corresponding parameters in the deep network.

5. The method according to claim 4, characterized in that said step 3.1 further comprises the steps of:

step 3.1.1: from h⁽⁰⁾Layer up to h^(L-K+1)Layers, a self-plaiting structure is constructed between each two adjacent layersA coder by which a mapping of a representation of an upper layer is derivable from a representation of a lower layer;

step 3.1.2: mapping back from the previous layer representation to obtain a reconstructed representation of the next layer;

step 3.1.3: adjusting parameters of the deep network up to h according to errors between a correct representation and a reconstructed representation^(L-K+1)And (3) a layer.

6. The method according to claim 5, characterized in that in step 3.1.3, the parameters of the deep network are adjusted using a minimum reconstruction cross entropy.

7. The method according to claim 4, characterized in that said step 3.2 further comprises the steps of:

step 3.2.1: utilizing a certain level y in the hierarchy of labels^(l)The label adjusts the slave h in the deep network⁽⁰⁾To h^(l)Parameters of the layer;

step 3.2.2: through h^(l)Layer and y^(l)Layer representation merge learning to get h^(l+1)The characteristic of the layer is expressed, and the corresponding parameter of the deep network is adjusted until h^(L)And (3) a layer.

8. The method according to claim 7, characterized in that in step 3.2.1 and step 3.2.2, parameters are adjusted for the deep network using a back propagation algorithm based on cross entropy loss.

9. The method of claim 7, wherein in step 3.2.2, h is^(l)Layer and y^(l)The representation of the layer is combined with h^(l+1)The representation of the layers constitutes an auto-encoder.

10. The method of claim 1, wherein the step 4 further comprises the steps of:

step 4.1: extracting bottom layer visual features of the test image;

step 4.2: obtaining a hierarchical feature representation of the bottom visual features of the test image by using the depth network;

step 4.3: predicting label information of the test image using a hierarchical feature representation of the test image.