CN111340051A - Picture processing method and device and storage medium - Google Patents

Abstract

Embodiments of the present invention provide a picture processing method, device, and storage medium; the method includes: preprocessing a picture to be detected to obtain a processed picture; and extracting image feature information and text feature respectively from the processed picture information; according to the image feature information and the text feature information, determine the type of the picture to be detected, wherein the picture type includes a normal picture or an abnormal picture. The picture processing method, device and storage medium provided by the present invention can improve the accuracy of picture detection and identification.

Description

Image processing method, device and storage medium

technical field

Embodiments of the present invention relate to the technical field of image processing, and in particular, to a picture processing method, device, and storage medium.

Background technique

With the popularity of the Internet, pictures are widely used in more and more web pages and applications because of their advantages of intuitive expression and rich content compared to text. For example, online shopping platforms provide various e-commerce companies with various commodity information release mechanisms, and merchants can upload multi-angle and multi-background product photos to attract users.

Many Internet e-commerce companies upload pictures that do not meet the regulations in order to gain attention. Therefore, it is more and more important to process risky pictures or abnormal pictures in a big data environment. In the existing technology, convolution is usually used. Neural network (Convolution Neural Network, CNN) technology detects and classifies pictures. Among them, CNN technology mainly extracts the image feature information in the picture, and judges whether the picture is a normal picture or an abnormal picture according to the image feature information of the picture.

However, since the CNN technology can only extract the image feature information in the picture, for the picture containing a large amount of text information, when the picture is detected by the CNN technology, the false detection rate of the picture is high, resulting in the accuracy of the picture detection. lower.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a picture processing method, device, and storage medium, which can improve the accuracy of picture detection and recognition.

According to a first aspect of the embodiments of the present invention, there is provided a picture processing method, the method comprising:

Preprocess the image to be detected to obtain the processed image;

From the processed picture, extract image feature information and text feature information respectively;

According to the image feature information and the text feature information, the type of the picture to be detected is determined, wherein the type of the picture includes a normal picture or an abnormal picture.

Optionally, determining the type of the picture to be detected according to the image feature information and the text feature information includes:

Perform splicing processing on the image feature information and the text feature information to obtain splicing feature information;

Perform dimensionality reduction processing of manifold data on the splicing feature information to obtain an embedded feature of the splicing feature information;

The type of the picture to be detected is determined according to the embedded feature of the splicing feature information and the predetermined hypersphere.

Optionally, determining the type of the picture to be detected according to the embedded feature of the splicing feature information and a predetermined hypersphere, including:

determining whether the embedded feature is within the hypersphere;

If the embedded feature is within the hypersphere, determining that the picture to be detected is an abnormal picture;

If the embedded feature is not within the hypersphere, it is determined that the picture to be detected is a normal picture.

Optionally, before determining the type of the picture to be detected according to the embedded feature of the splicing feature information and a predetermined hypersphere, the method further includes:

Get multiple sample images;

respectively extracting the mapping feature of each sample picture in the plurality of sample pictures;

Perform data description on each of the mapping features to obtain the hypersphere.

Optionally, performing data description on each of the mapping features to obtain the hypersphere includes:

The hypersphere is obtained by performing data description on each of the mapping features through the support vector data description SVDD algorithm.

Optionally, extract image feature information from the processed picture, including:

Image feature information is extracted from the processed image through a multi-level convolutional neural network CNN.

Optionally, the multi-level convolutional neural network CNN is used to extract image feature information from the processed picture, including:

The processed picture is input into the first-level CNN network, and the first mapping image corresponding to the processed picture and the first region frame coordinates of the target image in the processed picture are obtained;

The first mapping image and the coordinates of the first area frame are input into the second-level CNN network to obtain the second mapping image corresponding to the processed picture and the first image of the target image in the processed picture. Two area frame coordinates;

The second map image and the coordinates of the second region frame are input into the third-level CNN network to obtain the image feature information.

Optionally, extract text feature information from the processed picture, including:

The text feature information is extracted from the processed image through a multi-level convolutional neural network CNN and a recurrent neural network RNN.

Optionally, the said to-be-detected picture is preprocessed to obtain a processed picture, including:

Perform image pyramid processing on the picture to be detected to obtain the processed picture.

Optionally, before the image pyramid processing is performed on the picture to be detected and the processed picture is obtained, the method further includes:

Perform image color correction processing on the to-be-detected picture to obtain a corrected picture;

Performing image pyramid processing on the picture to be detected to obtain the processed picture, including:

Perform image pyramid processing on the corrected picture to obtain the processed picture.

According to a second aspect of the embodiments of the present invention, there is provided a picture processing apparatus, the apparatus comprising:

The preprocessing module is used to preprocess the image to be detected to obtain the processed image;

a first extraction module, used for extracting image feature information and text feature information from the processed picture, respectively;

A determination module, configured to determine the type of the picture to be detected according to the image feature information and the text feature information, wherein the type of the picture includes a normal picture or an abnormal picture.

Optionally, the determining module includes:

a splicing submodule, configured to perform splicing processing on the image feature information and the text feature information to obtain splicing feature information;

A dimensionality reduction processing submodule, configured to perform dimensionality reduction processing of manifold data on the splicing feature information to obtain the embedded feature of the splicing feature information;

A determination sub-module, configured to determine the type of the picture to be detected according to the embedded feature of the splicing feature information and the predetermined hypersphere.

Optionally, the determining submodule is specifically used for:

determining whether the embedded feature is within the hypersphere;

If the embedded feature is within the hypersphere, determining that the picture to be detected is an abnormal picture;

If the embedded feature is not within the hypersphere, it is determined that the picture to be detected is a normal picture.

Optionally, the device further includes:

The acquisition module is used to acquire multiple sample images;

The second extraction module is used to extract the mapping feature of each sample picture in the plurality of sample pictures respectively;

The data description module is used for performing data description on each of the mapping features to obtain the hypersphere.

Optionally, the data description module is specifically used for:

The hypersphere is obtained by performing data description on each of the mapping features through the support vector data description SVDD algorithm.

Optionally, the first extraction module is further configured to extract image feature information from the processed picture through a multi-level convolutional neural network CNN.

Optionally, the first extraction module is also used for:

The processed picture is input into the first-level CNN network, and the first mapping image corresponding to the processed picture and the first region frame coordinates of the target image in the processed picture are obtained;

The first mapping image and the coordinates of the first area frame are input into the second-level CNN network to obtain the second mapping image corresponding to the processed picture and the first image of the target image in the processed picture. Two area frame coordinates;

The second map image and the coordinates of the second region frame are input into the third-level CNN network to obtain the image feature information.

Optionally, the first extraction module is further configured to extract the text feature information from the processed picture through a multi-level convolutional neural network CNN and a recurrent neural network RNN.

Optionally, the preprocessing module is further configured to perform image pyramid processing on the picture to be detected to obtain the processed picture.

Optionally, the device further includes:

a correction module, configured to perform image color correction processing on the to-be-detected picture to obtain a corrected picture;

The preprocessing module is further configured to: perform image pyramid processing on the corrected picture to obtain the processed picture.

According to a third aspect of the embodiments of the present invention, a server is provided, characterized in that it includes:

processor;

memory; and

Computer program;

Wherein, the computer program is stored in the memory and configured to be executed by the processor, the computer program comprising instructions for performing the method according to the first aspect of the embodiments of the present invention.

According to a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and the computer program causes a server to execute the first aspect of the embodiments of the present invention Methods.

In the picture processing method, device and storage medium provided by the embodiments of the present invention, a processed picture is obtained by preprocessing the picture to be detected; further, image feature information and text feature information are respectively extracted from the processed picture; Feature information and text feature information to determine the type of pictures to be detected, where the types of pictures include normal pictures or abnormal pictures; since the pictures to be detected are preprocessed first, the preprocessed pictures are more convenient for feature extraction and picture In addition, by extracting image feature information and text feature information from the preprocessed pictures, the detection and recognition effects of the pictures to be detected can be improved, and the accuracy of picture detection can be improved.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of a picture processing method according to an exemplary embodiment of the present invention.

Fig. 2 is a flowchart of a picture processing method according to another exemplary embodiment of the present invention.

FIG. 3 is a schematic diagram of an image recognition algorithm in a picture according to a CNN network in the prior art.

Figure 4 is a schematic diagram of the structure of the local connections and shared weights of the CNN network.

Fig. 5 is a flowchart of a picture processing method according to another exemplary embodiment of the present invention.

Fig. 6 is a block diagram of a picture processing apparatus according to an exemplary embodiment of the present invention.

Fig. 7 is a block diagram of a picture processing apparatus according to another exemplary embodiment of the present invention.

FIG. 8 is a block diagram of a picture processing apparatus according to another exemplary embodiment of the present invention.

Fig. 9 is a block diagram of a picture processing apparatus according to yet another exemplary embodiment of the present invention.

FIG. 10 is a schematic structural diagram of a server according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The image processing method provided by the embodiment of the present invention can be applied to the scene of detecting the type of the image, and can especially be applied to the application scene of the e-commerce enterprise screening the images. In the prior art, when Internet e-commerce companies usually screen various pictures in a big data environment, they only perform detection and identification based on the similarity of the pictures, even if the convolutional neural network technology is used to identify the pictures, It can greatly improve the efficiency of image detection and recognition, but because the convolutional neural network technology only recognizes and detects specific image information in the picture, when faced with image variants and derived images, the convolutional neural network technology cannot be used effectively. May kill normal pictures by mistake. In addition, for pictures containing a large amount of text information, when the pictures are detected by the CNN technology, the false detection rate of the pictures is high, resulting in low picture detection accuracy.

In consideration of the above-mentioned technical problems, the present invention provides a picture processing method, by preprocessing the picture to be detected, and extracting image feature information and text feature information from the preprocessed picture, and then extracting image feature information according to the extracted image feature. Information and text feature information to determine whether the image to be detected is a normal image or an abnormal image. Since the image feature information and text feature information are extracted from the preprocessed pictures respectively, the type of the picture to be detected is jointly determined according to the image feature information and the text feature information, so that the effect of detecting and identifying the picture to be detected can be improved. Moreover, the accuracy of image detection is improved.

The technical solutions of the present invention will be described in detail below with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

FIG. 1 is a flowchart of a picture processing method according to an exemplary embodiment of the present invention,

FIG. 2 is a flowchart of a picture processing method according to another exemplary embodiment of the present invention. The method can be executed by any apparatus for executing the picture processing method, and the apparatus can be implemented by software and/or hardware. In this embodiment, the device may be integrated in the server. As shown in FIG. 1, the image processing method provided by the embodiment of the present invention includes the following steps:

Step 101: Preprocess the image to be detected to obtain a processed image.

In this step, by preprocessing the image to be detected, the preprocessed image can simplify subsequent operations of identifying and determining the type.

Among them, the picture to be detected is preprocessed, as shown in Figure 2, optionally, the picture to be detected can be preprocessed by image pyramid processing, and the pictures to be detected are arranged in a pyramid shape step by step according to the resolution of the image, In this way, the image features and text features of the pictures to be detected can be recognized and detected at multiple scales. The present invention does not impose any limitation on the level and scale of the image pyramid, and those skilled in the art can set it as required.

In this step, by performing image pyramid preprocessing on the image to be detected, the information of the image to be detected is divided into image feature information and text feature information, and the image feature information and text feature information are divided into different multi-scales, so that the This makes the preprocessed images more conducive to subsequent detection and recognition.

Optionally, before performing image pyramid processing on the image to be detected, image color correction processing may also be performed on the image to be detected to obtain a corrected image; in this way, the server will perform image pyramid processing on the corrected image to obtain a processed image. .

For example, when the image to be detected is a poster image, since there is a lot of text feature information in the poster image, and in order to highlight different information in the poster, different words may be used in different colors to achieve a striking effect. There may be multiple colors in the text of the poster. At this time, by performing image color correction processing on the poster image, the multiple colors in the poster can be compressed into a single color, and the more complex colors or tones in the poster can be discarded. It can make the color in the text more uniform. Then, the image after color correction is processed by image pyramid. When the image to be detected is processed by image pyramid, since the color of the corrected image is relatively single, the operation during image pyramid processing is simpler.

By performing image color correction processing on the image to be detected, and eliminating redundant color information, the color of the image to be detected can be made relatively single, so that when the corrected image is processed by image pyramid, the color of the image is single, so that the color of the image is single. The process of image processing is simpler, thereby improving the speed of image detection.

Step 102: Extract image feature information and text feature information from the processed picture, respectively.

In this step, after preprocessing the image to be detected, image feature information and text feature information are respectively extracted from the preprocessed image, where the image feature information may be the feature information of the image in the image to be detected, for example , the image feature information can be a person image, an animal image or a scenery image, etc. Specifically, a variety of image feature information can be preset according to needs. When extracting image feature information, detection and detection can be performed according to the preset image feature information. identification, and then extract image feature information. The text feature information can be information such as text or tables. Specifically, a variety of text feature information can be preset as needed. When extracting text feature information, detection and recognition can be performed according to the preset text feature information, and then the text Feature information extraction.

In addition, when extracting image feature information from preprocessed pictures, it can be extracted by multi-level CNN.

Let's first introduce the CNN network and how the CNN network performs image recognition and feature extraction.

CNN network is essentially a feed-forward neural network. Figure 3 is a schematic diagram of the image recognition algorithm in the picture according to the CNN network. As shown in Figure 3, the CNN network mainly consists of three convolutional layers (Convolutional layers), downsampling layers (Pooling layers) and fully connected layers (Fully connected layers). part of the structure. The structural characteristics of convolutional neural networks are mainly manifested in two aspects: first, the neurons of the CNN network are partially connected, not fully connected; second, the weights are shared between some neurons of the CNN network. The local connection attribute and weight sharing weight attribute of convolutional neural network can effectively extract image spatial features, and effectively reduce the complexity of the network model by reducing the number of weight parameters.

Using the local connection properties between neurons in adjacent layers in convolutional neural networks, CNN networks can effectively extract local image feature information in two-dimensional images. Figure 4 is a schematic diagram of the structure of the local connections and shared weights of the CNN network. As shown in Figure 4, in the CNN network, the connections between some neurons are copied to the entire current layer, in other words, between neurons Shared connection weights and biases. Using the special structure of local connections and shared weights of CNN network, CNN network has better generalization ability and performance advantages in image processing.

Typically, a convolutional neural network consists of a convolutional layer and a downsampling layer, while a deep convolutional neural network (DCNN) consists of several convolutional layers and downsampling layers stacked, which can achieve Deep network structure. Taking the traditional CNN network as an example, each convolutional layer C _l , weights the input data of the previous layer or the output feature data of the down-sampling layer S _l-1 with a learnable convolution kernel through convolution and weighting. and obtained, it can be seen that a feature map (Feature map) of the _{convolutional} layer C1 can be obtained by the following formula:

表示卷积层C_l的第j个特征图中位置(x,y)处的神经元的特征值； m表示与S_l-1层中当前卷积成第j个特征图相连接的特征图；

表示与当前神经元与S_l-1层第m个特征图卷积核在位置(p,q)上的权重值；P_l和Q_l分别表示二维卷积核的在两个方向上的尺寸；b_lj表示卷积层C_l的第j个特征的偏置量。g(·) 表示激活函数，一般来说可使用sigmoid或tanh函数，分别可表示为：in,

Represents the feature value of the neuron at the position (x, y) in the jth feature map of the convolutional layer C _l ; m represents the feature map connected to the jth feature map currently convolved in the S _l-1 layer ;

Represents the weight value of the current neuron and the m-th feature map convolution kernel of the S _l-1 layer at the position (p, q); P _l and Q _l respectively represent the two-dimensional convolution kernel in two directions. size; b _lj represents the bias of the jth feature of the convolutional layer C _l . g( ) represents the activation function. Generally speaking, the sigmoid or tanh function can be used, which can be expressed as:

Each feature map output by the convolutional layer is the result of the weighted summation of the feature map of the previous layer corresponding to the layer and different convolution kernels.

The down-sampling layer is another important part of the convolutional neural network. By down-sampling the feature map obtained by the convolution layer, the scale invariance of the feature information can be achieved. Each downsampling layer corresponds to a convolutional layer, and the neurons in the downsampling layer implement downsampling on the convolutional layer through the downsampling function. The most commonly used downsampling strategy in CNN networks is downsampling based on max pooling, which can generally be expressed as:

Among them, u(n,1) is the window function for the convolutional layer, and a _j represents the maximum value of the neighborhood.

The CNN network contains a fully connected layer, and generally uses Logistic Regression (LR) as the classifier for the final feature information output. LR is a probability statistics-based classifier that uses probability scores to calculate the correlation between categorical variables and input variables as a predicted value output. In general deep CNN networks, the multi-classification task is implemented using Softmax regression as the LR classifier of the final output layer. The output of the Softmax regression algorithm sums the predicted values to 1, so the output can be regarded as a set of conditional probabilities. For an input feature i, the probability that the input feature belongs to class i is:

Among them, W and b represent the weight and bias parameters of the LR layer, respectively, Wi and b _i represent the weight and bias parameters corresponding to class _i , respectively, Y is the class result output by the LR classifier, and s represents the Softmax function. The output dimension of the LR layer is the number of categories that the LR classifier needs to recognize. Taking the LR layer as the top layer of the deep convolutional neural network, the input feature V is the output feature vector of the convolutional network of the last layer of the deep CNN network.

The type of the picture is determined according to the image feature information obtained by the fully connected layer in the above CNN network.

In this step, on the basis of the existing technology, the CNN network is divided into multi-level tasks to extract the image feature information in the picture. Feature information extraction.

Specifically, continuing to refer to FIG. 2 , the multi-level CNN network includes three-level networks, namely, a first-level network, a second-level network, and a third-level network, so as to realize the step-by-step extraction of image feature information. First, the processed picture is input into the first-level CNN network, and the first mapping image corresponding to the processed picture and the coordinates of the first region frame of the target image in the processed picture are obtained; specifically, the first-level CNN The network can be a P network (Proposal Network), and the P network is a fully convolutional neural network (FCNN) including 5 convolutional layers and 4 pooling layers and a fully connected layer. The selection of the size of the convolution kernel needs to be set according to the size of the image features containing the sensitive information to be detected, which can reduce the influence of image distortion or stretching on the image feature extraction. The first-level CNN network can roughly extract the image feature information in the picture globally, and output the first mapping image. The first mapping image is the required image extracted from the image feature information in the picture according to the first-level CNN network. The first-level CNN network generates the first area frame of the target image under different aspect ratios according to the first mapping image, the first mapping image is in the first area frame, and outputs the coordinates of the first area frame of the target image at the same time. The coordinates can be the coordinates (a, b) and (c, d) of the two diagonal corners of the rectangular first area frame, and the coordinates of the first area frame of the first mapped image and the target image are used as the input of the next-level CNN network data.

Further, the first mapping image and the coordinates of the first region frame are input into the second-level CNN network to obtain the second mapping image corresponding to the processed picture and the second region box coordinates of the target image in the processed picture. Specifically, the second-level CNN network is an R network (Refine Network), and the R network is an FCNN network including 4 convolutional layers and 3 pooling layers. The first mapping image and the coordinates of the first region frame are input to the second In the high-level CNN network, the image feature information is further extracted from the input data, that is, the first mapped image is more accurately identified, the second mapped image is obtained through the accurate identification of the R network, and the target image in the picture is obtained at the same time. The coordinates of the second area frame of The coordinates are used as the input data of the next-level CNN network. The area of the rectangular second area frame output by the R network is smaller than the area of the rectangular second area frame output by the P network. The extraction of image features by the R network can not only make the image features The information is more accurate than the image feature information extracted by the P network, and it can also reduce the error rate of the image feature information output by the P network.

Input the second map image and the coordinates of the second area frame into the third-level CNN network to obtain image feature information. Specifically, the third-level CNN network can be an O network (Output Network), and the O network includes 4 convolutional layers. and 3 pooling layers, input the second map image and the coordinates of the second area frame into the third-level CNN network, and further extract the image feature information by extracting the input data, that is, the second map image is more accurate. Identify, and then determine whether the picture to be detected is a picture containing the required image feature information, and implement multi-level task union in the O network to detect the target image in the second area frame, and return to the area of the rectangular target image frame, the diagonal coordinates (x, y) and (m, n) of the rectangular area frame are precisely determined, and the extraction of image feature information through the O network can not only obtain the recognition result of the image to be detected, but also obtain the specific image feature information. The coordinate position can also reduce the error rate of the image feature information output by the R network.

The multi-level CNN network extracts the image feature information, and finally outputs the 6-dimensional feature information of the image detection. The 6-dimensional feature information includes the recognition result and the precise position coordinates of the image feature. The recognition result can be expressed as (1,0) or (0, 1), respectively representing that the recognition result of the image to be detected through the image feature information is a normal image or an abnormal image.

In this step, based on the extraction of image feature information in the prior art, the CNN network is divided into multi-level networks, and then the image feature information in the picture is extracted step by step, so that the extracted image feature information It is more accurate and can greatly improve the rate of image feature extraction.

In addition, to extract text feature information from the processed pictures, the text feature information can be extracted from the processed pictures through a multi-level convolutional neural network CNN and a Recurrent Neural Network (RNN). Specifically, the sensitive text information is extracted through multi-level CNN network and RNN network. Among them, the CNN network adopts the DenseNet network model structure to realize text feature detection, and the RNN network realizes the recognition of text information through a double-layer bidirectional RNN network. Then output the result of text feature detection and recognition, which is the high-dimensional feature information of various sensitive words, and the sensitive words are pre-stored text information, such as: "brush orders", "plant vegetables", "integrity", "brush" ” or “good reviews” can be defined as sensitive words.

In this step, two neural networks are included, namely, a multi-level CNN network and a DenseNet+RNN network. The multi-level CNN network is used to extract image feature information, and the DenseNet+RNN network is used to extract text feature information. The training of the neural network uses Labeled dataset and Back-propagation (BP) implementation. The BP algorithm is the core basic algorithm of all neural network training. For the training of the neural network, the error metric needs to be defined by defining a suitable loss function (Costfunction). A training strategy where the number of training sessions gradually decreases the learning rate. For general deep learning with a large amount of data, a batch gradient descent (mini-batch) strategy is generally used to learn the CNN network. For a mini-batch of input, the loss function can be defined as:

Among them, m is the batch gradient descent size, and x _i and z _i represent the output label and class label of the ith data of the batch data, respectively.

In this step, through the multi-level convolutional neural network CNN, the image feature information is extracted from the processed image, and the recognition result of the image in the image to be detected and a 6-dimensional image feature information of the image are obtained. The neural network CNN and the cyclic neural network RNN extract the text feature information from the processed pictures to obtain the high-dimensional feature information of the text. At the same time, the identification and detection of images and texts in pictures can greatly improve the problem that the CNN network algorithm in the prior art cannot effectively identify variants of images and derived images in pictures. In addition, text feature information and image feature information Simultaneous extraction and identification can effectively avoid the detection and identification of derived images by the CNN network algorithm in the prior art, thus providing a more comprehensive basis for the determination of the image type, and the determined image type is more accurate.

Step 103: Determine the type of the picture to be detected according to the image feature information and the text feature information, where the type of the picture includes a normal picture or an abnormal picture.

In this step, the image feature information and text feature information obtained above are used to determine the type of the picture to be detected according to the image feature information and the text feature information, wherein the type of the picture includes a normal picture or an abnormal picture, and the normal picture is all The required pictures and abnormal pictures are defined relative to normal pictures. Of course, for different users, the definition of normal pictures is not the same, that is, picture G is a normal user for user A, but for user B It is an abnormal picture. Therefore, the setting of the image feature and the text feature of the picture needs to be set according to the needs of the user, and the present invention does not make any limitation.

In the image processing method provided by the embodiment of the present invention, a processed image is obtained by preprocessing a to-be-detected image; further, image feature information and text feature information are respectively extracted from the processed image; and according to the image feature information and text feature information to determine the type of the picture to be detected, wherein the type of the picture includes a normal picture or an abnormal picture; since the picture to be detected is preprocessed through the line, the preprocessed picture is more convenient for feature extraction and picture type determination. , because the image feature information and the text feature information are extracted respectively for the preprocessed picture, the detection and recognition effect of the picture to be detected can be improved, and the accuracy of the picture detection can be improved.

FIG. 5 is a flowchart of a picture processing method according to another exemplary embodiment of the present invention. On the basis of FIG. 1 , on the process of how to determine the type of the picture to be detected according to the image feature information and the text feature information, Describing in detail, as shown in FIG. 5 , the method of this embodiment may include:

Step 501: Preprocess the image to be detected to obtain a processed image.

Step 502: Extract image feature information and text feature information from the processed picture, respectively.

Steps 501 to 502 are similar to steps 101 to 102, and are not repeated here.

Step 503: Perform splicing processing on the image feature information and the text feature information to obtain splicing feature information.

In this step, the splicing feature information is obtained by splicing the extracted image feature information and text feature information. Specifically, the 6-dimensional feature information of the image and the high-dimensional feature information of the text can be spliced to obtain the spliced feature. information, as shown in Figure 2. For example, the feature information of splicing can be (1, 0, x, y, m, n)+(1, 0, 0, 1...), where 0 and 1 in the second item represent the corresponding sensitive words (eg: brush Single, growing vegetables, integrity, brushing, praise...) detection and recognition results.

In step 504, the dimensionality reduction processing of the manifold data is performed on the splicing feature information to obtain the embedded feature of the splicing feature information.

In this step, after the splicing feature information is obtained, the splicing feature information is subjected to dimensionality reduction processing of the manifold data to obtain the embedded feature of the splicing feature information, wherein the dimensionality reduction processing of the manifold data is to use a popular learning algorithm to reduce the splicing feature. The information is used to reduce the dimensionality of the data. Specifically, the popular learning (Manifold embedding, ME) algorithm uses the sample points and their similar nearest neighbor samples and heterogeneous nearest neighbor samples to construct a local sample block, and uses the clustering idea on the local block to achieve Dimensionality reduction of data. And the transfer learning method is introduced to maintain the data structure characteristics. The optimal transformation matrix W ^* of the ME algorithm can be expressed as:

Among them, X is the matrix formed by the training sample set, and P is the principal component analysis transformation matrix. The second term is the transfer regular term, which is used to maintain the main features of the original data structure after feature extraction and dimension reduction, so that the discriminant information learned from the training samples can be effectively transferred to the test data.

In the data dimensionality reduction of the local block, the ME algorithm expects the sample points to be clustered as much as possible in the low-dimensional space with other similar samples, and separated from the heterogeneous samples in the low-dimensional space as much as possible, and the linear operation is used to complete the data block. local discrimination. According to the dimensionality reduction criterion, the local optimized expression matrix is obtained. On the basis of local optimization, the global permutation method is used, assuming that for each local block, its low-dimensional representation is a local choice of global low-dimensional coordinates. From this, a selection matrix is defined, and a low-dimensional local block is represented by the global low-dimensional coordinates and the selection matrix. A global arrangement is obtained by superimposing low-dimensional local patches all represented by global low-dimensional coordinates and selection matrices. The global optimized expression matrix is calculated iteratively, and the projection transformation matrix is calculated using standard eigenvalue decomposition or generalized eigenvalue decomposition methods. In this way, dimensionality reduction based on the data of sample category information is realized to realize feature extraction.

In this step, information interaction and data dimension reduction are performed on image feature information and text feature information by using manifold learning algorithm and transfer learning method, and the correlation between image feature information and text feature information is mined. The effective information in the feature information correlation is used to identify the picture. In addition, by performing dimensionality reduction processing of manifold data on the splicing information, the embedded feature of the splicing feature information can be improved to be more accurate.

Step 505: Determine the type of the picture to be detected according to the embedded feature of the splicing feature information and the predetermined hypersphere.

Optionally, before determining the type of the image to be detected according to the embedded features of the splicing feature information and the pre-determined hypersphere, the hypersphere may be determined in the following manner: acquiring multiple sample images; extracting each of the multiple sample images respectively; The mapping feature of the sample image; the data description of each mapping feature is performed to obtain a hypersphere.

Specifically, by first obtaining multiple pictures, and extracting the mapping features of each sample picture in the multiple sample pictures from the obtained pictures, the sample pictures are normal pictures or abnormal pictures, that is, the required image feature information and text are included. Feature information, the mapping feature of the picture may include image feature information and text feature information of the abnormal picture. Further, the data description is performed on the mapping features of each image. Specifically, the data description for each mapping feature can be performed through the support vector data description SVDD algorithm, and the hypersphere is obtained according to the result of the data description.

The above-mentioned hypersphere is a boundary of a closed decision to describe data defined according to the Support Vector Data Description (SVDD). Specifically, the hypersphere is defined by the center a and the radius R>0 of the The sample data should be included and the hypersphere volume should be as small as possible. Therefore, define the loss function:

F(R,a)=R ² +C∑ _i ξ _i

Among them, ξ _i >0 is the slack variable. Further, similar to the support vector machine, the Lagrangian operator α _i > 0 is introduced to equivalently transform the above optimization problem into:

where (·) represents the inner product operation. From this, the Lagrangian operator α can be solved, the center of the hypersphere is determined as a=∑ _i α _i x _i and the radius R of the hypersphere can be determined by the Lagrangian operator to satisfy the support vector and hypersphere of 0<α _i <C Heart a sure. For any data to determine whether it belongs to the current data category, it is only necessary to determine whether the current data is contained in the current hypersphere, that is, whether the distance between the data and the hypersphere is less than or equal to the hypersphere radius.

For the support vector data description containing negative samples, subscripts i and j represent abnormal image data points, l and m represent normal image data points, and the loss function is defined as:

F(R,a)=R ² +C ₁ ∑ _i ξ _i +C ₂ ∑ _l ξ _l

Among them, ξ _i > 0 and ξ _l > 0 are the slack variables of abnormal picture data points and normal picture data points, respectively. Similarly, introducing the Lagrangian operator, the equivalent optimization problem is:

The center of the hypersphere can be expressed as a=∑ _i α _i x _i -∑ _l α _l x _l , K(,) represents the sum function, and the radial basis sum function (RBF) is used in the technical solution of the present invention. The kernel function method is introduced into SVDD, which maps the data to a possibly higher dimensional feature space by using the kernel function instead of the inner product operation. Through the introduction of kernel function, the support vector data description algorithm can describe data more flexibly and effectively.

In this step, the abnormal picture feature information is described by using the SVDD data description method, which can be used as a picture data classifier. Compared with the traditional Support Vector Machine (SVM) classifier or the linear classification described by the fully connected network The SVDD data description uses a closed hypersphere in the feature space to describe and classify image data, which can greatly improve the classification accuracy and robustness of the algorithm, and improve the manslaughter problem of normal images.

After the hypersphere is determined, for the image to be detected, the type of the image to be detected will be determined according to the mapping feature and the hypersphere determined by pre-training, which may include: judging whether the mapping feature is within the hypersphere; if the mapping feature is within the hypersphere If the mapping feature is not within the hypersphere, it is determined that the to-be-detected picture is a normal picture.

Specifically, according to a predetermined hypersphere, it is determined whether the mapping feature of the picture is within the hypersphere. For example, the mapping feature is C. If the mapping feature C is not within the predetermined hypersphere, it is determined that the picture to be detected is a normal picture. If the mapping feature C is within the hypersphere, it is determined that the picture to be detected is an abnormal picture. Since the mapping feature of the picture to be detected is determined by the predetermined hypersphere, the time spent on determining the type of the picture to be detected can be reduced, that is, the rate of determining the type of the picture is increased.

The image processing method provided in this embodiment obtains the splicing feature information by splicing the image feature information and the text feature information, and then performs dimensionality reduction processing of the line manifold data on the splicing feature information to obtain the embedded feature of the splicing feature information, According to the manifold dimension reduction and the mapping features described by the data and the pre-determined hypersphere, the type of the image to be detected is determined; because the dimensionality reduction of the manifold data is performed on the splicing feature information, the data reduction of the sample category information can be realized. It can greatly improve the classification accuracy and robustness of the algorithm, improve the manslaughter problem of normal pictures, and reduce the number of pictures to be detected. The time spent on determining the type of the picture, that is, the rate of determining the type of the picture is increased.

FIG. 6 is a flowchart of a picture processing apparatus according to an exemplary embodiment of the present invention. As shown in FIG. 6 , the image processing apparatus may include: a preprocessing module 11 , a first extraction module 12 and a determination module 13 .

The preprocessing module 11 is used for preprocessing the image to be detected to obtain the processed image;

The first extraction module 12 is used to extract image feature information and text feature information from the processed picture respectively;

The determining module 13 is configured to determine the type of the picture to be detected according to the image feature information and the text feature information, wherein the type of the picture includes a normal picture or an abnormal picture.

In the picture processing device provided by the embodiment of the present invention, the preprocessing module 11 obtains a processed picture by preprocessing the picture to be detected; the first extraction module 12 extracts image feature information and text feature information respectively from the processed picture; The determining module 13 determines the type of the picture to be detected according to the image feature information and the text feature information, wherein, the type of the picture includes a normal picture or an abnormal picture; because the pictures to be detected are preprocessed by lines, the preprocessed pictures are more convenient. Perform feature extraction and image type determination. In addition, by extracting image feature information and text feature information for the preprocessed images, the detection and recognition effects of the images to be detected can be improved, and the accuracy of image detection can be improved. .

FIG. 7 is a flowchart of a picture processing apparatus according to another exemplary embodiment of the present invention. On the basis of FIG. 6 , the determining module 13 includes: a splicing sub-module 131, a dimensionality reduction processing sub-module 132 and a determining sub-module 133.

The splicing sub-module 131 is used for splicing the image feature information and the text feature information to obtain the splicing feature information.

The dimensionality reduction processing sub-module 132 is configured to perform dimensionality reduction processing of the manifold data on the splicing feature information to obtain the embedded feature of the splicing feature information.

The determination sub-module 133 is configured to determine the type of the picture to be detected according to the embedded feature of the splicing feature information and the predetermined hypersphere.

Optionally, determine the submodule 133, which is specifically used for:

Determine whether the embedded feature is in the hypersphere;

If the embedded feature is in the hypersphere, it is determined that the image to be detected is an abnormal image;

If the embedded feature is not within the hypersphere, it is determined that the image to be detected is a normal image.

In the image processing device provided in this embodiment, the splicing sub-module 131 obtains splicing feature information by splicing image feature information and text feature information, and the dimensionality reduction processing sub-module 132 performs dimensionality reduction processing of line manifold data on the splicing feature information , obtain the embedded feature of the splicing feature information, and determine the sub-module 133 to determine the type of the image to be detected according to the embedded feature of the splicing feature information and the predetermined hypersphere; It can realize dimensionality reduction with the data of sample category information to realize feature extraction, and judge the type of the picture to be detected according to the pre-determined hypersphere, which can greatly improve the classification accuracy and robustness of the algorithm, and improve the quality of normal pictures. The problem of manslaughter can be reduced, and the time spent on determining the type of the image to be detected can be reduced, that is, the rate of determining the image type can be improved.

Optionally, the image processing apparatus further includes: an acquisition module 14 , a second extraction module 15 and a data description module 16 , as shown in FIG. 8 .

an acquisition module 14, configured to acquire a plurality of sample pictures;

The second extraction module 15 is used to extract the mapping feature of each sample picture in the plurality of sample pictures respectively;

The data description module 16 is used for performing data description on each mapping feature to obtain a hypersphere.

Optionally, the data description module 16 is also used for:

The data description of each mapping feature is carried out through the support vector data description SVDD algorithm, and the hypersphere is obtained.

Optionally, the first extraction module 12 is configured to extract image feature information from the processed image through a multi-level convolutional neural network CNN.

Optionally, the first extraction module 12 is also used for:

Input the processed picture into the first-level CNN network, and obtain the first map image corresponding to the processed picture and the coordinates of the first region frame of the target image in the processed picture;

Input the first mapping image and the coordinates of the first region frame into the second-level CNN network to obtain the second mapping image corresponding to the processed picture and the second region frame coordinates of the target image in the processed picture;

The second map image and the coordinates of the second region frame are input into the third-level CNN network to obtain image feature information.

Optionally, the first extraction module 12 is further configured to extract text feature information from the processed pictures through a multi-level convolutional neural network CNN and a recurrent neural network RNN.

Optionally, the preprocessing module 11 is further configured to perform image pyramid processing on the picture to be detected to obtain the processed picture.

Optionally, the image processing apparatus further includes: a correction module 17 , as shown in FIG. 9 .

The correction module 17 is used to perform image color correction processing on the picture to be detected to obtain a corrected picture;

The preprocessing module 11 is further configured to perform image pyramid processing on the corrected picture to obtain the processed picture.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 10 is a schematic structural diagram of a server according to an embodiment of the present invention. The server shown in FIG. 10 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 10 , the server may include a transmitter 60 , a processor 61 , a memory 62 and at least one communication bus 63 . The communication bus 63 is used to realize the communication connection between the elements. The memory 62 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory. Various programs may be stored in the memory 62 for performing various processing functions and implementing the method steps of this embodiment. In addition, the server may further include a receiver 64. The receiver 64 in this embodiment may be a corresponding input interface with a communication function and a function of receiving information, and the transmitter 60 in this embodiment may be a corresponding input interface with a communication function and a function of receiving information. Output interface for sending information function. Optionally, the transmitter 60 and the receiver 64 may be integrated into one communication interface, or may be two independent communication interfaces respectively.

In addition, the memory 62 has stored in the memory 62 and is configured to be executed by the processor 61, the computer program comprising instructions for performing the methods of the embodiments shown in FIGS. 1 and 5 above or performing the methods shown in FIGS. Instructions for the method of the illustrated embodiment.

Embodiments of the present invention further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program causes the server to execute the image processing methods provided by the embodiments shown in FIG. 1 and FIG. 5 . Wherein, the above-mentioned readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM) ), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments may be completed by program instructions related to hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps including the above method embodiments are executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. scope.

Claims (22)

1. An image processing method, comprising:

preprocessing a picture to be detected to obtain a processed picture;

respectively extracting image characteristic information and text characteristic information from the processed picture;

and determining the type of the picture to be detected according to the image characteristic information and the text characteristic information, wherein the type of the picture comprises a normal picture or an abnormal picture.

2. The method according to claim 1, wherein the determining the type of the picture to be detected according to the image feature information and the text feature information comprises:

splicing the image characteristic information and the text characteristic information to obtain spliced characteristic information;

performing dimensionality reduction processing on the manifold data on the splicing characteristic information to obtain embedded characteristics of the splicing characteristic information;

and determining the type of the picture to be detected according to the embedded characteristic of the splicing characteristic information and a predetermined hypersphere.

3. The method according to claim 2, wherein the determining the type of the picture to be detected according to the embedded feature of the splicing feature information and a predetermined hyper-sphere comprises:

determining whether the embedded feature is within the hypersphere;

if the embedded features are in the hypersphere, determining that the picture to be detected is an abnormal picture;

and if the embedded feature is not in the hypersphere, determining that the picture to be detected is a normal picture.

4. The method according to claim 2 or 3, wherein before determining the type of the picture to be detected according to the embedded feature of the splicing feature information and a predetermined hyper-sphere, the method further comprises:

acquiring a plurality of sample pictures;

respectively extracting mapping characteristics of each sample picture in the plurality of sample pictures;

and carrying out data description on each mapping characteristic to obtain the hypersphere.

5. The method of claim 4, wherein said data describing each of said mapped features to obtain said hypersphere comprises:

and carrying out data description on each mapping characteristic through a Support Vector Data Description (SVDD) algorithm to obtain the hypersphere.

6. The method of claim 1, wherein extracting image feature information from the processed picture comprises:

and extracting image characteristic information from the processed picture through a multi-stage Convolutional Neural Network (CNN).

7. The method according to claim 6, wherein the extracting image feature information from the processed picture through a multi-stage Convolutional Neural Network (CNN) comprises:

inputting the processed picture into a first-level CNN network to obtain a first mapping image corresponding to the processed picture and a first area frame coordinate of a target image in the processed picture;

inputting the first mapping image and the first area frame coordinate into a second-level CNN network to obtain a second mapping image corresponding to the processed picture and a second area frame coordinate of a target image in the processed picture;

and inputting the second mapping image and the second area frame coordinate into a third-level CNN network to obtain the image characteristic information.

8. The method of claim 1, wherein extracting text feature information from the processed picture comprises:

and extracting the text characteristic information from the processed picture through a multi-stage Convolutional Neural Network (CNN) and a cyclic neural network (RNN).

9. The method according to any one of claims 1 to 8, wherein the pre-processing the picture to be detected to obtain a processed picture comprises:

and carrying out image pyramid processing on the picture to be detected to obtain the processed picture.

10. The method according to claim 9, wherein before the image pyramid processing is performed on the picture to be detected and the processed picture is obtained, the method further comprises:

carrying out image color correction processing on the picture to be detected to obtain a corrected picture;

the image pyramid processing is performed on the picture to be detected to obtain the processed picture, and the image pyramid processing comprises the following steps:

and carrying out image pyramid processing on the corrected picture to obtain the processed picture.

11. A picture processing apparatus, comprising:

the preprocessing module is used for preprocessing the picture to be detected to obtain a processed picture;

the first extraction module is used for respectively extracting image characteristic information and text characteristic information from the processed picture;

and the determining module is used for determining the type of the picture to be detected according to the image characteristic information and the text characteristic information, wherein the type of the picture comprises a normal picture or an abnormal picture.

12. The apparatus of claim 11, wherein the determining module comprises:

the splicing submodule is used for splicing the image characteristic information and the text characteristic information to obtain splicing characteristic information;

the dimension reduction processing submodule is used for carrying out dimension reduction processing on the manifold data on the splicing characteristic information to obtain the embedded characteristic of the splicing characteristic information;

and the determining submodule is used for determining the type of the picture to be detected according to the embedded characteristic of the splicing characteristic information and a predetermined hyper-sphere.

13. The apparatus according to claim 12, wherein the determination submodule is specifically configured to:

determining whether the embedded feature is within the hypersphere;

if the embedded features are in the hypersphere, determining that the picture to be detected is an abnormal picture;

and if the embedded feature is not in the hypersphere, determining that the picture to be detected is a normal picture.

14. The apparatus of claim 12 or 13, further comprising:

the acquisition module is used for acquiring a plurality of sample pictures;

the second extraction module is used for respectively extracting the mapping characteristics of each sample picture in the plurality of sample pictures;

and the data description module is used for carrying out data description on each mapping characteristic to obtain the hypersphere.

15. The apparatus of claim 14, wherein the data description module is specifically configured to:

and carrying out data description on each mapping characteristic through a Support Vector Data Description (SVDD) algorithm to obtain the hypersphere.

16. The apparatus of claim 11, wherein the first extraction module is further configured to extract image feature information from the processed picture through a multi-stage Convolutional Neural Network (CNN).

17. The apparatus of claim 16, wherein the first extraction module is further configured to:

inputting the processed picture into a first-level CNN network to obtain a first mapping image corresponding to the processed picture and a first area frame coordinate of a target image in the processed picture;

inputting the first mapping image and the first area frame coordinate into a second-level CNN network to obtain a second mapping image corresponding to the processed picture and a second area frame coordinate of a target image in the processed picture;

and inputting the second mapping image and the second area frame coordinate into a third-level CNN network to obtain the image characteristic information.

18. The apparatus of claim 11, wherein the first extraction module is further configured to extract the text feature information from the processed picture through a multi-stage Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN).

19. The apparatus according to any one of claims 11 to 18, wherein the preprocessing module is further configured to perform image pyramid processing on the picture to be detected to obtain the processed picture.

20. The apparatus of claim 19, further comprising:

the correcting module is used for carrying out image color correction processing on the picture to be detected to obtain a corrected picture;

the preprocessing module is further used for carrying out image pyramid processing on the corrected picture to obtain the processed picture.

21. A server, comprising:

a processor;

a memory; and

a computer program;

wherein the computer program is stored in the memory and configured to be executed by the processor, the computer program comprising instructions for performing the method of any of claims 1-10.

22. A computer-readable storage medium, characterized in that it stores a computer program that causes a server to execute the method of any one of claims 1-10.

Images