CN114627289B - Industrial part instance segmentation method based on voting mechanism - Google Patents

Abstract

The present invention relates to an industrial part instance segmentation method based on a voting mechanism. The steps are: select three image segmentation methods, namely Graph Cut, GrabCut and OneCut; input the industrial part picture to be segmented and initialize the initial parameters of the three image segmentation models of Graph Cut, GrabCut and OneCut respectively; define the energy function according to the grayscale histogram of the image, the RGB three-channel Gaussian mixture model and the L1-norm of the foreground and background of the whole image; minimize the energy function and output three segmentation binary images; based on the three segmentation binary images, introduce a voting mechanism, vote for each pixel of the three images, and output the final segmentation image. The method of the present invention can realize accurate and stable segmentation of parts in different industrial environments, with high precision and strong robustness; moreover, the present invention can be directly deployed on an ordinary PC without consuming GPU resources, and is simple to implement and highly practical.

Description

Industrial parts instance segmentation method based on voting mechanism

Technical Field

The present invention relates to the technical field of machine learning and computer vision, and in particular to an industrial part instance segmentation method based on a voting mechanism.

Background technique

China's industry is transforming towards automation and intelligence, and intelligent industrial robots with vision are gradually becoming the main direction of future development for manufacturing companies. In many industrial robot application scenarios, accurate segmentation of industrial parts is a necessary prerequisite for intelligent visual robots to achieve various industrial tasks. Therefore, realizing a high-precision industrial parts segmentation method is of great significance to the development of industrial automation and intelligence.

At present, traditional image segmentation algorithms do not have the characteristics of high accuracy and strong robustness for industrial parts segmentation, and are easily affected by the background environment, which makes the segmentation effect of the algorithm vary greatly under different circumstances. In recent years, deep learning related algorithms have been widely studied. Such algorithms have high segmentation accuracy and strong robustness, but the training of neural networks consumes a lot of computing resources and has high requirements for hardware resources, which increases the training cost and economic cost.

Summary of the invention

In view of the shortcomings of the prior art, the present invention implements an industrial parts instance segmentation method based on a voting mechanism. The method is based on graph cut theory and introduces a voting mechanism to transform three weak segmenters into a strong segmenter to achieve accurate segmentation of industrial parts. Compared with traditional segmentation algorithms, it has higher segmentation effect and stronger robustness, and compared with deep learning related algorithms, it has higher practicality.

The technical solution adopted by the present invention to achieve the above-mentioned purpose is: an industrial part instance segmentation method based on a voting mechanism, comprising the following steps:

Collect original images of industrial parts;

Based on the image segmentation methods Graph Cut, GrabCut and OneCut, a strong segmenter is constructed to process the original image, identify the foreground and background areas and output three binary images;

Based on the voting mechanism, pixel-by-pixel fusion optimization is performed to produce mask images;

Perform mask operation on the original image to obtain the industrial parts segmentation map.

The step of constructing a strong segmenter based on the image segmentation methods Graph Cut, GrabCut and OneCut comprises:

1) Define energy function: define Graph Cut energy function E(L) based on the grayscale histogram of the image, define GrabCut energy function E(α, k, θ, z) based on the image RGB Gaussian mixture model GMM, and define OneCut energy function E(S) based on the L1-norm of the foreground and background of the whole image, which is used to measure the similarity of the foreground area, background area and the junction of the foreground and background in the image, so as to achieve image segmentation;

2) Initialize the model parameters of the three image segmentation methods, and input the pixel features of the target foreground or background in the original image into the three energy functions respectively;

3) Execute the calculation process of minimizing the energy function to separate the foreground and background;

4) Output the corresponding three segmentation binary images.

The three energy functions are defined as follows:

Wherein, L is the category label of the image pixel, which can be expressed as L={ _l1 , _l2 ,..., _ln }, l _i ∈{0,1}, 0 indicates that the current pixel belongs to the background, 1 indicates that the current pixel belongs to the foreground, R(L) is the region term, B(L) is the boundary term, a∈[0,1] is the influence factor, indicating the degree of influence on the energy function; α∈{0,1}, indicating whether it is the GMM component of the background model or the GMM component of the foreground model, k is the GMM Gaussian component parameter, k∈{1,2,...,K}; θ is the GMM parameter corresponding to each pixel; U(α,k,θ,z) represents the region term, V(α,z) represents the boundary term, z is the value of the pixel in the image; β, λ are adjustable parameters, S represents the set of foreground points, θ ^S represents the color histogram of the foreground points, Represents the color histogram of background points,/> Indicates a region item, /> is a boundary term.

The pixel-by-pixel fusion optimization based on the voting mechanism is to vote for the pixels at the same position of three segmented binary images of the same size to determine the value of the pixel.

The mask operation is an operation of extracting the target area in the original image to be segmented according to the binary image.

The method also uses the evaluation index MIoU to evaluate the output segmented image and the standard image;

Wherein, N is the number of test samples, A and B are the target areas in the segmented image obtained according to this method and the standard image produced, the standard image is annotated pixel by pixel in the original image to distinguish the current pixel as the foreground or background, A∩B is the intersection of the two areas, and A∪B is the union of the two areas.

The present invention has the following beneficial effects and advantages:

1. The industrial part instance segmentation method based on the voting mechanism described in the present invention can effectively realize the accurate segmentation of industrial parts in different scenarios and has strong robustness.

2. The industrial part instance segmentation method based on the voting mechanism described in the present invention can achieve high-precision segmentation, good segmentation effect and small relative error.

3. The industrial part instance segmentation method based on the voting mechanism described in the present invention can be deployed on an ordinary PC, does not require GPU resources, saves computing resources, is simple to implement, and has higher practicality.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flow chart of the method of the present invention;

Figure 2 is a schematic diagram of the voting mechanism of the present invention. (a) is the area occupied by the target in the original image, (b), (c), and (d) are the segmentation maps output by the three methods respectively, and (e) is the final output segmentation map.

FIG3 is a binary image obtained by segmenting the method of the present invention and its comparative method in 11 different industrial environments; the first column is the input image, the second column is the manually labeled standard image, and the third column is the binary segmentation image output by the method of the present invention.

Figure 4 shows the final segmentation results obtained by the method of the present invention and its comparative method in 11 different industrial environments. The first line is the input image, the second line is the segmentation map converted from the standard image, and the third line is the final segmentation effect map output by the method of the present invention.

Detailed ways

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

As shown in Figure 1, the method of the present invention mainly comprises the following steps:

Step 1: Given an image to be segmented, use Labelme software to create a label file for the image and convert it into a binary segmentation map as a standard image. The label file defines the location information of the foreground in the image and is used to generate a binary standard image with a white foreground and a black background. After the final segmentation map is obtained using the method proposed in the present invention, the evaluation index MIoU needs to be calculated with the standard image.

Step 2: Based on graph segmentation theory, three graph segmentation image segmentation methods Graph Cut, GrabCut and OneCut are selected.

Define the energy functions of Graph Cut, GrabCut and OneCut. Including: define the Graph Cut energy function E(L) based on the grayscale histogram of the image; define the GrabCu energy function E(α, k, θ, z) based on the RGB three-channel Gaussian mixture model of the image; define the OneCut energy function E(S) based on the L1-norm of the global foreground and background of the image. The three energy functions are defined as follows:

Where L is the category label of the image pixel, which can be expressed as L = {l ₁ ,l ₂ ,...,l _n }, l _i ∈ {0,1}, 0 means the current pixel belongs to the background, 1 means the current pixel belongs to the foreground. R(L) is the region term, B(L) is the boundary term, a∈[0,1] is the influence factor, indicating the degree of influence on the energy function;

α∈{0,1}, indicating whether it is the GMM component of the background model or the GMM component of the foreground model, k is the GMM Gaussian component parameter, k∈{1,2,...,K}; θ is the GMM parameter corresponding to each pixel; U(α,k,θ,z) represents the region term, V(α,z) represents the boundary term, and z is the value of the pixel in the image;

β, λ are adjustable parameters, S represents the set of foreground points, θ ^S represents the color histogram of the foreground points, Represents the color histogram of background points,/> Indicates a region item, /> is a boundary term.

The grayscale histogram is obtained by counting the frequency of occurrence of all pixels in a digital image according to the size of their grayscale values.

The RGB Gaussian mixture model is a Gaussian mixture model constructed based on the RGB values of image pixels. The Gaussian mixture model uses the Gaussian probability density function (normal distribution curve) to accurately quantify things. It is a model that decomposes things into several Gaussian probability density functions (normal distribution curves).

L1-norm refers to the absolute difference of pixel values in an image and is defined as follows:

Among them, x _i , x _j represent pixel values, (i, j) represents a pixel pair, and n is the number of pixel pairs.

Step 3: Initialize the parameters of the Graph Cut, GrabCut and OneCut models. By clicking on the image with the mouse in the program or in a human-computer interactive way when running the program, specify some of the target foreground pixels and target background pixels in the image, so that the three energy functions can obtain the pixel features of the target foreground or background (i.e., the L＝{l ₁ ,l ₂ ,...,l _n } part of E(L), the pixel value z in E(α,k,θ,z), and the foreground point S in E(S)), minimize the energy function, and obtain the three corresponding binary images when minimized.

Minimize the energy function.

The basic idea of the image segmentation method based on graph cut theory is to map the image into a weighted undirected graph, regard pixels as nodes, the weight of the edge between nodes corresponds to the similarity measure between two pixels, and the capacity of the cut corresponds to the energy function. Image segmentation is achieved by minimizing the energy function.

The definition of the energy function includes boundary terms and region terms, where the region term is based on the basis for defining the energy function, namely the grayscale histogram, the RGB Gaussian mixture model, and the L1-norm of the global foreground and background. The region term represents the penalty for assigning a category label (foreground or background) to a pixel. When all pixels are correctly classified, the region term is the smallest, the penalty is the smallest, and the energy function is the smallest. The boundary term reflects the similarity between adjacent pixel pairs. When the similarity between two adjacent pixels is higher, the probability that the adjacent pixels are both foreground or background is greater; conversely, the probability that the adjacent pixels are at the junction of the foreground and background is greater. At this time, the boundary term is the smallest, thereby minimizing the energy function.

Step 4: Output binary image. After minimizing the energy function, the minimum cut of Graph Cut, GrabCut and OneCut is achieved, and the corresponding segmented binary image is output. At this point, the pixel point x in the image is divided into two categories: foreground and background, and the pixel value p satisfies the following relationship:

Step 5: Introduce a voting mechanism. The voting mechanism is a combination strategy for classification problems in ensemble learning. It is an ensemble learning model that follows the principle of majority rule, which reduces variance by integrating multiple models, thereby improving the robustness of the model. Ideally, the prediction effect of the voting method should be better than that of any base model. As shown in Figure 2, for the three segmented binary images obtained in step 5, a voting mechanism is used to vote on the values of the pixels at the same position in the three images to obtain the final value. According to the principle of the voting mechanism, the value after voting should be closer to the true category of the pixel, and at the same time, the robustness of the model can be improved. The voting process is as follows:

(1) Given three segmentation binary images, g1, g2, g3;

(2) Normalize the image and divide the value of each pixel in g1, g2, and g3 by 255 so that the pixel value is 0 or 1;

(3) For the three pixels g1, p2, and p3 at the same position, find their sum p;

(4) If p ≥ 2, assign 1 to the corresponding pixel on the final output segmentation map; otherwise, assign 0 to the corresponding pixel on the final output segmentation map, and obtain

(5) The value of each pixel in is multiplied by 255 to obtain the segmentation binary image g.

Step 6: Output the final segmentation map. Through mask operation, the binary image obtained in step 4 is converted into a segmentation map. The original image and the segmentation binary image are combined to extract the industrial parts in the original image separately to achieve segmentation. The specific implementation steps are as follows:

(1) Given the final segmentation binary image g and the original input image S;

(2) Divide the pixel value of each pixel in the segmented binary image g by 255 to obtain image g ₁ ;

(3) Multiply _g1 by the original input image S so that the pixel values of the background part are 0 and the pixel values of the foreground part remain unchanged;

(4) Output the final segmentation map G.

In order to verify the performance of the industrial parts segmentation method based on the voting mechanism of the present invention, images of 11 different industrial environments and different random combinations of industrial parts were tested. As can be seen from Figures 3 and 4, the algorithm of the present invention can comprehensively consider the advantages of other algorithms, strengthen the response of the algorithm at the boundary, and thus achieve more accurate segmentation, and the MIoU can reach 0.985. Moreover, in 11 different environments, VotingCut can maintain a high segmentation accuracy and segmentation effect, indicating that the method of the present invention has the advantage of strong robustness. Among them, the evaluation index MIoU is as follows:

Where N is the number of test samples, A and B are the target areas in the segmented image obtained by this method and the standard image produced by Labelme software. A∩B is the intersection of the two areas, and A∪B is the union of the two areas.

The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims (4)

1. The industrial part example segmentation method based on the voting mechanism is characterized by comprising the following steps of:

Step 1, collecting an original image of an industrial part;

Step 2, constructing a strong divider based on an image dividing method Graph Cut, grabCut and OneCut to process an original image, and recognizing foreground and background areas to output three binary images;

The step of constructing a strong segmenter based on the image segmentation methods Graph Cut, grabCut and OneCut comprises the following steps:

Step 21) defining an energy function: defining a Graph Cut energy function E (L) based on a gray histogram of an image, defining a GrabCut energy function E (alpha, k, theta, z) based on an image RGB Gaussian mixture model GMM, and defining OneCut an energy function E (S) based on L1-norm of a full-image foreground and a full-image background, wherein the energy function E (S) is used for measuring similarity of foreground areas, background areas and foreground and background junctions in the image so as to realize image segmentation; the three energy functions are defined as follows:

Wherein L is a class label of an image pixel, which can be expressed as l= { L ₁,l₂,...,l_n},l_i e {0,1},0 represents that the current pixel belongs to the background, 1 represents that the current pixel belongs to the foreground, R (L) is a region term, B (L) is a boundary term, a e [0,1] is an influence factor, and represents the influence degree on an energy function; alpha epsilon {0,1}, whether the model is a GMM component of a background model or a GMM component of a foreground model, K is a GMM Gaussian component parameter, and K epsilon {1, 2., K }; θ is the GMM parameter corresponding to each pixel; u (α, k, θ, z) represents a region term, V (α, z) represents a boundary term, and z is a value of a pixel in the image; beta, lambda is an adjustable parameter, S represents a set of foreground points, theta ^S represents a color histogram of the foreground points, represents a color histogram of the background points,/> represents a region term, is a boundary term

Step 22), initializing model parameters of three graph segmentation methods, and respectively inputting pixel characteristics of a target foreground or background in an original image into three energy functions;

Step 23) performing a calculation procedure of minimizing an energy function for separating the foreground and the background;

Step 24), outputting corresponding three segmentation binary images;

step 3, performing pixel-by-pixel fusion optimization based on a voting mechanism, and manufacturing a mask image;

And 4, performing mask operation on the original image to obtain an industrial part segmentation map.

2. The voting mechanism-based industrial part instance segmentation method according to claim 1, wherein the voting mechanism is based on pixel-by-pixel fusion optimization to vote on a pixel point at the same position of three segmentation binary images with the same size, and a value of the pixel point is determined.

3. The voting mechanism-based industrial part instance segmentation method according to claim 1, wherein the masking operation is an operation of extracting a target region in an original image to be segmented according to a binary image.

4. The voting mechanism-based industrial part example segmentation method according to claim 1, further comprising the step of evaluating the output segmented image with a standard image by using an evaluation index MIoU;

N is the number of test samples, A and B are target areas in the segmented image and the manufactured standard image obtained according to the method, the standard image is obtained by marking the original image pixel by pixel for distinguishing the current pixel as a foreground or a background, A N B is the intersection of two areas, and A U B is the union of the two areas.