CN115830004A - Surface defect detection method, device, computer equipment and storage medium - Google Patents

Abstract

The application provides a surface defect detection method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a surface image to be detected and a standard surface image of a product to be detected; performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one to-be-detected image block corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; inputting the image block to be detected and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block; and dividing the image block based on the defects, and determining whether the product to be detected has surface defects. So, this application utilizes the defect segmentation model that trains well to carry out the defect and cuts apart to this judges whether there is surface defect in waiting to examine the product, has improved the detection efficiency that surface defect detected and the degree of accuracy of testing result.

Description

Surface defect detection method, device, computer equipment and storage medium

technical field

The present application relates to the technical field of printed matter detection, and in particular to a surface defect detection method, device, computer equipment and storage medium.

Background technique

As users and manufacturers of industrial products have higher and higher requirements for product quality, the product needs to have good surface quality in addition to meeting the performance requirements. However, in the process of product manufacturing, surface defects are often unavoidable, therefore, it is necessary to detect surface defects on products. Among them, surface defect detection refers to the detection of scratches, foreign matter covering, wrinkles, spots, color pollution, holes and other defects on the surface of the product, so as to obtain a series of information such as the type, outline, position and size of the defects on the surface of the inspected product.

In the detection of surface defects of printed matter, manual detection is usually used. In this method, random sampling is first performed from a batch of printed matter according to the preset sampling rate, and then the surface of the extracted printed matter is manually observed for defects. However, manual detection has the problems of low accuracy, poor real-time performance, low efficiency, high labor intensity, and the detection results are greatly affected by manual experience and subjective factors.

Contents of the invention

The present application provides a surface defect detection method, device, computer equipment and storage medium, which can improve the detection efficiency of product surface defects.

In a first aspect, the present application provides a surface defect detection method, comprising:

Obtain the surface image and standard surface image of the product to be inspected;

performing image segmentation on the surface image to be inspected and the standard surface image, obtaining at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image; the number of image blocks to be inspected and the standard image block are the same, and The image areas where the image block to be checked and the standard image block are located in the corresponding surface image are also the same;

Input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmented image block; each defect segmented image block is used to represent the difference between an image block to be inspected and a standard image block pixel difference information;

Segment image blocks based on defects to determine whether there are surface defects in the product to be inspected.

In a possible implementation, image segmentation is performed on the surface image to be inspected and the standard surface image, and at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image are obtained, including:

Subtract the pixel values of the pixels at the same position in the surface image to be inspected and the standard surface image to obtain an intermediate surface image;

Perform connected region analysis on the intermediate surface image to obtain at least one candidate defect feature map;

According to the region position information corresponding to the candidate defect feature map in the intermediate surface image, image segmentation is performed on the surface image to be inspected and the standard surface image to obtain image blocks to be inspected and standard image blocks.

In a possible implementation, the image block is segmented based on the defect to determine whether there is a surface defect in the product to be inspected, including:

Obtain pixel intersection information between defect segmentation image blocks and candidate defect feature maps;

If the pixel intersection information is empty, it is determined that there is no surface defect on the surface of the product to be inspected;

If the pixel intersection information is not empty, it is determined that there are surface defects on the surface of the product to be inspected.

In a possible implementation, the defect segmentation model includes an encoder and a decoder;

Then input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmentation image block, including:

Input the image block to be checked and the standard image block into the encoder, and obtain the first multi-scale feature map of the image block to be checked at various preset resolutions and the first multi-scale feature map of the standard image block at various resolutions through the encoder The second multi-scale feature map;

The first multi-scale feature map and the second multi-scale feature map are input into the decoder, and the feature fusion processing is performed on the first multi-scale feature map and the second multi-scale feature map through the decoder, and the defect segmentation image block is output.

In a possible implementation manner, obtaining the surface image to be inspected and the standard surface image of the product to be inspected includes:

Obtain the initial image to be inspected and the initial standard image of the product to be inspected;

Based on the initial standard image, image registration processing is performed on the initial image to be inspected to obtain an intermediate image to be inspected;

Image preprocessing is performed on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image.

In a possible implementation, image preprocessing is performed on the intermediate image to be inspected and the initial standard image respectively to obtain the surface image to be inspected and the standard surface image, including:

Edge detection is performed on the intermediate image to be inspected and the initial standard image, and the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image are acquired;

Based on the first edge information, edge expansion processing is performed on the intermediate image to be inspected to obtain a surface image to be inspected;

Based on the second edge information, edge expansion processing is performed on the initial standard image to obtain a standard surface image.

In a possible implementation manner, the training process of the defect segmentation model includes:

Obtain the training sample image of the inspected product; the training sample image includes a standard sample image and multiple defect sample images, each sample image carries a sample label, and the sample label is used to indicate whether the sample image has surface defects;

Generate a plurality of training sample pairs according to the training sample images; each training sample pair includes two sample images;

Input a plurality of training sample pairs into the initial defect segmentation model to be trained, and calculate the training loss value of the target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

If the training loss value does not meet the preset convergence conditions, adjust the parameters of the initial defect segmentation model, and input multiple training sample pairs into the adjusted initial defect segmentation model again until the training loss value meets the preset convergence conditions , then end the training and get the trained defect segmentation model.

In a possible implementation, multiple training samples are generated according to the training sample images, including:

Combining the standard sample image with multiple defect sample images in turn to obtain multiple initial sample pairs;

Randomly perform data enhancement processing on multiple initial sample pairs to obtain multiple training sample pairs;

Among them, data enhancement processing includes at least one of the following:

Randomly exchanging the two sample images in the initial sample pair to obtain multiple first sample pairs;

Perform pixel offset processing on any sample image in the initial sample pair according to a preset pixel offset range to obtain a plurality of second sample pairs;

Randomly copying the standard sample image in the initial sample pair as a defective sample image to obtain a plurality of third sample pairs;

Paste the pre-extracted image defects into the standard sample image of the initial sample pair to obtain a plurality of fourth sample pairs;

The plurality of training sample pairs includes at least one sample pair among an initial sample pair, a first sample pair, a second sample pair, a third sample pair, and a fourth sample pair.

In a possible implementation manner, the target loss function of the initial defect segmentation model includes a Dice loss function and a cross-entropy loss function, and the calculation coefficients of the Dice loss function and the cross-entropy loss function in the target loss function are different.

In a possible implementation manner, if the product to be inspected is a printed matter, the surface defects include shallow sticking defects.

In a second aspect, the present application provides a surface defect detection device, comprising:

The image acquisition module is used to obtain the surface image to be inspected and the standard surface image of the product to be inspected;

The image segmentation module is used to perform image segmentation on the surface image to be inspected and the standard surface image, and obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image; the image block to be inspected and the standard image The number of blocks is the same, and the image areas where the image blocks to be checked and the standard image blocks are located in the corresponding surface images are also the same;

The defect segmentation module is used to input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmented image block; each defect segmented image block is used to represent an image block to be inspected and a Pixel difference information between standard image blocks;

The defect detection module is used for segmenting image blocks based on defects to determine whether there are surface defects in the product to be inspected.

In a third aspect, the present application provides a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor invokes and executes the computer program from the memory, it realizes any one of the above-mentioned first aspects. The steps of the surface defect detection method.

In a fourth aspect, the present application provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the surface defect detection method shown in any one of the above-mentioned first aspects are implemented.

In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the surface defect detection method shown in any one of the above-mentioned first aspects are implemented.

The technical solution provided by this application can at least achieve the following beneficial effects:

The surface defect detection method, device, computer equipment, and storage medium provided by this application, for obtaining the surface image to be inspected and the standard surface image of the product to be inspected, first perform image segmentation on the surface image to be inspected and the standard surface image to obtain the surface to be inspected At least one to-be-checked image block corresponding to the image and at least one standard image block corresponding to the standard surface image. Wherein, the numbers of the image blocks to be inspected and the standard image blocks are the same, and the image areas where the image blocks to be inspected and the standard image blocks are located in the corresponding surface images are also the same. Then, the image block to be inspected and the standard image block are input into the pre-trained defect segmentation model, and a defect segmented image block corresponding to an image block to be inspected and a standard image block is obtained through the defect segmentation model, and the defect segmented image block can be It reflects the pixel difference information between the image block to be checked and the standard image block. Finally, image blocks are segmented based on defects to determine whether there are surface defects in the product to be inspected. It can be seen that the present application utilizes a trained defect segmentation model for defect segmentation, which is more efficient than manual detection. Moreover, the defect segmentation model is a double-input neural network model, and its input includes a standard image block corresponding to a standard surface image and a standard image block corresponding to a surface image to be inspected, which solves the problem of weak generalization ability of the deep learning model and does not need to target different Model training for different scenarios and different products reduces the cost of model training. The defect segmentation model with two inputs can maintain good defect detection results even when encountering products that do not exist in the training dataset. In addition, the application performs image segmentation on the surface image to be inspected and the standard surface image in advance, obtains at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image, and then uses the defect segmentation model to classify each Image blocks are processed, which reduces the resource consumption in the image processing process and reduces the computing performance requirements for computer equipment when the defect segmentation model performs defect segmentation on image blocks, so that the surface defect detection method can be realized on low-end computing devices High-speed real-time calculation improves the efficiency of surface defect detection.

Description of drawings

FIG. 1 is a schematic structural diagram of a computer device shown in an exemplary embodiment of the present application;

Fig. 2 is a schematic flow chart of a surface defect detection method shown in an exemplary embodiment of the present application;

Fig. 3 is a schematic diagram of an image processing flow shown in an exemplary embodiment of the present application;

Fig. 4 is a schematic structural diagram of a defect detection model shown in an exemplary embodiment of the present application;

Fig. 5 is a schematic diagram of a training process of a defect segmentation model shown in an exemplary embodiment of the present application;

Fig. 6 is a schematic flow chart of generating training sample pairs shown in an exemplary embodiment of the present application;

Fig. 7 is a schematic flowchart of another surface defect detection method shown in an exemplary embodiment of the present application;

Fig. 8 is a schematic structural diagram of a surface defect detection device shown in an exemplary embodiment of the present application;

Fig. 9 is a schematic structural diagram of another surface defect detection device shown in an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be further described in detail below in conjunction with the drawings and embodiments.

Before explaining the surface defect detection method provided by the embodiment of the present application, the application scenario and implementation environment of the embodiment of the present application will be introduced first.

As users and manufacturers of industrial products have higher and higher requirements for product quality, industrial products must not only meet the performance requirements, but also have a good appearance, that is, a good surface quality. However, in the process of product manufacturing, the generation of surface defects is often unavoidable. Among them, defects can generally be understood as surface defects, surface defects or area deviations compared with normal products. The surface defect detection refers to the detection of scratches, defects, foreign body occlusion, color pollution, holes and other defects on the surface of the product, so as to obtain a series of defect types, contours, positions, and sizes of the surface defects of the product to be inspected that can describe the defect. information.

Moreover, surface defects not only affect the beauty and comfort of the product, but also generally have a negative impact on the performance of the product. Therefore, manufacturers attach great importance to the detection of surface defects in order to find surface defects in time and effectively control product quality. . Furthermore, it is also possible to analyze the technical/operational problems in the production process based on the surface defect detection results, so as to eliminate or reduce the generation of defective products, prevent potential trade disputes, and maintain corporate reputation.

Manual inspection is a traditional detection method for product surface defects, but this method has low sampling rate, low accuracy, poor real-time performance, low efficiency, high labor intensity, and is greatly affected by manual experience and subjective factors.

In addition, based on the traditional image processing algorithm, there are false positives and missed detections in the process of shallow sticky defect detection of printed matter, and it is necessary to manually adjust the parameters repeatedly. However, the surface defect detection method based on single image input based on deep learning has problems such as slow reasoning speed and weak generalization ability; and for different scenarios and different types of products, specific neural network models need to be trained, which increases the detection cost. .

Although the surface defect detection technology based on deep learning can largely overcome the above-mentioned drawbacks of manual detection, the types of products to be inspected are diverse, and with the improvement of the process, the surface defects are also relatively subtle. Therefore, how to construct a stable and efficient surface defect detection method based on machine vision has become the biggest problem facing the application of machine vision in the field of surface defect detection.

Based on this, this application provides a surface defect detection method, device, computer equipment and storage medium, using deep learning technology with appropriate data enhancement processing and training strategies to train the defect segmentation model, and through the defect segmentation model during actual detection Compare and analyze the pre-processed image block and the standard image block to determine whether there are surface defects in the product to be inspected. In this way, the defect segmentation model trained in this application has strong stability, generalization and high efficiency. Therefore, combined with image preprocessing and defect segmentation model, the detection efficiency of surface defects of the product to be inspected can be improved, and in a variety of Products and multiple scenarios can achieve stable detection results.

In an exemplary embodiment, the applicant can use computer equipment to perform image preprocessing on the surface image of the product to be inspected and the standard surface image of the product to be inspected, and analyze the surface to be inspected through a pre-trained defect segmentation model The pixel difference between the image and the standard surface image, so as to determine whether there are surface defects in the product to be inspected.

In a possible implementation manner, the computer device has a structure as shown in FIG. 1 , and the computer device 100 includes at least one processor 110 , a memory 120 , a communication bus 130 , and at least one communication interface 140 .

Wherein, the processor 110 may be a general-purpose central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), a microprocessor, or may be one or more integrated circuits for realizing the scheme of the present application For example, Application-Specific Integrated Circuit (Application-Specific Integrated Circuit, ASIC), Programmable Logic Device (Programmable Logic Device, PLD) or a combination thereof. The aforementioned PLD may be a complex programmable logic device (Complex Programmable Logic Device, CPLD), a field programmable logic gate array (Field-Programmable Gate Array, FPGA), a general array logic (Generic Array Logic, GAL) or any combination thereof.

Optionally, the processor 110 may include one or more CPUs. Computer device 100 may include multiple processors 110 . Each of these processors 110 may be a single-core processor (single-CPU), or a multi-core processor (multi-CPU).

It should be noted that the processor 110 here may refer to one or more devices, circuits and/or processing cores for processing data (such as computer program instructions).

The memory 120 can be a read-only memory (Read-Only Memory, ROM) or other types of static storage devices that can store static information and instructions, and can also be a random access memory (Random Access Memory, RAM) or can store information and instructions. Other types of dynamic storage devices for instructions can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disc storage , optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and any other medium that can be accessed by a computer, but is not limited to.

Optionally, the memory 120 may exist independently and be connected to the processor 110 through the communication bus 130 ; the memory 120 may also be integrated with the processor 110 .

The communication bus 130 is used to transmit information between various components (such as between the processor and the memory), and the communication bus 120 can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one communication bus is used for illustration in FIG. 1 , but it does not mean that there is only one bus or one type of bus.

The communication interface 140 is used for the computer device 100 to communicate with other devices or a communication network. The communication interface 140 includes a wired communication interface or a wireless communication interface. Wherein, the wired communication interface may be an Ethernet interface, for example. The Ethernet interface can be an optical interface, an electrical interface or a combination thereof. The wireless communication interface may be a wireless local area network (Wireless Local Area Networks, WLAN) interface, a cellular network communication interface, or a combination thereof.

In some embodiments, the computer device 100 may also include an output device and an input device (not shown in FIG. 1 ). Output devices are in communication with processor 110 and can display information in a variety of ways. For example, the output device may be a liquid crystal display (Liquid Crystal Display, LCD), a light emitting diode (Light Emitting Diode, LED) display device, a cathode ray tube (Cathode Ray Tube, CRT) display device, or a projector. The input device is in communication with the processor 110 and can receive user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, or a sensing device, among others.

In some embodiments, the memory 120 is used to store computer programs for implementing solutions of the present application, and the processor 110 can execute the computer programs stored in the memory 120 . For example, the computer device 100 can invoke and execute a computer program stored in the memory 120 through the processor 110, so as to implement the steps of the surface defect detection method provided in the embodiment of the present application.

It should be understood that the surface defect detection method provided in this application can be applied to a surface defect detection device, and the surface defect detection device can be implemented as part or all of the processor 110 through software, hardware, or a combination of software and hardware, integrated in In the computer device 100.

Next, the technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail through embodiments and in conjunction with the accompanying drawings. Various embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Apparently, the described embodiments are some of the embodiments of the present application, but not all of them.

In an exemplary embodiment, as shown in FIG. 2 , the present application provides a surface defect detection method, which is illustrated by applying the method to the computer device 100 shown in FIG. 1 above. The method may include the following steps:

Step 210: Obtain the surface image to be inspected and the standard surface image of the product to be inspected.

Wherein, the product to be inspected is any product produced on an industrial production line, and the surface image to be inspected is a surface image of the product to be inspected captured in real time. Correspondingly, the standard surface image is the same type as the product to be inspected and has no surface defects.

As an example, if the product to be inspected is a printed matter, the surface defects include light sticking defects.

It should be noted that, for a batch of products of the same type, during the production process, the corresponding surface images to be inspected may be sequentially acquired for a plurality of products to be inspected. For this batch of products, the same standard surface image can be used to implement the surface defect detection method provided by this application, and the surface defect detection method provided by this application can also be performed on multiple standard surface images that do not have surface defects pre-checked in this batch of products. The detection method is not limited in this embodiment of the present application.

It should be understood that the surface of the product to be inspected may or may not have defects. If there are defects on the surface of the product to be inspected, if the image of the surface to be inspected and the standard surface image are taken by the same image acquisition device, the image area with defects in the image of the surface to be inspected is different from the corresponding image area in the standard surface image There will be pixel differences between. Based on this, the present application analyzes the pixel difference between the surface image to be inspected and the standard surface image through the following steps 220 and 230 .

As an example, the image acquisition device for capturing the image of the surface to be inspected and the image of the standard surface may be an RG B camera.

Due to redundant information and noise in the initial image of the product to be inspected in real time on the production line, if the initial image to be inspected is directly used as the surface image of the product to be inspected, the information in the subsequent model processing process will be increased The processing capacity may also affect the final defect detection results.

Therefore, preprocessing can be performed on the actual image of the product to be inspected and the initial standard image, so as to obtain the surface image of the product to be inspected and the standard surface image.

In a possible implementation manner, as shown in FIG. 3, the implementation process of the above step 210 may include the following sub-steps:

Step 211: Obtain the initial image to be inspected and the initial standard image of the product to be inspected.

Optionally, before the following step 213 is performed, denoising processing may be performed on the initial image to be inspected and the initial standard image, so as to eliminate noise information and redundant information in the image.

Wherein, the denoising processing may be realized by a filtering algorithm. The filtering algorithm may be mean filtering, median filtering, Gaussian filtering, etc., which is not limited in this embodiment of the present application.

Optionally, for the convenience of calculation, the initial image to be inspected and the initial standard image may be grayscaled to convert the color image captured by the RGB camera into a grayscale image.

It should be understood that the values of the three components of R, G, and B in a color image determine a specific pixel, and a pixel can have tens of millions of colors; and a grayscale image is also a color image, but its The feature is that the specific values of the three components of R, G, and B are consistent, and the pixel value change range of each pixel in the grayscale image is 0 to 255.

The initial to-be-inspected image and the initial standard image are converted into corresponding grayscale images. On the basis of retaining the outline and features of the image, the grayscale image can still reflect the outline and texture of the entire image without affecting the subsequent defect detection.

Step 213: Based on the initial standard image, perform image registration processing on the initial image to be inspected to obtain an intermediate image to be inspected.

It should be understood that the initial image to be inspected is subjected to image registration processing to obtain an intermediate image to be inspected, and in step 213, the initial standard image does not need to be processed. In other words, after step 213 is executed, an intermediate candidate image and an initial standard image are obtained.

In the actual implementation process, due to reasons such as shooting angle and shooting light, the initial to-be-checked image and the initial standard image may not be able to achieve pixel-level alignment, and there will be a pixel offset between the two.

Therefore, it is necessary to perform image registration processing on the initial candidate image and the initial standard image, so that the initial candidate image and the initial standard image are aligned in terms of image size and pixel position.

Step 215: Perform image preprocessing on the intermediate image to be inspected and the initial standard image, respectively, to obtain the surface image to be inspected and the standard surface image.

Wherein, the image pre-processing may include at least one of down-sampling processing, eliminating redundant points, edge detection, erosion processing, and dilation processing.

In a possible implementation, the implementation process of step 215 is: perform edge detection on the intermediate image to be inspected and the initial standard image, and obtain the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image; The first edge information performs edge expansion processing on the intermediate image to be inspected to obtain the surface image to be inspected; based on the second edge information, the initial standard image is subjected to edge expansion processing to obtain the standard surface image.

Among them, the image edge is the most basic feature of the image, and the edge (Edge) refers to the discontinuity of the local characteristics of the image, and the sudden change of information such as grayscale or structure is called the edge. For example, the sudden change of gray level, the sudden change of color, the sudden change of texture structure, etc.

Specifically, the edge detection operators used in the edge detection operation may be: Roberts operator, Prewitt operator, Sobel operator, Laplacian operator, LoG operator (also called Marr edge detection operator, or Gaussian-Laplacian), Canny edge detection.

Among them, the Roberts operator is also called the cross-differential algorithm, which is a gradient algorithm based on cross-difference, and detects edge lines through local difference calculations; the template of the Roberts operator is divided into horizontal and vertical directions, which can better enhance the positive and negative 45 degrees of image edges. The principle of the Prewitt operator is to use the difference generated by the gray value of the pixel in a specific area to realize edge detection; the Prewitt operator uses a template to calculate the pixel value of each pixel in the area, so that the edge detection result is horizontal and vertical direction are more obvious than Robert operator. The Laplacian operator is a second-order differential operator in n-dimensional Euclidean space, which is divided into four neighborhoods and eight neighborhoods. The four neighborhoods are to calculate the gradient in the four directions of the center pixel of the neighborhood, and the eight neighborhoods It is to find the gradient for eight directions. The LoG operator generally performs Gaussian low-pass filtering before using the Laplacian operator. The basic idea of Canny edge detection is to first select a certain Gauss filter for image smoothing, and then use non-extreme value suppression technology to process to obtain the final edge image.

As an example, the embodiment of the present application uses a Sobel operator to perform edge detection on the intermediate image to be inspected and the initial standard image, so as to obtain image edge information.

Among them, the Sobel operator adds the concept of weight on the basis of the Prewitt operator. It believes that the distance between adjacent pixels has different effects on the current pixel. The closer the pixel, the greater the impact on the current pixel. , the farther away the pixel has less influence on the current pixel, so as to achieve image sharpening and highlight the edge contour.

In this way, the Sobel operator can fully consider the mutual influence between pixels, so that the calculated edge information is more accurate, it can better describe the real contour in the middle image to be checked, and show a clear contour edge.

Furthermore, image edge expansion is to add pixel values at the edge of the image, so that the overall pixel value is expanded, and then achieve the image expansion effect, which can also be called pixel interpolation processing.

Wherein, according to the positional relationship of adjacent pixels referenced during the expansion process, the image expansion process includes horizontal expansion, vertical expansion and omnidirectional expansion.

As an example, in the embodiment of the present application, the edge of the intermediate image to be inspected and the initial standard image may be expanded in all directions to obtain the surface image to be inspected and the standard surface image.

It should be understood that the omni-directional expansion process traverses the target area in the target image, only considers the gray value of the target pixel and its adjacent four pixels up, down, left, and right, and confirms whether there is an intersection with the expanded structural element, that is, whether There is at least one place where corresponding grayscale values are equal. If there is an intersection, the pixel is processed; otherwise, the pixel is deleted.

In this way, through all-round dilation processing, not only can the image boundary points be expanded, but also the background points that are in contact with the image edge can be merged into the edge information, so that after dilation processing, the relationship between the image edge and the background information can be better reflected. Real bounds of , the dilation handles better.

Step 220: Carry out image segmentation between the surface image to be inspected and the standard surface image, and obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image.

Wherein, the numbers of the image blocks to be inspected and the standard image blocks are the same, and the image areas where the image blocks to be inspected and the standard image blocks are located in the corresponding surface images are also the same.

In other words, when segmenting the surface image to be inspected and the standard surface image in step 220, the image is segmented based on the same image position, so that after the segmentation, the position coordinates of the image area corresponding to the image block to be inspected in the surface image to be inspected , the position coordinates of the image area corresponding to the standard image block in the standard surface image are the same; at the same time, the numbers of the image blocks to be inspected and the standard image blocks obtained after segmentation are also the same.

In a possible implementation, the implementation process of step 220 can be: performing pixel value subtraction operation on pixels at the same position in the surface image to be checked and the standard surface image to obtain an intermediate surface image; performing connected region analysis on the intermediate surface image , to obtain at least one candidate defect feature map; according to the region position information corresponding to the candidate defect feature map in the intermediate surface image, perform image segmentation on the surface image to be inspected and the standard surface image, and obtain image blocks to be inspected and standard image blocks.

Since the surface image to be inspected and the standard surface image are image pairs after image registration, the pixels at corresponding positions in the two surface images describe the same surface feature of the product to be inspected, so the pixel values can be directly subtracted.

It should be understood that if both the surface image to be inspected and the standard surface image are grayscale images, an intermediate surface image obtained after processing is also a grayscale image. The pixel value of each pixel in the intermediate surface image is specifically the absolute value of the difference between two pixel values at corresponding positions in the surface image to be inspected and the standard surface image.

As an example, at least one candidate defect feature map may be obtained by performing Blob analysis on the intermediate surface image.

Wherein, in computer vision, Blob refers to a connected area formed by features such as similar colors and textures in the image. The connected domain (the connected domain is called a Blob block) is analyzed to obtain the feature map of the connected domain, that is, the candidate defect feature map.

Specifically, Blob analysis is to binarize the pixel values of each pixel in the intermediate surface image to segment the foreground and background; then perform connected region detection to obtain Blob blocks; and then extract pixel features from each Blob block to obtain Candidate defect feature maps.

It should be noted that the number of connected domains in the intermediate surface image is the same as the number of candidate defect feature maps. That is, if there is only one connected domain in the intermediate surface image, a candidate defect feature map will be obtained after Blob analysis; if there are multiple connected domains in the intermediate surface image, then the candidate defect features corresponding to each connected domain will be obtained after Blob analysis picture.

Further, taking a candidate defect feature map as an example, based on the candidate defect feature map, image segmentation is performed on the surface image to be inspected and the standard surface image, and the implementation process of obtaining the image block to be inspected and the standard image block can be as follows: Obtain the feature map of the candidate defect According to the corresponding area position information in the intermediate surface image, the area position information is obtained; image segmentation is performed on the surface image to be inspected according to the area position information, so as to segment the partial image corresponding to the area position information from the surface image to be inspected, and obtain the area to be inspected. image block; similarly, image segmentation is performed on the standard surface image according to the location information of the region, so as to segment the local image corresponding to the location information of the region from the standard surface image to obtain a standard image block.

In this way, the embodiment of the present application does not directly input the surface image to be inspected and the standard surface image into the trained defect segmentation model, but combines image preprocessing and image segmentation to reduce the amount of processing information in the image block and reduce the impact of defect segmentation on defects. The computing resource requirements of the segmentation model can realize high-speed real-time computing of the defect segmentation method based on surface images on low-end computing equipment.

Step 230: Input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmented image block.

Wherein, each defective segmented image block is used to represent pixel difference information between an image block to be inspected and a standard image block.

It should be noted that the defect segmentation model provided by this application is a dual-input neural network model. When performing defect segmentation, the defect segmentation model requires the input of the model to be two corresponding images, and the input sequence of the two images no requirement.

Optionally, there are a large number of non-defect features in the candidate defect feature map. According to the area position information of each candidate defect feature map, after collecting small-sized image blocks at the corresponding positions on the surface image to be inspected and the standard surface image, it can be based on The area size of the candidate defect feature map, sort the image blocks to be inspected and the standard image blocks obtained by segmentation, to ensure that the image blocks to be inspected and standard image blocks corresponding to the large-area candidate defect feature map can be preferentially processed by the defect segmentation model .

As an example, take the shallow sticky defect existing in printed matter as an example, in order to improve the effect of shallow sticky defect detection and achieve better defect detection accuracy, the embodiment of the present application uses the Transformer model (which is a self- The deep learning model of the attention mechanism, the self-attention mechanism can assign different weights according to the importance of each part of the input data) Seg Former (a semantics that unifies the transformer and the lightweight multi-layer perceptron decoder Segmentation Framework), which serves as the backbone of the defect segmentation model. Among them, SegFormer uses a hierarchical encoder structure that can output multi-scale feature maps and fuse multi-scale feature maps together in the decoder.

Among them, SegFormer avoids complex decoders, and the proposed MLP decoder can aggregate information from different layers. In other words, SegFormer is similar to the fusion of shallow feature maps and deep feature maps in the convolutional neural network model. The purpose is to enable high-resolution coarse-grained features and low-resolution fine-grained features to be captured together, thereby optimizing Split results.

Moreover, the embodiment of the present application discards the Transformer model and the positional encoding in the self-attention-based encoder (segmentation transformer, SETR for short). When the size of the input picture is different from the size of the picture used for training, Interpolation of position encoding vectors is no longer required (leading to performance degradation when the test resolution differs from the training resolution).

In this way, based on the simple and effective decoder designed in this application, the defect detection accuracy rate of more than 90% can be achieved for shallow sticking defects.

Based on this, the defect segmentation model provided by this application includes an encoder and a decoder. The implementation process of step 230 can be: input the image block to be inspected and the standard image block into the encoder, and obtain the image block to be inspected by the encoder in the The first multi-scale feature map at multiple resolutions and the second multi-scale feature map of the standard image block at multiple resolutions; input the first multi-scale feature map and the second multi-scale feature map to the decoder In the method, feature fusion processing is performed on the first multi-scale feature map and the second multi-scale feature map through the decoder, and the defect segmented image block is output.

It should be noted that each resolution corresponds to a first feature map and a second feature map, and multiple resolutions correspond to multiple first feature maps (ie, the first multi-scale feature map) and multiple second feature maps (second multi-scale feature map).

As an example, refer to the structural diagram of the defect detection model shown in Figure 4, which only uses the processing flow of one image block to be inspected and one standard image block as an example, and multiple image blocks to be inspected and multiple standard image blocks are input The specific processing flow after the defect segmentation model is similar to this, and will not be repeated here.

Among them, the encoder is composed of 4 Transformer blocks (Transformer block 1, Transformer block 2, Transformer block 3, and Transformer block 4). After each Transformer block, the feature map is down-sampled to 1/4, 1/8, 1/16, 1/32. These four feature maps of different resolutions are then fused in the decoder. The decoder can be implemented through a convolutional neural network, which performs bilinear difference and 3x3 convolution operations on all feature maps in the encoder to achieve the effect of transposed convolution, while avoiding the checkerboard effect on the network model. Impact.

Specifically, in the decoder, feature maps of different sizes are first upsampled to the original size of the image, and then concatenated to obtain a concatenated feature map containing high-resolution coarse-grained features and low-resolution fine-grained features. Further, the spliced feature map is input into the subsequent convolutional network, and three convolution operations are used to reduce the dimensionality of the feature map to 64 dimensions, 32 dimensions, and 16 dimensions, respectively. Finally, the 16-dimensional convolution feature map is input to the final convolution layer, and the dimensionality is reduced to 2 dimensions.

Wherein, the 2D here indicates whether the current pixel has surface defects. For example, whether there is a shallow sticky flower defect.

Based on the above explanations, as shown in Figure 5, the process of pre-training the defect segmentation model in this application may include the following steps:

Step 510: Obtain training sample images of inspected products.

Wherein, the inspected product is a product marked with surface defects, and the inspected product and the above-mentioned product to be inspected may be the same type of product, or may be a different type of product.

In this step, the training sample image includes a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used to indicate whether there is a surface defect in the sample image;

It should be noted that multiple defect sample images of the same type of product may also be combined with multiple standard sample images to form a training sample pair, and the embodiment of the present application does not limit the number of standard sample images.

As an example, for the inspected product A, multiple defect sample images detected in advance of the inspected product may be obtained, and two standard sample images of the inspected product A may be obtained at the same time. Among them, the shooting angle and lighting information of the two standard sample images are slightly different.

Step 520: Generate a plurality of training sample pairs according to the training sample images.

Wherein, each training sample pair includes two sample images.

In a possible implementation, as shown in FIG. 6, the implementation process of step 520 may be:

Step 521: combining the standard sample image with multiple defective sample images in sequence to obtain multiple initial sample pairs;

That is, each initial sample pair includes a defect sample image and a standard sample image.

Step 523: Randomly perform data enhancement processing on multiple initial sample pairs to obtain multiple training sample pairs.

In a possible implementation manner, the data enhancement processing includes at least one of the following:

(1) Randomly exchange two sample images in the initial sample pair to obtain multiple first sample pairs;

It should be noted that each initial sample pair can be expressed as (sample image 1, sample image 2), that is, for the defect segmentation model, there is a sequence between the two sample images in the initial sample pair.

As an example, in order to complete the modeling of the difference between the two input images from the training strategy, when inputting the defect sample image D and the standard sample image R, the initial sample pair (D, R) is randomly exchanged, and the defect label is not changed , so that f(D, R) = f(R, D) holds, thus conforming to the dual-input difference modeling.

(2) Perform pixel offset processing on any sample image in the initial sample pair according to a preset pixel offset range to obtain multiple second sample pairs.

In the sample image pair registration preprocessing, due to the stability of the algorithm used, hardware limitations and speed requirements, etc., the defect sample image and the standard sample image cannot be aligned at the pixel level, and there will be an offset greater than 5 pixels. .

Based on this, in order to adapt the deep learning network model to this situation, when training the initial defect detection model, a random offset operation is performed on the defect sample image or the standard sample image in the initial sample pair, so that there is a gap between the two sample images. position offset.

In this way, by simulating the common image shift phenomenon in the actual scene, the defect segmentation model finally trained can be more robust.

(3) Randomly copy the standard sample images in the initial sample pair as defect sample images to obtain multiple third sample pairs.

That is to say, the third sample pair includes two identical defect sample images. At this time, no matter how the image is shifted or rotated, the defect label corresponding to the third sample pair should be a matrix of all zeros, indicating that the two There are no differences between sample images, and therefore no defects.

(4) Paste the pre-extracted image defects into the standard sample images of the initial sample pairs to obtain multiple fourth sample pairs.

Specifically, the marked surface defects in the historical inspection data set are extracted and pasted randomly on the standard sample image of the initial sample pair to obtain a randomly generated defect sample image. Furthermore, the fourth sample pair is formed by using the original standard sample image and the randomly generated defect sample image in the initial sample pair to alleviate the data volume demand of the deep learning model.

Based on this, the multiple training sample pairs used to train the initial defect segmentation model in the embodiment of the present application include: the above-mentioned initial sample pair, the above-mentioned first sample pair, the above-mentioned second sample pair, the above-mentioned third sample pair and the above-mentioned fourth sample pair. At least one of the sample pairs.

Step 530: Input multiple training sample pairs into the initial defect segmentation model to be trained, and calculate the training loss value of the target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

In a possible implementation, the implementation process of step 530 may be: divide multiple training sample pairs into a training set and a verification set, and use the training sample pairs in the training set to input them into the initial defect segmentation model in batches, so that The initial defect segmentation model can fully learn the feature information of standard sample images and defect sample images. After the training set is used to preset rounds according to the preset iterative training rounds, the training sample pairs in the verification set are used to test the learning effect of the initial defect segmentation model, and the training loss value of the target loss function of the initial defect segmentation model is calculated.

Wherein, both the training set and the verification set include a plurality of training sample pairs, and the number of training sample pairs in the two sets may be the same or different, which is not limited in this embodiment of the present application.

It should be understood that the training loss is determined based on the difference between the pixel information in the defect segmentation image block predicted by the initial defect segmentation model and the input training sample to the pre-marked defect label. If the difference decreases, it means that When the training result is close to the label information, the training can be ended.

As an example, the preset iterative training rounds may be any number, for example, 10 rounds, 20 rounds, and so on.

Furthermore, for shallow sticky flower defects, it only accounts for a small part of the entire single color box image, so there may be a class imbalance problem when performing pixel-level classification. Although the aforementioned data enhancement processing can alleviate some of the problems, the embodiment of the present application still needs to use the target loss function to constrain the imbalance in categories, so as to avoid the occurrence of missed detection.

In a possible implementation manner, the target loss function of the initial defect segmentation model includes a Dice loss function and a cross-entropy loss function, and the calculation coefficients of the Dice loss function and the cross-entropy loss function in the target loss function are different.

It should be understood that Dice is a region-related loss function, which is used to calculate the similarity between two samples. It has good performance in the scene where the positive and negative samples are seriously unbalanced. During the training process, more emphasis is placed on mining the foreground area; Cross-entropy can measure the degree of difference between two different probability distributions in the same random variable. In machine learning, it is expressed as the difference between the real probability distribution and the model-predicted probability distribution. The smaller the cross-entropy value, the better the model prediction effect. the better.

As an example, the Dice loss function and the cross-entropy (CrossEntropy) loss function are used to set the target loss function of the initial defect segmentation model, as shown in the following formula (1):

Loss=1.0*Dice(A,A')+0.1*CrossEntropy(A,A') (1)

Among them, the Dice loss function and the cross-entropy loss function calculate the pixel position deviation, see the following formula (2) and formula (3)

Dice(A, A')=1-2|A&A'|/(|A|+|A'|) (2)

CrossEntropy(A, A')=-\sum A'log(A) (3)

In the formula, A' represents the defect segmentation result output by the initial defect segmentation model, that is, the pixel position information of the shallow sticky defect on the defect sample image; A represents the manually pre-labeled shallow sticky defect position information in the defect sample image; Dice loss The function calculation coefficient is 1, and the calculation coefficient of the cross-entropy loss function is 0.1.

In this way, by calculating the corresponding pixel position deviation through the Dice loss function and the cross-entropy loss function, and superimposing the loss weight (that is, the calculation coefficient) to set the target loss function of this application, the problem of serious imbalance between positive and negative samples can be solved.

Step 540: If the training loss value does not meet the preset convergence condition, adjust the parameters of the initial defect segmentation model, and input multiple training sample pairs into the adjusted initial defect segmentation model again until the training loss value meets the preset If the convergence condition is met, the training ends and the trained defect segmentation model is obtained.

Wherein, adjusting the parameters of the initial defect segmentation model includes adjusting the processing parameters of each Transformer block in the initial defect segmentation model and the network parameters of the convolutional neural network.

As an example, the preset convergence condition includes: the training loss value reaches a preset loss threshold; the training loss value does not change in preset iterative training rounds; and the total number of training rounds reaches a preset total number of training times.

For example, the loss threshold can be 0.1, the iterative training rounds can be 10 rounds, and the total number of training rounds can be 100 rounds.

Optionally, before the trained defect segmentation model is put into use, some product surface images can also be obtained as a test set to test the accuracy of the defect segmentation image blocks predicted by the defect segmentation model, and the test results meet the requirements In the case of , the trained defect segmentation model can assist in surface defect detection in actual scenes.

In the above process of training the defect segmentation model, the embodiment of the present application obtains multiple training sample pairs by randomly performing data enhancement processing on the initial sample pairs, which improves the diversity of training sample pairs and increases the number of training sample pairs, so that the final The trained defect segmentation model has better robustness. At the same time, the embodiment of the present application uses the Dice loss function and the cross-entropy loss function to set the target loss function of the initial defect segmentation model, which improves the segmentation accuracy of the defect segmentation model and avoids the occurrence of missed detection.

Step 240: Segment image blocks based on defects, and determine whether the product to be inspected has surface defects.

Wherein, the defect segmented image block is used to represent the pixel difference information between an image block to be inspected and a standard image block. In other words, the pixels in the defect segmentation map can reflect whether there is surface defect information in the surface image to be inspected compared with the standard surface image, and the surface defect information includes defect position, size, contour and other information.

It should be understood that the number of defect segmented image blocks is the same as the number of to-be-inspected image blocks or standard image blocks input into the defect segmentation model, and for a defect segmented image block, the number of pixels included may be 0 , and may not be 0.

Among them, if the number of pixels in the defect segmentation image block is 0, it means that there is no pixel difference between the image block to be inspected and the standard image block corresponding to the defect segmentation model judgment; if the number of pixels in the defect segmentation image block If it is not 0, it means that there are differences between the image block to be inspected corresponding to the judgment of the defect segmentation model and the standard image block at these non-zero pixel points.

In a possible implementation, the implementation process of step 240 can be: segment image blocks according to defects, and determine whether there are pixels describing defect information in each defect segmented image block; , that is, the defect segmented image block is empty, it is determined that there is no surface defect in the corresponding image area of the defect segmented image block in the surface image to be inspected; if there are pixels describing defect information in the defect segmented image block, that is If the defect segmented image block is not empty, it is determined that the defect segmented image block has a surface defect in a corresponding image region in the surface image to be inspected.

Specifically, when there is no pixel point describing defect information in the defect segmentation image blocks corresponding to all the image blocks to be inspected in the surface image to be inspected, it is determined that there is no surface defect in the surface image to be inspected, and the corresponding product to be inspected There is also no surface defect, and the product to be inspected is a good product.

Further, if there are pixels describing defect information in the defect segmented image block, the position information of these pixels in the defect segmented image block can be determined, and the position information can be mapped to the surface image to be inspected to determine the surface defect Defect information such as the specific location, size, and contour of the object.

In another possible implementation, the implementation process of step 240 can be: obtain the pixel intersection information between the defect segmentation image block and the candidate defect feature map; if the pixel intersection information is empty, then determine that there is no Surface defect; if the pixel intersection information is not empty, it is determined that there is a surface defect on the surface of the product to be inspected.

It should be understood that when the pixel values of two pixel points are different, the difference between the pixel values itself can reflect the difference between the two. Therefore, when judging whether there is a surface defect in the product to be inspected based on the defect segmentation image block, it is also possible to combine the pixels at the same position in the surface image to be inspected and the standard surface image to perform pixel value subtraction and connect region analysis to obtain candidate defect features Figure, thereby improving the accuracy of surface defect detection results.

Therefore, if there are pixels describing defect information between the defect segmentation image block and the candidate defect feature map, after taking the intersection, the pixel intersection information includes these pixels describing defect information.

Further, if the pixel intersection information is not empty, it is also possible to determine the position information of each pixel in the intersection in the defect segmentation image block or the candidate defect feature map, and map the position information to the surface image to be inspected to determine Defect information such as the specific position, size, and contour of surface defects.

In the embodiment of the present application, after the computer equipment acquires the surface image to be inspected and the standard surface image of the product to be inspected, it first performs image segmentation on the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected At least one standard image patch corresponding to the standard surface image. Wherein, the numbers of the image blocks to be inspected and the standard image blocks are the same, and the image areas where the image blocks to be inspected and the standard image blocks are located in the corresponding surface images are also the same. Then, the image block to be inspected and the standard image block are input into the pre-trained defect segmentation model, and a defect segmented image block corresponding to an image block to be inspected and a standard image block is obtained through the defect segmentation model, and the defect segmented image block can be It reflects the pixel difference information between the image block to be checked and the standard image block. Finally, image blocks are segmented based on defects to determine whether there are surface defects in the product to be inspected. It can be seen that the present application utilizes a trained defect segmentation model for defect segmentation, which is more efficient than manual detection. Moreover, the defect segmentation model is a double-input neural network model, and its input includes a standard image block corresponding to a standard surface image and a standard image block corresponding to a surface image to be inspected, which solves the problem of weak generalization ability of the deep learning model and does not need to target different Model training for different scenarios and different products reduces the cost of model training. The defect segmentation model with two inputs can maintain good defect detection results even when encountering products that do not exist in the training dataset. In addition, the application performs image segmentation on the surface image to be inspected and the standard surface image in advance, obtains at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image, and then uses the defect segmentation model to classify each Image blocks are processed, which reduces the resource consumption in the image processing process and reduces the computing performance requirements for computer equipment when the defect segmentation model performs defect segmentation on image blocks, so that the surface defect detection method can be realized on low-end computing devices High-speed real-time calculation improves the efficiency of surface defect detection.

Based on the above embodiments, in an exemplary embodiment, as shown in FIG. 7 , the present application also provides another surface defect detection method, which is illustrated by applying the method to the computer device 100 shown in FIG. 1 above, The implementation process of the surface defect detection method is as follows:

The initial image to be inspected and the initial standard image of the product to be inspected are obtained, and based on the initial standard image, image registration processing is performed on the initial image to be inspected to obtain an intermediate image to be inspected. Then the Sobel operator is used to perform edge detection and edge expansion processing on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image. Then, feature comparison and Blob analysis are performed on the surface image to be inspected and the standard surface image to obtain N feature maps of candidate defects; where N≥1.

Further, based on the area position information of each candidate defect feature map, sort each candidate defect feature map according to the size of the image area, and perform image segmentation on the surface image to be inspected and the standard surface image based on each candidate defect feature map, and obtain N waiting Check image blocks and N standard image blocks.

Wherein, the N image blocks to be inspected correspond to the N standard image blocks one by one, and the corresponding image blocks to be inspected and the standard image blocks describe the same surface area on the product to be inspected.

Further, N image blocks to be inspected and N standard image blocks are input into the double-input defect segmentation model provided by this application in the order of image block pairs, and the correspondence between each pair of input image blocks to be inspected and standard image blocks is obtained. The defect segmented image blocks.

Wherein, the defect segmented image block is used to describe the pixel difference information between an image block to be inspected and a standard image block, and the pixels included therein may be empty, that is, there is no pixel difference between the image block to be inspected and the standard image block; The pixels included therein may not be empty, that is, there is a pixel difference between the image block to be checked and the standard image block.

It should be understood that, after the N image blocks to be inspected and the N standard image blocks are processed by the defect segmentation model, N defect segment image blocks are obtained.

Further, the N defect segmented image blocks and the corresponding N candidate defect feature maps are intersected to determine whether each defect segmented image block has a surface defect.

Finally, the surface defect judgment results of each defect segmentation image block are screened to determine whether the product to be inspected has surface defects.

It should be noted that when the embodiment of the present application implements the above defect detection method, its implementation principle and technical effect can refer to the relevant content of step 210-step 240 in the previous embodiment, which will not be repeated here.

It should be understood that although the various steps in the flow charts involved in the above embodiments are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flow charts involved in the above embodiments may include multiple steps or stages, and these steps or stages are not necessarily executed at the same time, but may be executed at different times. These steps Or the execution order of the stages is not necessarily performed sequentially, but may be executed in turn or alternately with other steps or at least a part of steps or stages in other steps.

Based on the above-mentioned surface defect detection method and using the same technical idea, the embodiment of the present application also provides a surface defect detection device for implementing the above-mentioned surface defect detection method. The solution to the problem provided by the device is similar to the implementation described in the above-mentioned method embodiments, so the specific limitations in one or more embodiments of the surface defect detection device provided below can be referred to above for surface defect detection The limitation of the method will not be repeated here.

In an exemplary embodiment, as shown in FIG. 8, the surface defect detection device 800 includes:

An image acquisition module 810, configured to acquire a surface image to be inspected and a standard surface image of the product to be inspected;

The image segmentation module 820 is used to perform image segmentation on the surface image to be inspected and the standard surface image, and obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image; the image block to be inspected and the standard image block The number of image blocks is the same, and the image areas where the image blocks to be checked and the standard image blocks are located in the corresponding surface images are also the same;

The defect segmentation module 830 is configured to input the image block to be inspected and the standard image block into the pre-trained defect segmentation model to obtain at least one defect segmented image block; each defect segmented image block is used to represent an image block to be inspected and Pixel difference information between a standard image block.

The defect detection module 840 is configured to segment the image block based on the defect, and determine whether there is a surface defect in the product to be inspected.

In a possible implementation, the image segmentation module 820 includes:

The pixel processing unit is used to subtract the pixel values of pixels at the same position in the surface image to be checked and the standard surface image to obtain an intermediate surface image;

A region analysis unit, configured to perform connected region analysis on the intermediate surface image to obtain at least one candidate defect feature map;

The image segmentation unit is configured to perform image segmentation on the surface image to be inspected and the standard surface image according to the region position information corresponding to the feature map of the candidate defect in the intermediate surface image, to obtain image blocks to be inspected and standard image blocks.

In a possible implementation manner, the defect detection module 840 includes:

an information acquisition unit, configured to acquire pixel intersection information between defect segmentation image blocks and candidate defect feature maps;

The first determination unit is configured to determine that there is no surface defect on the surface of the product to be inspected if the pixel intersection information is empty;

The second determining unit is configured to determine that surface defects exist on the surface of the product to be inspected if the pixel intersection information is not empty.

In a possible implementation, the defect segmentation model includes an encoder and a decoder; the defect segmentation module 830 includes:

The feature extraction unit is used to input the image block to be inspected and the standard image block into the encoder, and obtain the first multi-scale feature map of the image block to be inspected at various preset resolutions and the standard image block through the encoder. The second multi-scale feature map at multiple resolutions;

A feature fusion unit, configured to input the first multi-scale feature map and the second multi-scale feature map into the decoder, perform feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and output the defect Segment image blocks.

In a possible implementation manner, the image acquisition module 810 includes:

An image acquisition unit, configured to acquire an initial image to be inspected and an initial standard image of the product to be inspected;

An image registration unit, configured to perform image registration processing on the initial image to be inspected based on the initial standard image to obtain an intermediate image to be inspected;

The image processing unit is used to perform image preprocessing on the intermediate image to be inspected and the initial standard image to obtain the surface image to be inspected and the standard surface image.

In a possible implementation manner, the image processing unit is specifically used for:

Edge detection is performed on the intermediate image to be inspected and the initial standard image, and the first edge information of the intermediate image to be inspected and the second edge information of the initial standard image are obtained;

Based on the first edge information, edge expansion processing is performed on the intermediate image to be inspected to obtain a surface image to be inspected;

Based on the second edge information, edge expansion processing is performed on the initial standard image to obtain a standard surface image.

In a possible implementation, as shown in FIG. 9, the surface defect detection device 800 further includes:

The sample acquisition module 850 is used to obtain the training sample image of the inspected product; the training sample image includes a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used to indicate whether the sample image has a surface defect;

The sample pair preparation module 860 is used to generate a plurality of training sample pairs according to the training sample images; each training sample pair includes two sample images;

The loss calculation module 870 is configured to input a plurality of training sample pairs into the initial defect segmentation model to be trained, and calculate the training loss value of the target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

The model training module 880 is used to adjust the parameters of the initial defect segmentation model if the training loss value does not meet the preset convergence condition, and input multiple training sample pairs into the initial defect segmentation model after adjusting the parameters again until the training loss If the value satisfies the preset convergence condition, the training ends and a trained defect segmentation model is obtained.

In a possible implementation, the sample pair preparation module includes:

A sample combining unit is used to sequentially combine the standard sample image with a plurality of defective sample images to obtain a plurality of initial sample pairs;

The sample pair processing unit is used to randomly perform data enhancement processing on multiple initial sample pairs to obtain multiple training sample pairs;

Among them, data enhancement processing includes at least one of the following:

Randomly exchanging the two sample images in the initial sample pair to obtain multiple first sample pairs;

Perform pixel offset processing on any sample image in the initial sample pair according to a preset pixel offset range to obtain a plurality of second sample pairs;

Randomly copying the standard sample image in the initial sample pair as a defective sample image to obtain a plurality of third sample pairs;

Paste the pre-extracted image defects into the standard sample image of the initial sample pair to obtain a plurality of fourth sample pairs;

The plurality of training sample pairs includes at least one sample pair among an initial sample pair, a first sample pair, a second sample pair, a third sample pair, and a fourth sample pair.

In a possible implementation manner, the target loss function of the initial defect segmentation model includes a Dice loss function and a cross-entropy loss function, and the calculation coefficients of the Dice loss function and the cross-entropy loss function in the target loss function are different.

In a possible implementation manner, if the product to be inspected is a printed matter, the surface defects include shallow sticking defects.

It should be noted that each module in the surface defect detection device 800 shown in FIG. 8 and FIG. 9 can be fully or partially realized by software, hardware or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In addition, it should be understood that the embodiments of the present application may be fully or partially implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product comprises a computer program. When the computer program is loaded and run on the computer device, the processes or functions according to the embodiments of the present application will be generated in whole or in part.

Among them, the computer program may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program may be transmitted from a website site, a terminal, a server or a data center through Wired or wireless transmission to another website site, terminal, server or data center.

The computer-readable storage medium may be any available medium that can be accessed by a computer device, or a data storage device including a server, a data center, and the like integrated with one or more available media.

It should be understood that, the above are only specific implementation manners of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. All modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims (13)

1. A method of surface defect detection, comprising:

acquiring a surface image to be detected and a standard surface image of a product to be detected;

performing image segmentation on the surface image to be detected and the standard surface image to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block;

and determining whether the product to be detected has surface defects or not based on the defect segmentation image blocks.

2. The method according to claim 1, wherein the image segmentation of the surface image to be inspected and the standard surface image to obtain at least one image block to be inspected corresponding to the surface image to be inspected and at least one standard image block corresponding to the standard surface image comprises:

carrying out pixel value subtraction operation on pixels at the same positions in the surface image to be detected and the standard surface image to obtain an intermediate surface image;

analyzing a connected region of the intermediate surface image to obtain at least one candidate defect feature map;

and according to the corresponding region position information of the candidate defect characteristic diagram in the intermediate surface image, carrying out image segmentation on the surface image to be detected and the standard surface image to obtain the image block to be detected and the standard image block.

3. The method as claimed in claim 2, wherein said dividing image blocks based on said defects to determine whether said product to be inspected has surface defects comprises:

acquiring pixel intersection information between the defect segmentation image block and the candidate defect feature map;

if the pixel intersection information is empty, determining that no surface defect exists on the surface of the product to be detected;

and if the pixel intersection information is not null, determining that the surface of the product to be detected has surface defects.

4. The method of claim 1, wherein the defect segmentation model comprises an encoder and a decoder;

inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block, wherein the defect segmentation image block comprises:

inputting the image block to be detected and the standard image block into the encoder, and acquiring a first multi-scale feature map of the image block to be detected under multiple preset resolutions and a second multi-scale feature map of the standard image block under the multiple resolutions through the encoder;

inputting the first multi-scale feature map and the second multi-scale feature map into the decoder, performing feature fusion processing on the first multi-scale feature map and the second multi-scale feature map through the decoder, and outputting the defect segmentation image block.

5. The method according to any one of claims 1 to 4, characterized in that said acquiring of the surface image to be inspected and the standard surface image of the product to be inspected comprises:

acquiring an initial to-be-detected image and an initial standard image of the product to be detected;

based on the initial standard image, carrying out image registration processing on the initial image to be detected to obtain an intermediate image to be detected;

and respectively carrying out image preprocessing on the intermediate image to be detected and the initial standard image to obtain the surface image to be detected and the standard surface image.

6. The method of claim 5, wherein said image preprocessing said intermediate inspection image and said initial standard image to obtain said inspection surface image and said standard surface image comprises:

performing edge detection on the intermediate image to be detected and the initial standard image to obtain first edge information of the intermediate image to be detected and second edge information of the initial standard image;

performing edge expansion processing on the intermediate image to be detected based on the first edge information to obtain the surface image to be detected;

and performing edge expansion processing on the initial standard image based on the second edge information to obtain the standard surface image.

7. The method according to any one of claims 1 to 4, wherein the training process of the defect segmentation model comprises:

acquiring a training sample image of a detected product; the training sample image comprises a standard sample image and a plurality of defect sample images, each sample image carries a sample label, and the sample label is used for indicating whether the sample image has surface defects or not;

generating a plurality of training sample pairs according to the training sample images; each training sample pair comprises two sample images;

inputting the training sample pairs into an initial defect segmentation model to be trained, and calculating a training loss value of a target loss function of the initial defect segmentation model according to the output of the initial defect segmentation model;

and if the training loss value does not meet the preset convergence condition, adjusting parameters of the initial defect segmentation model, inputting the training sample pairs into the initial defect segmentation model after the parameters are adjusted again, and ending the training until the training loss value meets the preset convergence condition to obtain the trained defect segmentation model.

8. The method of claim 7, wherein generating a plurality of training samples from the training sample images comprises:

combining the standard sample image with the plurality of defect sample images in sequence to obtain a plurality of initial sample pairs;

performing data enhancement processing on the plurality of initial sample pairs randomly to obtain a plurality of training sample pairs;

wherein the data enhancement processing comprises at least one of:

randomly exchanging two sample images in the initial sample pair to obtain a plurality of first sample pairs;

performing pixel offset processing on any sample image in the initial sample pair according to a preset pixel offset range to obtain a plurality of second sample pairs;

randomly copying the standard sample images in the initial sample pairs into defect sample images to obtain a plurality of third sample pairs;

pasting pre-extracted image defects to the standard sample images of the initial sample pairs to obtain a plurality of fourth sample pairs;

the plurality of training sample pairs includes at least one of the initial sample pair, the first sample pair, the second sample pair, the third sample pair, and the fourth sample pair.

9. The method according to claim 7, wherein the objective loss function of the initial defect segmentation model comprises a Dice loss function and a cross entropy loss function, and the Dice loss function and the cross entropy loss function have different calculation coefficients in the objective loss function.

10. The method according to any one of claims 1 to 4, characterized in that the surface defects comprise low-tack defects if the product to be examined is a print.

11. A surface defect detecting apparatus, comprising:

the image acquisition module is used for acquiring a surface image to be detected and a standard surface image of a product to be detected;

the image segmentation module is used for carrying out image segmentation on the surface image to be detected and the standard surface image to obtain at least one image block to be detected corresponding to the surface image to be detected and at least one standard image block corresponding to the standard surface image; the number of the to-be-detected image blocks is the same as that of the standard image blocks, and the image areas of the to-be-detected image blocks and the standard image blocks in the corresponding surface images are also the same;

the defect segmentation module is used for inputting the to-be-detected image block and the standard image block into a defect segmentation model trained in advance to obtain at least one defect segmentation image block; each defect division image block is used for representing pixel difference information between one to-be-detected image block and one standard image block;

and the defect detection module is used for dividing the image blocks based on the defects and determining whether the product to be detected has surface defects.

12. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when retrieving from the memory and executing the computer program implements the steps of the method of any of the preceding claims 1 to 10.

13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of one of the preceding claims 1 to 10.

Images