KR102818095B1 - Method of detecting defect of manufactured product based on high-resolution image using autoencoder and convolutional neural network - Google Patents

Abstract

The present invention relates to a method for detecting defects in manufactured products based on high-resolution images using an autoencoder and a convolutional deep neural network. According to the present invention, the method includes: a step of dividing a high-resolution manufactured product image captured by an image capturing device into a plurality of grid images; a step of extracting restored images of the plurality of grid images based on a pre-trained autoencoder; a step of extracting a differential image through a difference operation between the restored image and the grid image; a step of extracting a defect candidate image patch based on the differential image; a step of filtering defect candidates from the defect candidate image patches; and a step of determining whether the filtered defect candidate image patches are defective based on a pre-trained convolutional deep neural network.

Description

{Method of detecting defect of manufactured product based on high-resolution image using autoencoder and convolutional neural network}

The present invention relates to a method for detecting defects in manufactured products based on high-resolution images using an autoencoder and a convolutional deep neural network. More specifically, the present invention relates to a method for detecting defects in manufactured products based on high-resolution images using an autoencoder and a convolutional deep neural network, which detects defect candidates in an image of a manufactured product using an autoencoder and determines whether the detected defect candidates are defective using a convolutional deep neural network in an image-based defect detection technology for improving the quality of manufactured products.

Recently, defect detection based on images of manufactured products captured by imaging devices such as cameras has been widely performed. Such image-based defect detection can be mainly performed by image recognition technology, artificial intelligence technology, etc.

However, conventional image-based manufacturing defect detection techniques using artificial intelligence technology have the following problems.

First, the class imbalance problem, which is a situation where abnormal data is insufficient compared to normal data in manufacturing defect detection, is very frequent. In addition, even for the same manufacturing product, the amount of data is often different depending on the shooting location, which leads to a data imbalance problem. These problems are the cause of lowering the performance of image-based manufacturing defect detection techniques using conventional artificial intelligence technology.

Second, in the case of high-resolution manufactured product images, the number of defective pixels exists in a very small proportion compared to the total number of pixels in the manufactured product image. Therefore, most of the conventional technologies based on deep neural networks have the problem that the model is likely to easily overfit to the learning data set when learning is performed based on high-resolution images.

Third, when using data with a large amount of features, such as high-resolution images, for learning artificial intelligence models, there is a problem that computing resources may reach their limits very quickly.

Fourth, noise such as scratches, stains, and dust inevitably occur in manufactured products during the manufacturing process. If these noises and defects cannot be accurately distinguished, it can lead to the problem of over-detection, in which normal products are mistakenly judged as defective products during defect detection.

Thus, according to the present invention, in an image-based defect detection technology for improving the quality of manufactured products, an autoencoder is utilized to detect a defect candidate in an image of a manufactured product, and a convolutional deep neural network is utilized to determine whether the detected defect candidate is defective. The present invention provides a high-resolution image-based defect detection method for manufactured products using an autoencoder and a convolutional deep neural network.

According to an embodiment of the present invention for achieving such a technical task, a method for detecting a defect in a manufactured product based on a high-resolution image using an autoencoder and a convolutional deep neural network is provided, the method including: a step of dividing a high-resolution manufactured product image captured by an image capturing device into a plurality of grid images; a step of extracting a restored image of the plurality of grid images based on a pre-trained autoencoder; a step of extracting a differential image through a difference operation between the restored image and the grid image; a step of extracting a defect candidate image patch based on the differential image; a step of filtering defect candidates from the defect candidate image patches; and a step of determining whether the filtered defect candidate image patches are defective based on a pre-trained convolutional deep neural network.

The step of extracting the above defect candidate image patch may extract defect candidate pixels from the difference image, and extract the defect candidate image patch based on the extracted defect candidate pixels. In this case, the defect candidate pixels may be extracted using a preset threshold value from the difference image, or the top N pixels (N is a natural number) with high pixel intensities may be extracted as the defect candidate pixels according to the pixel intensities.

The step of filtering the above defect candidates may filter the defect candidates by comparing the numerical values between adjacent pixels in the defect candidate image patch, or by comparing the defect size for each inspection area or statistical characteristics of the defect candidates with previously stored criteria.

According to the present invention, class imbalance and data imbalance problems can be solved by extracting a defective candidate image patch and performing defect detection based on an autoencoder, which is one of the unsupervised learning anomaly detection models that uses only normal data for learning.

In addition, by extracting defect candidate image patches with a relatively small number of features from a high-resolution image and using them to learn a deep neural network model, the overfitting problem that may occur in conventional techniques can be solved.

In addition, the problem of over-consumption of computing resources can be solved by significantly reducing the number of features that the model needs to learn through segmentation of high-resolution images and extraction of defect candidate image patches.

Furthermore, the performance of defect detection can be improved by using the defect candidate image patches extracted through the unsupervised learning-based anomaly detection model in the supervised learning-based classification model, and the over-detection problem can be solved by a filtering task that compares the defect candidate image patches with the defect conditions of the inspection area of the product.

Figure 1 is a drawing for explaining inspection equipment for detecting defects in manufactured products.
FIG. 2 is a flowchart of a learning process in a high-resolution image-based manufacturing defect detection method using an autoencoder and a convolutional deep neural network according to one embodiment of the present invention.
FIG. 3 is a flowchart of an inference process in a high-resolution image-based manufacturing defect detection method using an autoencoder and a convolutional deep neural network according to one embodiment of the present invention.
FIG. 4 is a drawing for explaining a step of dividing a high-resolution manufactured product image into a plurality of grid images according to an embodiment of the present invention.
FIG. 5 is a diagram for explaining a step of extracting a differential image of a high-resolution manufactured product image using an autoencoder according to one embodiment of the present invention.
FIG. 6 is a diagram for explaining a step of extracting a defect candidate image patch based on a differential image and determining whether the defect candidate image patch is defective based on a convolutional deep neural network according to one embodiment of the present invention.
Figures 7 to 9 are drawings for explaining one embodiment of the present invention.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings. In this process, the thickness of lines and the size of components illustrated in the drawings may be exaggerated for the sake of clarity and convenience of explanation.

In addition, the terms described below are terms defined in consideration of their functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definitions of these terms should be made based on the contents throughout this specification.

Figure 1 is a drawing for explaining inspection equipment for detecting defects in manufactured products.

Referring to Fig. 1, a manufactured product to be inspected can be fed into the inspection equipment through an input unit (110). The input unit (110) can identify the manufactured product to be inspected by recognizing a barcode attached to the manufactured product, for example, through a barcode scanner, and move the manufactured product to the photographing unit (120) through a conveyor belt. The identification information of the manufactured product to be inspected identified in the input unit (110) can be managed and stored in connection with data generated in each process of the inspection equipment.

The photographing unit (120) can capture an image of a manufactured product moved through the input unit (110). The photographing unit (120) is configured to include, for example, a camera for capturing an image of a manufactured product, lighting for adjusting the brightness of the image, etc., and can capture and store an image of a manufactured product according to a control signal of a PLC (150). In this case, the photographing unit (120) is additionally provided with a means for moving, reversing, or rotating the manufactured product, and can capture a desired surface of the manufactured product by controlling the position and direction of the manufactured product through this, and store the captured image in the Edge PC (140).

The judgment unit (130) can output the product as a good product or a defective product based on the judgment result of the product to be inspected. To this end, the artificial intelligence model can receive the image captured by the photographing unit (120) and judge whether the product is normal or defective, and the judgment result can be stored in the Edge PC (140) together with the identification information of the product. Accordingly, the judgment unit (130) can identify the product moving through the conveyor belt through the barcode scanner and check the judgment result of the identified product to divide the product into a good product or a defective product.

According to an embodiment of the present invention, an artificial intelligence model for determining whether a manufactured product is normal or defective may be a model that combines an unsupervised learning-based anomaly detection model and a supervised learning-based classification model.

For example, it can be configured to extract a defect candidate image patch based on an autoencoder, which is one of the unsupervised learning anomaly detection models, and determine whether the defect candidate image patch is a defect based on a convolutional deep neural network, which is one of the supervised learning-based classification models.

Here, the autoencoder refers to a deep neural network in which the sizes of input data and output data are the same, and is mainly used to learn low-dimensional representations from high-dimensional input data. The autoencoder may be composed of an input layer that accepts input data, an encoder that performs an operation to derive a low-dimensional latent vector from the high-dimensional input data, a decoder that generates restoration data from the derived latent vector, and an output layer that derives output data having the same size as the input data. According to the present invention, the autoencoder can restore the input data similarly to the normal data by learning the main features of normal data. Such autoencoders may be provided in the number of grid images of each manufactured product image.

Meanwhile, a convolutional deep neural network is a type of deep neural network mainly used to analyze visual images, and can be composed of an input layer that receives an input vector, a hidden layer that performs a convolution operation of the output value of the previous neural network and the current neural network weight, and an output layer that derives the final output value calculated from each neural network, and can learn the features of the input through the convolution operation of the input vector and the current neural network weight. This convolutional deep neural network is used to perform defect detection by learning the patterns of normal images and defective images.

Hereinafter, with reference to FIGS. 2 to 9, a learning and inference method of an artificial intelligence model according to an embodiment of the present invention and an embodiment thereof will be specifically described.

FIG. 2 is a flowchart of a learning process in a high-resolution image-based manufacturing defect detection method using an autoencoder and a convolutional deep neural network according to one embodiment of the present invention.

Referring to Fig. 2, first, in order to create a data set for model learning, a high-resolution normal manufactured product image is divided into multiple grid images (S210).

Here, high resolution means a resolution of 1920 x 1080 pixels or higher, which is the standard for FHD (Full High-Definition) resolution. High-resolution images of manufactured products can be captured by an imaging device, and are data saved in image file formats such as jpg, png, and bmp, and the pixel values can have values between 0 and 255, which are general pixel values. In addition, in order to capture all inspection areas of the manufactured product, multiple imaging devices can be used to capture images of the manufactured product from various angles and use them.

In addition, the grid image means small images divided into a grid shape from the input image. The resolution of the grid image can have various values such as 256 x 256, 512 x 512, etc., depending on the resolution of the high-resolution normal manufactured product image, and can be flexibly adjusted depending on the status of computing resources and defect detection performance. In addition, the grid image can include all inspection areas of the manufactured product image.

After performing grid division on all manufactured product images, a data set for autoencoder learning is generated using multiple grid images (S220). Since the autoencoder used in the embodiment of the present invention is an unsupervised learning-based model for learning the characteristics of normal data and generating restored data, only normal data is used to generate grid images, which are a data set for autoencoder learning.

Afterwards, autoencoder training is performed using the generated data set (S230).

Thereafter, based on the learned autoencoder, a defect candidate image patch is extracted from the input grid image (S240). The extracted defect candidate image patches are used to generate a data set for learning a convolutional deep neural network (S250). Since the convolutional deep neural network used in the embodiment of the present invention is a classification model based on supervised learning, both normal data and defect data are required for learning. Therefore, the normal image patch data can be extracted from a normal grid image, and the defect candidate image patch data can be extracted from defect candidate pixels extracted from an abnormal grid image, or by cutting out adjacent pixels centered on the acquired defect coordinates. The normal image patches and defect candidate image patches extracted in this way are used as a data set for learning a convolutional deep neural network.

Afterwards, training of a convolutional deep neural network is performed using the generated data set (S260).

FIG. 3 is a flowchart of an inference process in a high-resolution image-based manufacturing defect detection method using an autoencoder and a convolutional deep neural network according to one embodiment of the present invention.

Referring to FIG. 3, first, a high-resolution image of a manufactured product is captured using an image capturing device (S310), and the captured high-resolution image of a manufactured product is divided into a plurality of grid images (S320).

FIG. 4 is a drawing for explaining a step of dividing a high-resolution manufactured product image into a plurality of grid images according to an embodiment of the present invention.

Referring to FIG. 4, when a high-resolution manufactured product image as shown on the left is input as an input image, it can be divided into a plurality of grid images having a preset resolution to obtain a plurality of divided manufactured product images, i.e., grid images.

Thereafter, a restored image of the grid image is extracted based on the learned autoencoder (S330), and a differential image, which is the difference between the input image and the restored image, is extracted (S340). Here, the restored image means an image derived from the input image (i.e., the grid image) through a generative model based on an autoencoder consisting of a deep neural network, and since it is a value calculated from the weights of the generative model and the input image, it may not have a value between 0 and 255. In addition, the differential image means an image derived by calculating the difference between the input image and the restored image, and each pixel of the differential image may include a positive value, a negative value, or an absolute value that can be calculated by calculating the difference between the input image and the restored image. In addition, since the pixel value of the differential image is a value derived from the difference between the input image and the restored image, if the numerical value is converted to an absolute value, it may not have a value between 0 and 255.

FIG. 5 is a diagram for explaining a step of extracting a differential image of a high-resolution manufactured product image using an autoencoder according to one embodiment of the present invention.

Referring to Fig. 5, multiple high-resolution manufacturing images captured by multiple image capturing devices are each divided into multiple grid images and input to an autoencoder. The pre-trained autoencoder can extract a restored image that does not include noise and defects from the input image. A difference image is extracted by calculating the difference between the restored image and the input image, and the difference image may include noise and actual defective pixels.

Thereafter, a defect candidate image patch is extracted based on the difference image (S350), the defect candidates are filtered based on the manufacturing inspection area criteria (S360), and then whether the defect candidate image patch is defective is determined based on the learned convolutional deep neural network (S370).

FIG. 6 is a diagram for explaining a step of extracting a defect candidate image patch based on a differential image and determining whether the defect candidate image patch is defective based on a convolutional deep neural network according to one embodiment of the present invention.

Referring to Fig. 6, first, defect candidate pixels, which are pixels likely to be utilized for defect detection based on the differential image, are extracted. Here, the defect candidate pixels are pixels extracted from the differential image, and mean optimal pixels likely to be utilized for defect detection. The defect candidate pixels mean the pixels themselves, and may include information about the characteristics that the pixels may have, such as absolute coordinate information of the pixels or pixel intensity. For example, defect candidate pixels may be extracted by using a preset threshold value from the differential image, or by extracting the top N pixels (N is a natural number) with high pixel intensities as defect candidate pixels according to pixel intensity.

Next, a defect candidate image patch can be extracted based on the extracted defect candidate pixels. Here, the defect candidate image patch means an image in which some pixels of adjacent input images and differential images are extracted based on the coordinates of the defect candidate pixels. In other words, the defect candidate image patch is a small-sized image, such as 64 x 64, 128 x 128, etc., and is composed of a relatively smaller number of pixels than the input image. For example, since the defect candidate pixels include pixel intensity, coordinate information, etc., the coordinate information can be used to extract surrounding pixels adjacent to the defect candidate pixels in the input image and the differential image to generate a defect candidate image patch. By extracting the defect candidate image patch from the input image and the differential image in this way, the high dimensionality of the high-resolution manufactured product image can be reduced.

Since the defect candidate image patches generated as described above are likely to contain noise and actual defects, a task of filtering defect candidates based on the manufacturing inspection area criteria can be performed to separate them. For example, defect candidates can be filtered by comparing the numerical values between adjacent pixels, or by comparing the defect sizes by inspection area, statistical characteristics of defect candidates, etc.

After that, the convolutional deep neural network can determine whether the defect candidate image patch on which filtering has been performed is defective through convolutional operations. For example, the defect candidate image patch converted into the input vector of the convolutional deep neural network derives a softmax output value through the convolutional operation of the hidden layer, and the softmax output value can be converted into a normal or defective legend based on a preset threshold value, and whether there is a defect can be inferred based on this.

The process described above with reference to FIG. 3 can be performed identically for all manufactured product images captured by the imaging device, and if all images captured for one manufactured product are inferred to be normal, the manufactured product can be judged to be a normal manufactured product without any defects.

The method according to the present invention described above with reference to FIGS. 2 and 3 can be performed by a processing device capable of learning and inferring an artificial intelligence model based on input image data.

Figures 7 to 9 are drawings for explaining one embodiment of the present invention.

Referring to FIGS. 7 to 9, first, when a high-resolution manufactured product image is input as shown in FIG. 7, it can be divided into multiple grid images.

Afterwards, as shown in Fig. 8, a differential image for each grid image is extracted based on the learned autoencoder, and a defect candidate image patch is extracted based on this, and then the presence or absence of a defect can be determined through convolutional deep neural network learning.

FIG. 9 is an example of a final result when defect detection is performed according to an embodiment of the present invention. In the defect detection result image shown on the right, white squares represent areas of a segmented grid, blue squares represent defect candidate image patches, green squares represent image patches filtered out as defect candidates according to defect candidate filtering, and red squares represent image patches finally determined to be defects.

Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention should be determined by the technical idea of the following patent claims.

Claims (4)

A step of dividing a high-resolution product image captured by an imaging device into a plurality of grid images;
A step of extracting a restored image of the plurality of grid images based on the learned autoencoder;
A step of extracting a difference image through a difference operation between the above restored image and the above grid image;
A step of extracting a defect candidate image patch based on the above difference image;
A step of filtering out defect candidates from the above defect candidate image patches; and
It includes a step of determining whether a filtered defect candidate image patch is defective based on a pre-trained convolutional deep neural network.
The step of extracting the above defect candidate image patch is:
A method for detecting defects in manufactured products based on high-resolution images, utilizing an autoencoder and a convolutional deep neural network, for extracting defect candidate pixels from a difference image using a preset threshold value, or extracting the top N pixels (N is a natural number) with high pixel intensities as the defect candidate pixels according to pixel intensity, and extracting a defect candidate image patch based on the extracted defect candidate pixels. delete delete In paragraph 1,
The step of filtering the above defect candidates is:
A method for detecting defects in manufactured products based on high-resolution images, utilizing an autoencoder and a convolutional deep neural network, for filtering defect candidates by comparing numerical values between adjacent pixels in the defect candidate image patches or by comparing the defect size or statistical characteristics of the defect candidates by inspection area with previously stored criteria.