TWI745767B - Optical inspection secondary image classification method - Google Patents

Abstract

The optical inspection secondary image classification method of the present invention is to pair good product images and defective product images into an image group marked 1, and pair good product images and screened images into an image marked 0 Group, one by one input class of each image group through the network architecture for image classification model training, and after the image classification model is generated and verified, the input image prediction classification results are recorded, which will not be accurately predicted and classified After images are amplified, paired, and labeled, they are re-entered into a neural network architecture such as the image classification model for enhanced training, and then another image classification model is generated. Through at least one enhanced training, excessive screening can be effectively reduced. Improve the accuracy and efficiency of detection equipment, and reduce the cost of secondary screening.

Description

Optical inspection secondary image classification method

The present invention is related to the deep learning image classification technology, and particularly refers to an optical detection secondary image classification method that is particularly suitable for applying to the product after the product passes the automatic optical detection, and then discriminates the product as a good product or a defective product.

Automatic Optical Inspection (AOI) is a non-contact method that uses machine vision technology to capture images for analysis, and then determine whether the product (finished or semi-finished) is defective, and can be deployed in the middle of the node of the automated processing line Testing, without affecting production capacity, is a testing method widely used in the industry, and it is a necessary investment with a high proportion in the manufacturing process of the circuit board and display panel industry.

Taking the most widely used circuit board processing production line as an example, the AOI inspection system process is mainly to use optical instruments to scan the circuit board to be inspected to obtain images, and then the system compares the qualified parameters in the database and passes the computer image The processing technology detects whether there are defects on the circuit board.

Usually the circuit board processing production line has extremely high yield requirements, so the AOI parameters are set very strict, and the optical principle is easy to make the AOI sensitive to light and shadow interference, so as long as there is some slight external light and shadow influence, the equipment is It will be automatically judged as defective, which causes AOI inspection to often face the phenomenon of over-screening (misjudgment of good products as defective).

It is known that the image data can be processed through artificial intelligence and deep learning neural networks, and the algorithm that uses the law to predict unknown data can identify unknown defects and assist the subsequent optimization of AOI detection, thereby improving the accuracy of detection equipment Rate, reduce the cost of manual second screening.

The reason why the image classification model based on the neural network architecture can fully and correctly learn the key to distinguishing images, in addition to the need to provide a large amount of labeled image data in the training process of deep learning (Training) In addition, improved neural network architecture may be a feasible approach.

However, the parameter options that can be adjusted in the neural network architecture are very cumbersome, and it is not easy to systematically find the best permutation and combination among the cumbersome parameter options; therefore, how to improve the image classification model under the condition of a limited number of samples The accuracy of the discrimination has long been a problem that the industry and academia desperately want to solve.

In view of this, the present invention is to provide a secondary image classification method for optical detection that can effectively reduce excessive screening and improve the accuracy of predictive classification of the image classification model for automatic optical detection, as its main purpose.

The optical inspection secondary image classification method of the present invention is used in the image classification judgment of a good product or defective product provided by a processing line equipped with an automatic optical inspection system; the optical inspection secondary image classification method , It includes the following steps: (a) The step of providing an image feature automatic identification system, an image feature automatic identification system integrates a storage unit, a processing unit and an image capture unit, and the storage unit is loaded with Multiple good images, multiple over-screened images, and multiple defective images; (b) the step of creating a training data set, at least creating a training data set in the storage unit of the image feature automatic identification system, the training data The set includes a plurality of first image groups marked as 1 paired by any good product image and any defective product image in the storage unit, and a plurality of first image groups marked as 1 by any good product image in the storage unit and any one that has been screened. The image pairing becomes the second image group marked as 0; (c) the step of creating a verification data set, a verification data set is created in the storage unit of the image feature automatic identification system, and the verification data set includes at least a plurality of Good product images, multiple defective product images; (d) a step of establishing a neural network-like architecture, in the storage unit of the image feature automatic identification system, creating a processing unit to perform operations on the input image A neural network architecture for predicting and categorizing good or defective products. This neural network architecture has two feature core modules that can be operated independently and a fully connected model connected to the two feature core modules. (E) The image classification model training step, through the image capturing unit, input the second image group with the plural number as label 0 and the second image group with the plural number as label 1 in the training data set one by one This type of neural network architecture, after comparing the prediction and classification output results of the type of neural network architecture, adjusts the characteristics and weights of the type of neural network architecture, and stores the adjustments that successfully reduce the error, and then produces a better understanding of the input An image classification model for predicting and classifying good or defective images of the image; (f) The image classification model verification step, each time an image classification model is generated, the image capture unit uses one of the verification data sets The multiple good images and multiple defective images are input into the image classification model generated this time one by one, and the image classification model generated this time predicts and classifies each input good image and each defective image, thereby Evaluate the prediction classification accuracy of the image classification model generated this time; (g) Statistical classification prediction error distribution and enhanced training steps, each time the image classification model is generated, and when the image classification model generated this time is verified, Record the input of the predicted classification results of each of the good image and each of the defective image, and calculate the predicted classification error distribution, according to the first image image and the second image group in the image classification model training step The pairing, labeling mode, and training mode are used to increase the quantity of each image that is actually a good image but cannot be accurately predicted and classified. The images that have been accurately predicted and classified are produced after the number is increased After re-pairing and labeling each defective product image, input the neural network architecture to which the generated image classification model belongs to strengthen training, and then generate another image classification model; (h) Model confirmation step After performing step (f) to step (g) at least once, confirm at least one image classification model among the image classification models, and then use the at least one confirmed image classification model to process the The products provided by the production line are distinguished by image classification of good or defective products.

According to the above technical features, in the optical detection secondary image classification method, in the confirming model step, after repeating the step (f) to the step (g) at least once, confirming one of the image classification models An image classification model with better accuracy uses the confirmed image classification model to classify images of good or defective products provided by the processing line.

According to the above technical features, in the optical detection secondary image classification method, in the confirming model step, after repeating the step (f) to the step (g) at least once, confirm the plural number in the image classification models An image classification model with better accuracy, and the confirmed image classification models are combined, and the combined image classification models are used to map the products provided by the processing line for good or defective products Like classification and discrimination.

According to the above-mentioned technical characteristics, the number of good product images in the training data set and the verification data set is determined by the automatic optical detection system as good products, and the original images of good products that are manually confirmed to be good are amplified by the number of good products; the training data set The number of over-screened images is achieved by amplifying multiple original images of good products that were judged to be defective by the automatic optical inspection system but confirmed to be good products manually; the number of defective images in the training data set and the verification data set It is achieved by amplifying a plurality of original images of defective products that are identified as defective by the automatic optical inspection system and manually confirmed as defective.

According to the above technical features, each of the good product images in the verification data set is not exactly the same as each of the good product images in the training data set, and each of the defective product images in the verification data set is the same as the training data The images of the defective products in the set are not exactly the same.

According to the above-mentioned technical characteristics, the number of good product images in the training data set and the verification data set is determined by the automatic optical detection system as good products, and the original images of good products that are manually confirmed to be good are amplified by the number of good products; the training data set The number of over-screened images is achieved by amplifying multiple original images of good products that were judged to be defective by the automatic optical inspection system but confirmed to be good products manually; the number of defective images in the training data set and the verification data set It is achieved by amplifying the original images of defective products identified as defective by the automatic optical inspection system and manually confirmed as defective; and, each of the good product images in the verification data set and the training data set Each of the good product images is not exactly the same, and each of the defective product images in the verification data set is not exactly the same as each of the defective product images in the training data set.

According to the above-mentioned technical characteristics, the predicted output value of each of the neural network architectures of each of the image classification models for each input image is presented in the output type of 0 or 1.

According to the above technical features, the predicted output value of each of the neural network architectures of each of the image classification models for each input image is presented in the output type with the probability of prediction being 0 and the probability of prediction being 1.

According to the above technical features, in the optical detection secondary image classification method, in the step of building a neural network architecture, a storage unit of the image feature automatic identification system is created for the input image Convolutional neural network (CNN) for predicting and classifying good or defective products.

The optical detection secondary image classification method disclosed in the present invention pairs a good product image and a defective product image into an image group marked 1, and a good product image and a screening image are paired to form an image group. For image groups marked as 0, each image group is input into the network architecture one by one for image classification model training, and after the image classification model is generated and verified, the input image prediction classification results are recorded, and the After the images that cannot be accurately predicted and classified are amplified, matched, and labeled, they are re-entered into a neural network architecture such as the image classification model belongs to for enhanced training, and another image classification model is generated. Through at least one enhanced training, it is effective Reduce excessive screening and improve the predictive classification accuracy of the image classification model. It is especially suitable for applications after the product has passed the automatic optical inspection to determine the product as a good product or a defective product again, and use a relatively more active and reliable method to improve the performance of the testing equipment. Identify the correct rate and efficiency, and reduce the cost of secondary screening.

The present invention mainly provides automatic optical inspection, an optical inspection secondary image classification method that can effectively improve the accuracy of image classification. The optical inspection secondary image classification method of the present invention is used in a processing with an automatic optical inspection system The image classification of the product provided by the production line is good or defective; as shown in Figures 1 to 3, the optical inspection secondary image classification method 100 includes the following steps:

(a) Provide an image feature automatic identification system step. An image feature automatic identification system 10 integrates a storage unit 11, a processing unit 12, and an image capture unit 13, and the storage unit 11 is loaded with a plurality of Good product images, multiple over-screened images, and multiple defective product images; in implementation, the processing unit 12 may be a graphics processing unit (GPU).

(b) The step of creating a training data set is to create at least a training data set 20 in the storage unit 11 of the image feature automatic identification system 10, and the training data set 20 includes a plurality of good images from any one of the storage units 11 The image and any defective product image are paired into the first image group marked as 1 (as shown in Figure 5), and the plural number is matched by any good product image and any screened image in the storage unit 11. The second image group marked as 0 (as shown in Figure 6).

(c) The step of establishing a verification data set is to create a verification data set 30 in the storage unit 11 of the image feature automatic identification system 10, and the verification data set 30 includes at least a plurality of good product images and a plurality of defective product images.

(d) A step of establishing a neural network-like architecture, in the storage unit 11 of the image feature automatic identification system 10, creating an operation executed by the processing unit 12 for predicting the quality or defect of the input image Classified neural network architecture 40, this neural network architecture 40 has two feature core modules 41 that can be operated independently and a fully connected module 42 connected to the two feature core modules 41; In implementation, a convolutional neural network architecture (Convolutional neural network, which is used to predict and classify the input image for good or defective products) can be established in the storage unit 11 of the image feature automatic identification system 10 CNN).

(e) The image classification model training step, through the image capturing unit 13, the second image group with the plural number labeled as label 0 and the second image group with the plural number label as label 1 in the training data set 20 are input one by one This type of neural network architecture 40 adjusts the characteristics and weights of the type of neural network a-channel architecture 40 after comparing the prediction and classification output results of the type of neural network architecture 40, and stores the adjustments that successfully reduce the error, and then produces a better one. An image classification model for predicting and classifying input images with good or defective products; when implemented, two images in each first image group or second image group are each input neural network architecture Any one of the feature core modules 41 of 40, then the fully connected module 42 completes the image prediction and classification output (as shown in Figure 3).

(f) The image classification model verification step. Each time an image classification model is generated, the multiple good product images and the multiple defective product images in the verification data set 30 are inputted one by one through the image capture unit 13 The generated image classification model is used to predict and classify each input good product image and each defective product image by the generated image classification model, so as to evaluate the prediction classification accuracy of the image classification model generated this time.

(g) Statistical classification prediction error distribution and enhanced training steps, each time an image classification model is generated, and when the image classification model generated this time is verified, each input good image and each defective product are recorded The predicted classification result of the image, and the statistical prediction of the classification error distribution, according to the pairing, labeling mode and training mode of the first image image and the second image group in the image classification model training step, the actual is Good product images but cannot be accurately predicted and classified by the number of images that are generated by the number amplification of each good product image, and each of the actual defective images but cannot be accurately predicted and classified by the number of images generated after amplifying the number After re-pairing and labeling each defective product image, it is re-input to the neural network architecture to which the generated image classification model belongs to strengthen training, and then another image classification model is generated.

(h) The step of confirming the model, after performing steps (f) to (g) at least once, confirming at least one image classification model among the image classification models, and then using the at least one confirmed image The image classification model classifies the products provided by the processing line as good or defective images.

In the optical detection secondary image classification method of the present invention, in the confirming model step, after repeating the steps (f) to (g) at least once, an accuracy can be confirmed among the image classification models A better image classification model uses the confirmed image classification model to classify images of good or defective products provided by the processing line.

In the optical detection secondary image classification method of the present invention, in the step of confirming the model, after repeating the step (f) to the step (g) at least once, the plural accuracy of the image classification models can be confirmed A better image classification model, and combine the confirmed image classification models, and use the combined image classification models to classify the products provided by the processing line for good or defective images Discriminate.

Because of the optical inspection secondary image classification method of the present invention, a good product image and a defective product image are paired to form an image group labeled 1, and a good product image and a screening image are paired to form a label If the image group is 0, input each image group one by one through the network architecture for image classification model training, and after the image classification model is generated and verified, the input image prediction classification results will be recorded. After accurate prediction and classification of images are amplified, matched, and labeled, they are re-entered into a neural network architecture such as the image classification model belongs to for enhanced training, and then another image classification model is generated. Through at least one enhanced training, it can be effective Reduce excessive screening, improve the predictive classification accuracy of the image classification model, use relatively more active and reliable means to improve the accuracy and efficiency of the detection equipment, and reduce the cost of secondary screening.

As shown in Figures 1 and 4, the optical detection secondary image classification method of the present invention reveals the number of good images in the training data set 20 and the verification data set 30 in various possible implementation modes. It can be achieved by amplifying the original images of good products that are judged to be good by the automatic optical detection system and manually confirmed as good; the number of over-screened images in the training data set 20 can be multiplied by the automatic optical detection system The product is judged to be defective but is achieved by augmenting the original image of the good product that is manually confirmed as good; the number of defective product images in the training data set 20 and the verification data set 30 is determined by the automatic optical inspection system to be a defective product and Achieved by augmenting the original image of the defective product that was manually confirmed as a defective product.

And, in the optical detection secondary image classification method of the present invention, each of the good image in the verification data set 30 and each of the good images in the training data set 20 in the various possible implementation modes described above The images are not exactly the same, each of the defective product images in the verification data set 30 and each of the defective product images in the training data set 20 are not completely the same.

Of course, the optical detection secondary image classification method of the present invention, in the various possible implementation modes, and the number of good images in the training data set 20 and the verification data set 30 can be determined by the automatic The optical inspection system judges the product as good and is achieved through the amplification of the original image of the good product that has been manually confirmed as good; The original image of the good product confirmed as good is amplified; the number of defective product images in the training data set 20 and the verification data set 30 is determined by the automatic optical inspection system as defective and manually confirmed as defective The original image of the defective product is amplified; and, each of the good product images in the verification data set 30 is not exactly the same as each of the good product images in the training data set 20, and each of the good product images in the verification data set 30 It is preferable that the defective product image is not exactly the same as each of the defective product images in the training data set 20.

In the optical detection secondary image classification method of the present invention, in various possible implementation modes, the predicted output value of each type of neural network architecture of each image classification model for each input image can be selected It is presented in the output type of 0 or 1. Of course, the predicted output value of each of the neural network architectures of each image classification model for each input image can also be selected based on the probability of the prediction being 0 and the prediction being 1. Probability of the output pattern is presented.

Specifically, the optical detection secondary image classification method disclosed in the present invention is to pair a good product image and a defective product image into an image group marked as 1, and combine a good product image and a screened image group. The images are paired into image groups marked as 0, and each image group is input into the network architecture for image classification model training, and after the image classification model is generated and verified, the input image prediction is recorded As a result of the classification, the number of images that cannot be accurately predicted and classified are amplified, paired, and labeled, and then re-entered into the neural network architecture of the image classification model to strengthen training, and then another image classification model is generated at least once Strengthen training, effectively reduce excessive screening, and improve the predictive classification accuracy of the image classification model. It is especially suitable for applications after the product has passed the automatic optical inspection to judge the product as a good product or a defective product again with a relatively more active and reliable method. Improve the accuracy and efficiency of detection equipment, and reduce the cost of secondary screening.

The above-mentioned embodiments are only to illustrate the technical ideas and features of the present invention, and their purpose is to enable those who are familiar with the art to understand the content of the present invention and implement them accordingly. When they cannot be used to limit the patent scope of the present invention, That is, all equal changes or modifications made in accordance with the spirit of the present invention should still be covered by the patent scope of the present invention.

100: Optical inspection secondary image classification method 10: Image feature automatic identification system 11: storage unit 12: Processing unit 13: Image capture unit 20: Training data set 30: Validation data set 40: Neural Network Architecture 41: Feature Core Module 42: Fully connected module

Figure 1 is a block diagram of the basic structure of the image feature automatic recognition system in the present invention. Figure 2 is the basic flow chart of the optical detection secondary image classification method of the present invention. Figure 3 is a schematic diagram of a possible implementation of the image classification model training process in the present invention. Figure 4 is a flow chart of the original image confirmation in one of the possible implementations of the present invention. Figure 5 is the first image group pairing structure diagram of the training data set in the present invention. Figure 6 is the second image group pairing structure diagram of the training data set in the present invention.

100: Optical inspection secondary image classification method

Claims (8)

An optical inspection secondary image classification method, which is used in the image classification judgment of a good or defective product provided by a processing line with an automatic optical inspection system; the optical inspection secondary image classification method (100 ), including the following steps: (a) providing an image feature automatic identification system step, an image feature automatic identification system (10) is integrated with a storage unit (11), a processing unit (12) and an image capture The fetching unit (13), the storage unit (11) is loaded with multiple good images, multiple over-screened images, and multiple defective images; (b) the step of establishing a training data set, at least in the automatic identification of image features A training data set (20) is created in the storage unit (11) of the system (10), and the training data set (20) includes a plurality of images of any good product and any defective product in the storage unit (11) The image pairing becomes the first image group marked 1, and the plural numbers are paired with any good image and any screened image in the storage unit (11) to form the second image group marked 0; (c) The step of establishing a verification data set is to create a verification data set (30) in the storage unit (11) of the image feature automatic identification system (10). The verification data set (30) includes at least a plurality of good images and a plurality of Defective product images, where each good product image in the verification data set (30) is not exactly the same as each good product image in the training data set (20), and each of the good product images in the verification data set (30) Each of the defective product images is not exactly the same as each of the defective product images in the training data set (20) ; (d) the step of establishing a neural network-like architecture is performed in the image feature automatic identification system (10) The storage unit (11) establishes a type of neural network architecture (40) that is executed by the processing unit (12) for predicting and classifying the input image for good or defective products. This type of neural network architecture (40) It has two feature core modules (41) that can be operated independently and a fully connected module (42) connected to the two feature core modules (41); (e) Image classification model In the training step, the second image group with the plural number labeled as label 0 and the second image group with the plural number label as label 1 in the training data set (20) are input into this type of nerve one by one through the image capturing unit (13) The network architecture (40), after comparing the prediction and classification output results of the neural network architecture (40), adjust the characteristics and weights of the neural network architecture (40), and store the adjustments that successfully reduce the error, and then produce a The image classification model that is better at predicting and classifying the input image with good or defective products; (f) The image classification model verification step, each time an image classification model is generated, through the image capture unit ( 13) The multiple good images and multiple defective images in the verification data set (30) are input into the image classification model generated this time one by one, and the image classification model generated this time compares each input good image And predict and classify each defective image, so as to evaluate the prediction of the image classification model generated this time Measure classification accuracy; (g) Statistical classification prediction error distribution and enhanced training steps, each time the image classification model is generated, and when the image classification model generated this time is verified, each input of the good image is recorded And the predicted classification results of each defective image, and the statistical prediction of the classification error distribution, according to the pairing, marking mode and training mode of the first image image and the second image group in the image classification model training step, The quantity of each good image produced by the number of each image that is actually a good image but cannot be accurately predicted and classified, and the number of each image that is actually a defective image but cannot be accurately predicted and classified After the amplification, the defective images are re-paired and labeled, and then input the neural network architecture (40) to which the generated image classification model belongs again to strengthen training, and then another image classification model is generated; (h) The step of confirming the model, after performing steps (f) to (g) at least once, confirming at least one image classification model among the image classification models, and then using the at least one confirmed image The image classification model classifies the products provided by the processing line as good or defective images. The optical detection secondary image classification method according to claim 1, wherein, in the optical detection secondary image classification method (100), in the confirming model step, at least one is repeated After step (f) to step (g), confirm an image classification model with better accuracy among the image classification models, and use the confirmed image classification model to provide the processing line The image classification of good or defective products is performed for the products. The optical detection secondary image classification method according to claim 1, wherein, in the optical detection secondary image classification method (100), in the confirmation model step, the step (f) to the After step (g), among the image classification models, the image classification model with better plural accuracy is confirmed, and the confirmed image classification models are merged, and the merged image classification models are used Perform image classification judgment of good or defective products on the products provided by the processing line. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the number of good images in the training data set (20) and the verification data set (30) is determined by the automatic The optical inspection system judges it to be good and is achieved by augmenting the original image of the good product that is manually confirmed as good; the number of over-screened images in the training data set (20) is determined by the automatic optical inspection system as defective but through Amplification of the original images of good products that are manually confirmed as good products is achieved; the number of defective images in the training data set (20) and the verification data set (30) is determined by the automatic optical inspection system as defective products by a plurality of manuals. The original image of the defective product confirmed as a defective product is amplified. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the number of good images in the training data set (20) and the verification data set (30) is determined by the automatic The optical inspection system judges it to be good and is achieved by augmenting the original image of the good product that is manually confirmed as good; the number of over-screened images in the training data set (20) is determined by the automatic optical inspection system as defective but through The original image of a good product that was manually confirmed as a good product was amplified up to The number of defective images in the training data set (20) and the verification data set (30) is determined by the automatic optical inspection system as defective products and the original images of defective products that are manually confirmed as defective products are expanded by the number of defective images Increase reached. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the predicted output value of each of the neural network architectures (40) of each of the image classification models for each input image , Is presented in the output form of 0 or 1. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the predicted output value of each of the neural network architectures (40) of each of the image classification models for each input image , Is presented in the output form of the probability that the prediction is 0 and the probability that the prediction is 1. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the optical detection secondary image classification method (100) is used in the step of establishing the neural network architecture (40) In the storage unit (11) of the image feature automatic identification system (10), a convolutional neural network architecture (Convolutional neural network) for predicting and classifying the input image for good or defective products is established. ,CNN).

Images