TWI819698B - Method for determining defect and electronic apparatus - Google Patents

Abstract

A method for determining defect and an electronic apparatus are provided. First, an interference image corresponding to an object to be tested is input into multiple trained neural network models to obtain multiple sets of Zernike coefficients. Next, a fused coefficient set is obtained based on the sets of Zernike coefficients. Afterwards, a point spread function and modulation transfer function are calculated using the fused coefficient set. Then, based on the fused coefficient set, point spread function and modulation transfer function, whether the object to be tested is defective is determined.

Description

Methods and electronic devices for determining defects

The present invention relates to an image recognition technology, and in particular, to a method and electronic device for determining defects based on image recognition.

Interferometers are widely used in scientific research and industrial production to measure small displacements, refractive index and surface flatness. Currently, most optical inspections of lenses use an interferometer to generate an interference image, and then the human eye observes the pattern of the interference image to determine whether the lens is defective. Direct judgment by the human eye may not be accurate enough due to different shooting conditions of each lens, time pressure, eye fatigue, or inconsistencies in human judgment standards. Each interferometer requires an inspector, which involves labor costs. In addition, the shape of the interference image may be composed of multiple optical factors, which is difficult to automatically determine through a single threshold or judgment rule. Furthermore, if the judgment criteria differ depending on the customer's standards, it will be difficult to accurately and dynamically adjust the threshold value or judgment rules.

The present invention provides a method and electronic device for determining defects, and provides a scientific method to achieve the purpose of accurate automatic detection.

The method of determining defects based on image recognition of the present invention uses a processor to perform multiple steps, including: inputting interference images corresponding to the object to be tested into multiple trained neural network models to obtain multiple sets of ZERO Zernike coefficient; obtain a fused coefficient set based on the multiple sets of Zernike coefficients; use the fused coefficient set to calculate a Point Spread Function (PSF) and a Modulation Transfer Function (MTF) ; and based on the fused coefficient group, point spread function and modulation transfer function, determine whether the object to be tested is defective.

In an embodiment of the present invention, the neural network model includes a model using a convolutional neural network (CNN), an autoencoder (AutoEncoder), and a generative adversarial network (GAN). .

In an embodiment of the present invention, the step of obtaining the fused coefficient set based on the multiple sets of Zernike coefficients includes: using one of the average method, the median, the maximum value, the minimum value and the weighted average to calculate the Obtain the fused coefficient group from multiple groups of Zernike coefficients.

In an embodiment of the present invention, the step of obtaining the fused coefficient set based on the multiple sets of Zernike coefficients includes: using a voting mechanism to obtain the fused coefficient set from the multiple sets of Zernike coefficients.

In an embodiment of the present invention, based on the fused coefficient group, point spread function and modulation transfer function, the step of determining whether the object to be tested is defective includes: combining at least one of the fused coefficient group, point spread function and modulation transfer function. It is compared with one or more corresponding thresholds to determine the degree of defects or the grade of the object to be tested.

In an embodiment of the present invention, based on the fused coefficient group, point spread function and modulation transfer function, the step of determining whether the object to be tested is defective includes: combining at least one of the fused coefficient group, point spread function and modulation transfer function. or input into its corresponding inspection model to determine the degree of defects or the grade of the object to be tested.

The electronic device of the present invention includes: a storage device including a plurality of trained neural network models; a processor coupled to the storage device and configured to: input interference images corresponding to the object to be measured to the neural network path model to obtain multiple sets of Zernike coefficients; obtain a fused coefficient set based on the multiple sets of Zernike coefficients; use the fused coefficient set to calculate the point spread function and modulation transfer function; and based on the fused coefficient set, point Diffusion function and modulation transfer function are used to determine whether the object to be tested is defective.

Based on the above, this disclosure obtains a scientific method from an optical perspective, that is, deducing the Zernike coefficient from the interference image, and then deriving the point spread function and modulation transfer function from the Zernike coefficient to judge the quality of the object to be tested. In this way, accurate automatic detection can be achieved.

FIG. 1 is a block diagram of an electronic device according to an embodiment of the present invention. Referring to FIG. 1 , the electronic device 100 includes a processor 110 and a storage device 120 . Processor 110 is coupled to storage device 120 . The processor 110 is, for example, a central processing unit (CPU), a physical processing unit (PPU), a programmable microprocessor (Microprocessor), an embedded control chip, or a digital signal processor (Digital Signal). Processor (DSP), Application Specific Integrated Circuits (ASIC), programmable logic controller (PLC) or other similar devices.

The storage device 120 is, for example, any type of fixed or removable random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), flash memory (Flash memory), hardware disc or other similar device or a combination of these devices. The storage device 120 includes one or more program code fragments. After the program code fragments are installed, the processor 110 will execute each step of the method for determining defects described below.

FIG. 2 is a flow chart of a method for determining defects based on image recognition according to an embodiment of the present invention. Please refer to Figure 2. In step S205, interference images corresponding to the object to be measured are input to multiple trained neural network models to obtain multiple sets of Zernike coefficients (Zernike coefficients, hereinafter referred to as Zernike coefficients). The object to be tested is, for example, a lens to be tested. The lens to be detected can be placed on the platform of the interferometer, then illuminated, and then photographed by the camera on the interferometer to obtain the interference image.

Here, predicting the Zernike coefficient through the image recognition function of the neural network model can reduce the mathematical operations commonly used in interferometers, thereby improving measurement accuracy. In one embodiment, the neural network model includes a model using a convolutional neural network (CNN), an autoencoder (AutoEncoder), and a generative adversarial network (GAN).

For example, CNN is currently the most commonly used neural network, which consists of several layers. These layers include convolution layer, pooling layer and fully connected layer. The number of convolutional layers, pooling layers and fully connected layers can be configured as one or more, with different configurations depending on the architecture. Features are extracted from the input image (interference image) in the convolutional layer. In the pooling layer the data size of the extracted features will be reduced but the main functionality will be retained. The fully connected layer increases the depth of the network and predicts the answer (Zernike coefficient). For example, the last fully connected layer can be set to 8 neurons to correspond to 8 Zernike coefficients. However, this is only an example and is not limited to this. The number of Zernike coefficients finally output can be adjusted as required.

The CNN model is trained in advance, and then the interference image obtained by the interferometer is input to the trained CNN model, so that the CNN model outputs the corresponding Zernike coefficient. The CNN model is trained by training a large number of interference images and their corresponding known Zernike coefficients, so that the CNN model can automatically extract features from the input interference images and adjust the parameter values in the CNN model to obtain the best output (i.e., a set of Zernike coefficients).

Autoencoder is an unsupervised learning algorithm of multi-layer neural network. A raw data is input to the autoencoder, and the autoencoder can output reconstructed data that is the same as the original data. The architecture of the autoencoder includes an encoder (Encoder) and a decoder (Decoder), which perform compression and decompression operations respectively, so that the output data and the input data represent the same meaning. The encoder includes one or more hidden layers, whose function is to encode the input high-dimensional vector into a low-dimensional vector. The function of the decoder is to restore the low-dimensional vector of the hidden layer to the initial high-dimensional vector. After training, the autoencoder will finally obtain a low-dimensional vector representing the input data in the hidden layer. After the training is completed, the decoder is removed, leaving only the encoder. By inputting the interference image to the trained autoencoder, the corresponding set of Zernike coefficients can be obtained. By training an encoder and a decoder network at the same time, the decoder can help the encoder learn in the process of learning to restore the original data, and finally an encoder with better prediction ability can be obtained as the prediction coefficient. Neural network model.

GAN includes Generator Network and Discriminator Network. The generation network receives an interference image and outputs a mosaic image labeled as a fake image. Each piece of the mosaic image is associated with a Zernike coefficient. For example, a mosaic image can be designed to include 32 blocks of the same size, each block corresponding to a Zernike coefficient. The discriminating network is used to receive another mosaic image marked as ground truth, and to receive the mosaic image output by the generating network marked as a fake image, and the discriminating network competes with each other through the two. to produce results. The discriminative network learns to classify the output of the generative network as real or fake.

In one embodiment, three sets of Zernike coefficients can be obtained using three trained neural network models using CNN, autoencoder and GNN. However, this is only an example and does not limit the number of neural network models and the algorithms used.

Next, in step S210, a fused coefficient set is obtained based on the multiple sets of Zernike coefficients. For example, the average method, median, maximum value, minimum value, weighted average, etc. can be used to obtain the fused coefficient group from the multiple groups of Zernike coefficients. For example, using the averaging method, the corresponding Zernike coefficients obtained from all neural network models are accumulated and averaged, and the average value is used as the fusion coefficient group. Assuming that three neural network models are used, the number of each set of Zernike coefficients obtained is 8 (N1~N8). Therefore, the N1 and N2 obtained by the three neural network models are averaged. ..., N8 is averaged, and the Zernike coefficient after 8 averages is obtained. Or, in terms of the median, take the median of N1, the median of N2,..., the median of N8 obtained by the three neural network models, and obtain 8 Zernike coefficients as the fusion coefficients group. The same goes for maximum value, minimum value, and weighted average.

In addition, regression models can be established separately for each Zernike coefficient. For example, assume that the first coefficients obtained by the three neural network models are [0.2, 0.3, 0.5] respectively, and use them as input features of the regression model to train the regression model and use the output of the regression model As the first fusion coefficient of the fusion coefficient group. Regression models can use methods such as Deep Neural Network (DNN), Support Vector Regression (SVR), and Gaussian process.

In addition, a voting mechanism can also be used to obtain a set of fused coefficients among the multiple sets of Zernike coefficients. The Zernike coefficients obtained from different neural network models are weighted and synthesized. For example, assuming that M trained neural network models are used, the number of each set of Zernike coefficients obtained is 8, and 8×M Zernike coefficients (N _1,1 ~ N _1,M , N _2,1 ～N _2,M ,...,N _8,1 ～N _8,M ). The i-th fused coefficient N _i in the fused coefficient group is: N _i = W _i,1 ×N _i,1 + W _i,2 ×N i _,2 +...+ W _i,M ×N _{i, M} )/M, where W _i,j represents the weight corresponding to the i-th Zernike coefficient predicted by the j-th neural network model. N _i,1 ~N _i,M respectively represent the i-th Zernike coefficient predicted by the 1st-Mth neural network model.

In step S215, a point spread function (Point Spread Function, PSF) and a modulation transfer function (Modulation Transfer Function, MTF) are calculated using the fused coefficient group.

In the field of optics, due to the orthogonality and balanced expression of classical aberrations of Zernike polynomials, it has the smallest variance on the circular pupil, and has been widely used in the field of optics. In an optical system, the plane where the chief ray and the optical axis are located is the meridional plane, and the plane where the chief ray is perpendicular to the meridional plane is the sagittal plane. The exit pupil coordinates ( x _p , y _p ) and polar coordinates ( ρ , θ ) of any point P in the unit circle plane. Where x _p is located on the longitudinal plane, y _p is located on the meridian plane, ρ is the radial component, and θ is the angular coordinate.

wavefront function The description is as follows: Among them, n and m represent the power of the radial coordinate (radial coordinate) and image point height coordinate (image point height coordinate) in the standard aberration function respectively, and s represents the summed annotation, such as summing n items. Or nm items. is the average wavefront, , , , , , is the Zernike coefficient.

Next, based on the wavefront function To obtain the aberrated pupil function , and then based on the aberration pupil function to obtain the coherent transfer function .

Aberration pupil function for: , in, is the exit pupil radius, k is the wave number, and i is an imaginary number.

homology transfer function for: , where f _U is U/ λ z , f _V is V/ λ z , λ is the wavelength, and z is the pupil distance.

After that, based on the homology conversion function To calculate the point spread function h(u, v) and the modulation transfer function MTF(f _U , f _V ) .

The point spread function h(u, v) describes the response of an imaging system to a point light source (object) and can be obtained through the following equation: , in, is the Fourier transform function, u and v are coordinates.

The modulation transfer function MTF (f _U , f _V ) is: .

In step S220, it is determined whether the object to be tested is defective based on the fused coefficient group, point spread function and modulation transfer function. For example, at least one of the fused coefficient set, point spread function, and modulation transfer function can be compared with one or more corresponding thresholds to determine the degree of defects or the grade of the object to be tested.

Set one or more thresholds for the modulation transfer function (MTF) and substitute them into the modulation transfer function to determine the lens grade or defect level. The determination method using MTF is as follows. In terms of the modulation transfer function, one or more thresholds are set according to product quality requirements, and then the thresholds are used to cut out intervals of each quality level (level) in the MTF value space. After the product to be inspected obtains the MTF value, it is determined according to its interval. Determine the quality level. In the model training stage: products of each quality level can be sampled to obtain samples of products of each quality level. For each sample, the MTF value is obtained through the above method, and then the MTF value is set to feature X and the quality level is set to Y, and then training or Export a regression model. In the practical stage: use the above method to predict the MTF value of the interference image of the object to be measured, and then set the MTF value to the X of the regression model, and the quality level Y can be obtained by the autoregressive model. Here, multiple quality levels are set as continuous values, which can be regarded as scores of quality. For example, the quality levels can be set from 0 to 100.

In addition, one or more thresholds are set for the point spread function (PSF) and substituted into the point spread function to determine the lens grade or the degree of defects. Alternatively, one or more thresholds can be set for the fused coefficient group to determine the Lens level or defect level. The determination method using PSF or the fused coefficient group can refer to the above-mentioned determination method using MTF, which will not be described again here.

In other embodiments, the basis for judging quality can also be comprehensively considered by setting the weights of the three methods, for example, setting it as: Q( I) = w ₁ × Statistics ( MTF ( I ))+ w ₂ × Statistics ( PSF ( I ))+ w ₃ × Statistics ( N _i ); Among them, Statistics is a statistical function, which can be a statistical method such as maximum value, minimum value, average value, mode, variation, standard deviation or summation, w ₁ , w ₂ and w ₃ are weights, I is the examined interference image, and _Ni is the Zernike coefficient obtained from the interference image I.

Alternatively, corresponding inspection models can be pre-established and trained for the modulation transfer function, point spread function and fused coefficient group respectively. Input at least one of the fused coefficient group, point spread function and modulation transfer function into its corresponding inspection model to determine the degree of defects or the grade of the object to be tested. The inspection model may use a classification model trained by Support Vector Machine (SVM), DNN, eXtreme Gradient Boosting (XGBoost), etc., or a regression model trained by SVR, Gaussian process, etc.

For the inspection model corresponding to MTF, in the training stage of the inspection model: Sampling products of each quality level to obtain samples of products of each quality level, obtaining the MTF value for each sample through the above method, and then setting the MTF value as a feature X, the quality level is set to Y, and then an inspection model (classification model) is trained or exported. In the practical stage of the inspection model: use the above method to predict the MTF value of the interference image of the object to be measured, and then set the MTF value to the X of the inspection model and substitute it into the inspection model to obtain the quality level Y. Here, quality levels include " excellent ", "good", "average", "minor defects", and "major defects". The inspection model corresponding to the PSF or the fused coefficient group can also be derived by referring to the above description.

FIG. 3 is an architecture diagram for determining defects according to an embodiment of the present invention. As shown in Figure 3, the interference image is input to the neural network models 1 to N to obtain N sets of Zernike coefficients. Afterwards, these Zernike coefficients are fused to obtain the final fusion coefficient group. After the point spread function and modulation transfer function are calculated using the fusion coefficient group, the fusion coefficient group, point spread function and modulation transfer function can be used to determine the lens ( The object under test) is defective.

To sum up, this disclosure obtains a scientific method from an optical perspective, that is, deducing the Zernike coefficient from the interference image, and then deriving the point spread function and modulation transfer function from the Zernike coefficient to judge the quality of the object to be tested.

100: Electronic devices 110: Processor 120:Storage device S205～S220: Each step of the method to determine defects based on image recognition

FIG. 1 is a block diagram of an electronic device according to an embodiment of the present invention. FIG. 2 is a flow chart of a method for determining defects based on image recognition according to an embodiment of the present invention. FIG. 3 is an architecture diagram for determining defects according to an embodiment of the present invention.

S205~S220: Each step of the method to determine defects based on image recognition

Claims (10)

A method of determining defects based on image recognition, which uses a processor to perform multiple steps, including: Input an interference image corresponding to an object to be measured to multiple trained neural network models to obtain multiple sets of Zernike coefficients; Obtain a fused coefficient set based on the multiple sets of Zernike coefficients; Using the fused coefficient set, calculate a point spread function and a modulation transfer function; and Based on the fused coefficient group, the point spread function and the modulation transfer function, it is determined whether the object to be tested is defective. As claimed in claim 1, the method for determining defects based on image recognition, wherein the neural network models include a model using a convolutional neural network, an autoencoder and a generative adversarial network. The method for determining defects based on image recognition as described in claim 1, wherein the step of obtaining the fused coefficient set based on the multiple sets of Zernike coefficients includes: One of an average method, a median, a maximum value, a minimum value, and a weighted average is used to obtain the fused coefficient set among the multiple sets of Zernike coefficients. The method for determining defects based on image recognition as described in claim 1, wherein the step of obtaining the fused coefficient set based on the multiple sets of Zernike coefficients includes: A voting mechanism is adopted to obtain the fused coefficient set among the multiple sets of Zernike coefficients. The method for determining defects based on image recognition as described in claim 1, wherein based on the fused coefficient group, the point spread function and the modulation transfer function, the step of determining whether the object to be tested is defective includes: At least one of the fused coefficient set, the point spread function and the modulation transfer function is compared with one or more corresponding thresholds to determine a defect degree or a test object level. The method of determining defects based on image recognition as described in claim 1, wherein the step of determining whether the interference image is defective based on the fused coefficient group, the point spread function and the modulation transfer function includes: At least one of the fused coefficient set, the point spread function and the modulation transfer function is input to a corresponding inspection model to determine a defect degree or a test object grade. An electronic device including: a storage device including multiple trained neural network models; A processor coupled to the storage device and configured to: Input an interference image corresponding to an object to be measured into the neural network models to obtain multiple sets of Zernike coefficients; Obtain a fused coefficient set based on the multiple sets of Zernike coefficients; Using the fused coefficient set, calculate a point spread function and a modulation transfer function; and Based on the fused coefficient group, the point spread function and the modulation transfer function, it is determined whether the object to be tested is defective. The electronic device of claim 7, wherein the neural network models include models using a convolutional neural network, an autoencoder and a generative adversarial network. The electronic device of claim 7, wherein the processor is configured to: One of an average method, a median, a maximum value, a minimum value, a weighted average and a voting mechanism is used to obtain the fused coefficient set among the multiple sets of Zernike coefficients. The electronic device of claim 7, wherein the processor is configured to: Compare at least one of the fused coefficient set, the point spread function and the modulation transfer function with its corresponding one or more thresholds to determine a defect degree or a test object level; or At least one of the fused coefficient set, the point spread function and the modulation transfer function is input to a corresponding inspection model to determine a defect degree or a test object grade.