CN112816499B

CN112816499B - Hyperspectral and deep learning combined industrial detection system

Info

Publication number: CN112816499B
Application number: CN202110415789.7A
Authority: CN
Inventors: 方志斌; 和江镇; 王岩松; 都卫东; 吴健雄; 王天翔
Original assignee: Zhengtu Intelligent Technology Jiangsu Co ltd; Focusight Technology Co Ltd
Current assignee: Focusight Technology Co Ltd
Priority date: 2021-04-19
Filing date: 2021-04-19
Publication date: 2021-06-29
Anticipated expiration: 2041-04-19
Also published as: CN112816499A

Abstract

The invention relates to an industrial detection system of hyperspectral combined with deep learning, comprising a spectrometer for taking points and a hyperspectral camera for imaging; the detection steps are: S1, using a spectrometer to take defect points and normal points on a tested sample, And obtain the vector data of defect points and normal points; S2. Use the algorithm based on comparing the mean square error between classes and between classes and the algorithm to calculate the correlation coefficient to verify the feasibility of hyperspectral detection; S3. If it is feasible, use the method based on measurement Select the channel of the hyperspectral image by the method of selecting the eigenvalue gap between the class and the class; S4, use the method based on the eigenvalue of each point and the eigenvalue of the representative vector to compare the selected channel to the image composed of the selected channels. Mark, and send the marked image into the neural network for training; S5, use the trained network to detect the object to be measured. The invention has the characteristics of simplicity, high efficiency and the like.

Description

Hyperspectral and deep learning combined industrial detection system

Technical Field

The invention relates to the technical field of visual detection, in particular to a hyperspectral deep learning combined industrial detection system.

Background

The earliest hyperspectral detection is that after hyperspectral imaging, whether a defect area or a normal area generates an absorption peak in which wave band or not is seen, when only one of the defect area and the normal area generates the absorption peak in a certain wave band, the wave band is only used for imaging, an image with high defect contrast is generated, and detection is performed by using a simple algorithm.

A series of classification algorithms are developed later, and pixel values of a plurality of wave bands of each point in the hyperspectral image can be subjected to dimensionality reduction and then are brought into the classification algorithms for classification.

In recent years, deep learning convolutional neural networks are also used in hyperspectral defect detection and are divided into supervised learning and unsupervised learning; a large number of labeled hyperspectral image samples are needed for supervised learning, and labeling of the hyperspectral image samples is not required for unsupervised learning.

However, the hyperspectral techniques described above each have the following disadvantages:

1. the traditional hyperspectral imaging detection excessively depends on absorption peaks, so that spectral information is not fully utilized, and the condition that a complex algorithm is feasible is probably considered to be infeasible;

2. later appearing classifier algorithms are traversed point by point on application to execute the classifier algorithms with high complexity of time, and the efficiency is too low;

3. in the hyperspectrum combination machine learning and deep learning methods appearing in recent years, a large amount of labeled data are required to be used as input for training, and for defects with great difficulty in manual detection, the defects are very difficult to label on a defect sample; due to the fact that the number of channels of the hyperspectral image is too many, the defect part is difficult to effectively visualize, and the image is difficult to label, a large amount of labeled data is difficult to obtain;

however, the unsupervised deep learning method has no defect information label at all, and the spectrum is affected by non-defect factors (such as impurities with untuned defects in the object to be detected), so that the probability of occurring wrong classification is too high.

Disclosure of Invention

The technical problem to be solved by the invention is as follows: because the number of channels of the hyperspectral image is too many, an ideal visualization method is not available, and whether the defects on the current object to be detected can be detected by hyperspectrum or not can be seen in a short time; the time, cost and data complexity of the hyperspectral imaging test are very high expenses under the condition of uncertainty of feasibility; if not feasible, resources are wasted; under the condition that the hyperspectral imaging is feasible, if the defect and the normal area have no absorption peak with high contrast, the classification algorithm is very complex, and the parallelism cannot be realized on the program, so that the program execution efficiency is very low; if other convolutional neural networks with high parallelism and strong distinguishing capability are tried to realize rapid detection, in the training stage, due to the fact that the labeled sample amount is small and the hyperspectral image almost has no method for obtaining the global visualization effect, enough labeled data are difficult to obtain for effective training; therefore, the invention provides an industrial detection system combining hyperspectrum and deep learning to solve the problems.

The technical scheme adopted by the invention for solving the technical problems is as follows: a hyperspectral depth learning combined industrial detection system comprises a spectrometer for taking points and a hyperspectral camera for imaging; the detection steps are as follows:

s1, using a spectrometer to take the defect point and the normal point on the sample to be detected, and obtaining the vector data of the defect point and the normal point;

s2, verifying the feasibility of hyperspectral detection by using an algorithm based on the mean square error between the comparison class and an algorithm for calculating a correlation coefficient;

s3, if the method is feasible, selecting a channel of the hyperspectral image by using a method based on measurement of the difference of the selected characteristic values between the intra-class and the inter-class;

s4, automatically marking the selected channel by using a method of comparing the characteristic value of each point with the characteristic value of the representative vector, and sending the marked image to a neural network for training;

and S5, detecting the detected object by using the trained network.

Further, in step S2, the method for verifying the feasibility of hyperspectral detection includes:

pairwise defective point vectors and normal point vectors are calculated according to the formula MSE =

Calculating the Mean Square Error (MSE) between the defect point vector and the normal point vector in the class and between the classes; wherein, V_1iThe ith number, V, representing the first vector_2iAn ith number representing a second vector; m is the wavelength resolution of the spectrometer and is equal to the number of channels of the hyperspectral image;

according to the formula:

calculating a correlation coefficient R between the defect point vector and the normal point vector in the class and between the classes; wherein

，

；

For each MSE there is: within the MSE class < between MSE classes, or, for each R: r within class > R between classes; if the defect has separability under the hyperspectral condition, carrying out the following detection steps; otherwise, the hyperspectral detection is not feasible without separability.

Still further, in step S3 of the present invention, the method for selecting the channels of the hyperspectral image is as follows:

1) respectively solving an intra-class mean square error mean value MSE _ mean class, an inter-class mean square error mean value MSE _ mean class, an intra-class correlation coefficient mean value R _ mean class and an inter-class correlation coefficient mean value R _ mean class; and calculating relative proportion MSE _ mean inter-class/MSE _ mean intra-class and R _ mean intra-class/value R _ mean inter-class; if the MSE _ mean inter-class/MSE _ mean intra-class is larger, taking the negative number-1 MSE of the mean square error as a characteristic value, and otherwise, taking a correlation coefficient R as the characteristic value;

2) taking an average value of each dimension of the defect point vector and the normal point vector to form a defect representative vector and a normal representative vector, and continuously and averagely segmenting m-dimensional data of the two representative vectors respectively, wherein each segment is m/c dimension, m is the number of channels of the hyperspectral image, and c is the number of segments;

3) sorting (m/c) -1 eigenvalues determined in the step 1) of the c-section data from large to small, and marking as f₁, f₂...f_（m/c）-1A 1 is to f₁, f₂...f_（m/c）-2Making a difference with the next adjacent element, denoted as d₁, d₂...d_（m/c）-2(ii) a Let the maximum value in the set of difference values be d_jThen f is selected₁, f₂...f_jThe corresponding segments constitute a new representative vector, f₁, f₂...f_jThe corresponding channel is the selected channel.

In step S4, the method of the present invention further includes traversing each point by using the path determined in step S3, and calculating the eigenvalue f between each point and the defect representative vector and the normal representative vector_dAnd f_n；

If, f_d>f_nAnd f is_d>= set value a f_iThen the point is marked as defect class;

if, f_d<f_nAnd f is_n>Set value B f_bIf yes, the point is marked as a normal class;

if the two are not satisfied, marking the point as a false defect point;

wherein f is_bIs a representative characteristic value between classes, f_iThe characteristic value is represented in the class.

Still further, the present invention includes performing manual review of the points marked as false defect points, the review including:

i. statistical defect points due to non-compliance with f_d >= set value a f_iIs marked as falseReducing the value of the set value A for the points of the defect points so that the defect points in all the pseudo defects can be divided into the defect points under the new rule;

ii. Normal point of statistics due to non-compliance with f_n>Set value B f_bThe point marked as a false defect point is decreased in the value of the set value B so that the normal point can be classified as a normal point under the new rule in just all the false defects.

The invention has the advantages of overcoming the defects in the background technology,

firstly, a high-efficiency and low-cost spectrometer is used for taking points to detect defect areas and normal areas on a small amount of marked samples, and the separability of the marked samples is quickly and effectively analyzed;

after the fact that the mark can be distinguished is proved, after the mark which is easy to recognize by hyperspectral imaging is made around the defect, hyperspectral imaging is carried out, a proper waveband channel is selected by utilizing the difference between the mean square error and the correlation coefficient of the pixels of the known defect region and the normal region, dimension reduction is realized, and classification is carried out on the training image after dimension reduction by using a method based on the mean square error and the correlation coefficient, so that accurate automatic marking is realized, and the problems that the marking of supervised learning data marking and unsupervised learning marking is easy to make mistakes are solved;

after the hyperspectral data of the labeled training set are obtained, the hyperspectral data are sent to a convolutional neural network for training, a finally obtained training model is used for hyperspectral image detection of a measured object, and the problem of low execution efficiency of a complex algorithm program using a classifier is solved by utilizing the characteristics of strong fitting performance of the convolutional neural network and high execution efficiency of a convolution calculation program based on multiplication.

Drawings

FIG. 1 is a schematic diagram of the hardware components and the general flow of the system of the present invention.

FIG. 2 is a block diagram of the detection process of the present invention.

Detailed Description

The invention will now be described in further detail with reference to the drawings and preferred embodiments. These drawings are simplified schematic views illustrating only the basic structure of the present invention in a schematic manner, and thus show only the constitution related to the present invention.

As shown in fig. 1-2, the hyperspectral depth learning combined industrial detection system comprises an optical hardware component and a hyperspectral camera, wherein the optical hardware component comprises a spectrometer used for taking points and the hyperspectral camera used for imaging;

the detection steps are as follows:

s1, using a spectrometer to obtain vector data of the defect point and the normal point on the industrial measured sample;

s3, if the method is feasible, selecting a channel of the hyperspectral image by using a method based on measurement of difference of selected characteristic values (mean square error or correlation coefficient) between the intra-class and the inter-class;

s4, automatically marking the selected channel by using a method of comparing a characteristic value (mean square error or correlation coefficient) based on each point with a representative vector characteristic value, and sending the marked image to a neural network for training;

and S5, detecting the detected object by using the trained network.

The following is illustrated by way of specific embodiments:

1. point taking by a spectrometer: as shown in the first step in fig. 1, on a defect sample marked with a mark recognizable to the human eye, a spectrometer is used to take 5 points at the position marked with the defect mark to obtain spectral data of the defect sample, and 50 points are randomly taken in an unmarked area to also obtain spectral data of the defect sample, the wavelength resolution of the spectrometer is set to be m, that is, if the obtained spectral data is regarded as an array, the value of m is the length of the array, and the value of m is equal to the number of channels of a subsequent hyperspectral image, so that a total of 55 m-dimensional vectors is obtained.

2. And (4) feasibility judgment: performing Gaussian fitting calculation on each dimension of data in vectors at 50 points in the unmarked area, calculating each one-dimensional fitting expectation to form a fitting expectation vector, and taking 5 vectors closest to the fitting expectation vector from the vectors at the 50 unmarked points, and marking as normal pointsVector, and the pairwise distance between 5 defective point vectors and 5 normal point vectors according to the formula MSE =

In which V is_1iAnd V_2iWherein 1 and 2 represent vector 1 and vector 2 respectively, 2 different vectors are used for distinguishing and selecting, i represents when regarding the vector as an array, the ith value in the array calculates the mean square error MSE between the defect and normal vector in and between classes, total 45 MSEs, wherein: between 25 MSE classes and within 20 MSE classes, within normal classes and within defect classes, 10 each;

according to the formula:

wherein

，

(ii) a Calculating the correlation coefficient R between the intra-class and the inter-class, wherein the total number of R is 45, wherein: between 25R classes and within 20R classes, within the normal class and within the defect class, 10 each.

If one of the following conditions is met:

(1) for each MSE there is: within the MSE class < between MSE classes,

(2) for each R there is: r within class > R between classes;

the defect has separability under the hyperspectral condition, the following detection steps can be carried out, otherwise, the defect does not have separability, and the hyperspectral detection is not feasible.

Here, the inter-class means that vectors of the normal class and the defect class are pairwise compared; that is, the normal class is compared with the defect class; the intra-class means that the vectors of the normal class are pairwise compared, and the vectors of the defect class are pairwise compared, namely the normal class is compared with the normal class; the defect class is compared to the defect class.

3. Selecting characteristics: and (3) calculating the intra-class mean square error mean value MSE _ mean class, the inter-class mean square error mean value MSE _ mean class, the intra-class correlation number mean value R _ mean class and the inter-class correlation coefficient mean value R _ mean class in the step (2), calculating the relative proportion MSE _ mean class inter-MSE _ mean class/MSE _ mean class and R _ mean class inter-class/value R _ mean class, if the intra-MSE _ mean class/MSE _ mean class is larger, taking the negative number-1 MSE of the mean square error as a characteristic value, and otherwise, taking the correlation coefficient R as the characteristic value.

4. And (3) reducing the dimensionality: making a pure red mark which can be easily identified by a hyperspectral camera around a defect of a measured object, performing hyperspectral imaging on the measured object, taking 5 defect vectors and 5 normal vectors on a hyperspectral image in the same way as in the steps 1 and 2, setting the number of channels of the obtained hyperspectral image to be m, namely setting the vector of each point to be m dimension, taking the mean value of each dimension of the defect vectors and the normal vectors to form a defect representative vector and a normal representative vector, respectively continuously and averagely dividing m-dimension data of the two representative vectors into c sections, wherein each section is m/c dimension, c is generally 20, sequencing (m/c) -1 characteristic value (characteristic value determined in the step 3) of the c sections of data from large to small, and marking the characteristic value as f₁，f₂...f_（m/c）-1A 1 is to f₁，f₂...f_（m/c）-2Making a difference with the next adjacent element, denoted as d₁，d₂...d_（m/c）-2Let the maximum value in the set of difference values be d_jThen f is selected₁, f₂...f_jThe corresponding segments form new representative vectors, then only the channel forming vectors corresponding to the characteristic values are selected to be compared with the representative vectors for classification, and f₁, f₂...f_jThe corresponding channel is used as the channel selected in the subsequent steps.

5. Labeling: combining the 5 defect vectors and the 5 segments in the normal vectors determined in the step 4 into a new characteristic vector, solving 25 inter-class characteristic values and 20 intra-class characteristic values among the vectors, respectively carrying out Gaussian fitting to obtain an expected value serving as a representative characteristic value f among the classes_bAnd the representative characteristic value f in the class_i(ii) a Selecting the channel of the hyperspectral image determined in the step 4, traversing each point, and calculating the characteristic value f between the hyperspectral image and the defect representative vector as well as the characteristic value f between the hyperspectral image and the normal representative vector_dAnd f_n；

If: f. of_d>f_nAnd f is_d>=0.8 (setting a is 0.8) × f_iThen the point is marked as defect class;

if: f. of_d<f_nAnd f is_n>1.2 (setting B of 1.2) × f_bIf yes, the point is marked as a normal class;

if: if the two are not satisfied, the point is marked as a false defect point;

then, randomly extracting 50 pseudo-defect points for manual rechecking, wherein the manual rechecking mode is as follows:

I. statistical defect points due to non-compliance with f_d >=0.8 *f_iReducing the point marked as the false defect point by a value of 0.8 to ensure that the defect point in all the false defects can be divided into defect points under a new rule;

i I, the statistical normality point is not satisfied with f_n>1.2 *f_bReducing the point marked as the false defect point by a value of 1.2 to ensure that the normal point in all the false defects can be divided into normal points under a new rule; training and testing neural networks.

6. And (4) introducing the training set data labeled in the step (5) into a proper convolutional neural network for training, testing by using a manually labeled test set, adjusting a network structure and training hyper-parameters according to a test result until the accuracy of the test meets the requirement (determined according to the specific missed detection and killing requirement), and taking the channel in the hyperspectral imaging determined in the step (4) as input to realize hyperspectral imaging defect detection by using the trained neural network.

The invention uses a small amount of collected samples marked (namely knowing where the defect is on the measured object) to carry out point acquisition detection by a spectrometer, obtains the spectral data of a plurality of defects in normal position areas, and then judges the separability of the spectral data by using the difference between the defect positions and the normal position spectral data and the mean square difference between the defect positions and the normal position and the correlation coefficient;

if the hyperspectral image can be distinguished, selecting a proper waveband by using a mean square error threshold value and a correlation coefficient segmentation and combination method to perform hyperspectral imaging; and finally, setting a threshold value according to the mean square error and the correlation coefficient of spectral data of each point and a defect point on imaging to judge the classification of the points, marking the batch of unlabeled hyperspectral images, finally sending the labeled hyperspectral images into a neural network for training, and detecting the hyperspectral images through the trained acquired neural network at a speed much higher than that of analyzing the mean square error and the correlation coefficient point by point.

While particular embodiments of the present invention have been described in the foregoing specification, various modifications and alterations to the previously described embodiments will become apparent to those skilled in the art from this description without departing from the spirit and scope of the invention.

Claims

1. a hyperspectral combined with a deep learning industrial detection system, characterized in that: comprising a spectrometer for taking a point and a hyperspectral camera for imaging; the detection step is:

S1. Use a spectrometer to take defect points and normal points on the tested sample, and obtain the vector data of defect points and normal points;

S2. Verify the feasibility of hyperspectral detection using an algorithm based on comparing intra-class and inter-class mean square errors and an algorithm for calculating correlation coefficients;

S3. If feasible, use a method based on measuring the difference between selected eigenvalues within a class and between classes to select a hyperspectral image channel;

The channel selection method of hyperspectral image is as follows:

1) Respectively obtain the intra-class mean square error MSE_mean intra-class, inter-class mean square error MSE_mean inter-class, intra-class correlation coefficient mean R_mean intra-class, inter-class correlation coefficient mean R_mean inter-class; and find the relative ratio MSE_mean inter-class/MSE_mean Intra-class, and R_mean intra-class/value R_mean inter-class; if MSE_mean inter-class/MSE_mean intra-class is larger, take the negative of the mean square error -1*MSE as the eigenvalue, otherwise take the correlation coefficient R as the eigenvalue;

2) Take the mean value of each dimension of the defect point vector and the normal point vector to form a defect representative vector and a normal representative vector, respectively divide the m-dimensional data of the two representative vectors into continuous and average segments, each segment is m /c dimension, where m is the number of channels of the hyperspectral image, and c is the number of segments;

3) Sort the (m/c)-1 eigenvalues determined in step 1) of the c segment data from large to small, denoted as f ₁ , f ₂ ... f _(m/c)-1 , Perform a difference between f ₁ , f ₂ ... f _(m/c)-2 and the next adjacent element, denoted as d ₁ , d ₂ ... d _(m/c)-2 ; let this group The maximum value in the difference is d _j , then the segment corresponding to f ₁ , f ₂ ... f _j is selected to form a new representative vector, and the channel corresponding to f ₁ , f ₂ ... f _j is the selected channel ;

S4, using the method based on the comparison of the eigenvalue of each point and the eigenvalue of the representative vector on the selected channel to automatically mark the image composed of the selected channel, and send the marked image into the neural network for training;

S5. Use the trained network to detect the object to be measured.

2. A kind of hyperspectral combined with deep learning industrial detection system as claimed in claim 1, it is characterized in that: in described step S2, the method for verifying the feasibility of hyperspectral detection is:

Between the defect point vector and the normal point vector according to the formula MSE=

, Calculate the mean square error MSE between the defect point vector and the normal point vector within and between classes; where V _1i represents the ith number of the first vector, and V _2i represents the ith number of the second vector; m is the spectrometer The wavelength resolution is equal to the number of channels of the hyperspectral image;

According to the formula:

, calculate the correlation coefficient R between the defect point vector and the normal point vector within and between classes; where,

,

;

For each MSE: within MSE class < between MSE classes, or, for each R: within R class > between R classes; it means that the defect is separable under hyperspectral, and the following detection steps are performed; otherwise , without separability, hyperspectral detection is not feasible.

3. The industrial detection system of hyperspectral combined with deep learning as claimed in claim 2, characterized in that: in the step S4, using the channel determined in the step S3, traverse each point, and calculate each point and the defect representative vector and the eigenvalues f _d and f _n between the normal representative vectors;

If f _d >f _n and f _d >= set value A*f _i , the point is marked as a defect class;

If f _d < f _n and f _n > set value B*f _b , the point is marked as normal;

If the above two are not satisfied, the point is marked as a pseudo-defect point;

Among them, f _b is the representative eigenvalue between classes, and f _i is the representative eigenvalue within the class.

4. A kind of hyperspectral combined with deep learning industrial detection system as claimed in claim 3, it is characterized in that: comprising manually checking the points marked as pseudo-defect points, and the checking comprises:

i. Statistical defect points are marked as pseudo defect points because they do not meet f _d >= set value A * f _i , reduce the value of set value A, so that the defect points in just all pseudo defects are in the new rule The following can be divided into defect points;

ii. Statistically normal points are marked as pseudo-defect points because they do not meet f _n > set value B *f _b , and the value of set value B is reduced, so that just all the normal points in pseudo-defects are under the new rule can be classified as normal.