CN117875408A - Federal learning method of pulse neural network for flaw detection - Google Patents

Abstract

The invention provides a federal learning method of a pulse neural network facing flaw detection, which comprises the steps of inputting a product image into an artificial neural network branch for forward propagation, inputting a receptive field matrix code into the pulse neural network branch for forward propagation in a time window, and fusing the characteristics output by the artificial neural network branch with the characteristics output by each time step in the pulse neural network branch in the forward propagation process of the pulse neural network branch, so that the influence of peak noise is effectively relieved; and constructing a loss function according to the characteristics and flaw type labels output by all time steps in the last layer in the pulse neural network branch, performing back propagation training on the pulse neural network branch through the loss function to obtain a local federal pulse neural network, inputting network parameters uploaded by a plurality of local industrial equipment into a central server for aggregation to obtain a global model, and effectively solving the disadvantage of low accuracy when the pulse neural network is used for flaw detection while realizing light-weight distribution collaborative training.

Description

A Federated Learning Method of Spiking Neural Networks for Defect Detection

Technical Field

The present invention relates to the technical field of federated learning and pulse neural network, and in particular to a federated learning method of a pulse neural network for defect detection.

Background Art

With the increasing popularity of edge devices in industrial scenarios, small edge devices are increasingly widely used. For example, in factory environments, cameras, sensors, and Raspberry Pis are used to detect production processes. These small devices have certain computing capabilities, but they are weak. How to use the computing power of these small edge devices to detect product appearance defects on the production line, thereby effectively improving the yield rate. Traditional distributed computing learning methods and large deep learning models are not suitable for such small edge devices. Therefore, a new distributed collaborative learning method is needed that can achieve lightweight model deployment on small edge devices.

With the emergence of the third-generation neural network model, the data in the pulse neural network branch is encoded with the spatial information of the neuron pulse signal. The input and output of the network and the information transmission between neurons are expressed as the pulses sent by the neurons and the time information of the pulses sent. Neurons need to run in parallel at the same time; event-driven computing is achieved by simulating the pulse coding and synaptic plasticity of the biological nervous system, and it has computing characteristics that are more similar to artificial nervous systems. Due to its low energy consumption, small amount of calculation, and fast reasoning speed, it is very suitable for small industrial edge devices, but the discontinuity of the neuron model of the pulse neural network branch, the complexity of the coding, and the uncertainty of the network structure make it difficult to complete the mathematical description of the entire network, and it is easy to have noise spikes, so it is difficult to limit its computing scale and accuracy.

Summary of the invention

The present invention provides a federated learning method of a pulse neural network for defect detection, the purpose of which is to improve the accuracy of defect detection of small edge devices.

In order to achieve the above object, the present invention provides a federated learning method of a pulse neural network for defect detection, comprising:

Step 1: Obtain product images through multiple local industrial devices, and pre-process the product images to obtain receptive field matrix encoding and defect type labels;

Step 2, the central server sends the constructed initial federated spiking neural network including the artificial neural network branch and the spiking neural network branch to each local industrial device, and allocates a time window to each local industrial device, wherein the output end of the activation layer of the artificial neural network branch and the output end of each time step in the spiking neural network branch are connected to the input end of the multiplier in the spiking neural network branch, the multiplier is used to fuse the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch, and the artificial neural network branches share the network parameters of the spiking neural network branch;

Step 3: The local industrial equipment inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix encoding into the spiking neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch during the forward propagation process of the spiking neural network branch.

Step 4: The local industrial equipment constructs a loss function based on the features and defect category labels outputted at all time steps in the last layer of the spiking neural network branch, and performs back-propagation training on the spiking neural network branch based on the back-propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federated spiking neural network.

Step 5: The central server aggregates the network parameters of the local federated pulse neural network uploaded by multiple local industrial devices to obtain a global model and updates the time window assigned by the central server to the local industrial device to the time window of the current network parameter aggregation;

Step 6, determine whether the global model meets the preset training conditions; if so, the training ends; otherwise, the global model is transmitted to multiple local industrial devices as the initial federated pulse neural network in step 2 and the updated time window, and return to execute step 3.

Furthermore, the product image is preprocessed to obtain the receptive field matrix encoding and defect type labels, including:

Step 11, using an autoencoder to perform image denoising on the product image to obtain a denoised product image, and using histogram equalization to perform contrast enhancement on the denoised product image to obtain a product image with highlighted defect areas and mark them to obtain a defect type label;

Step 12, averaging the product images of the prominent defective areas to obtain an average pixel value matrix;

Step 13, constructing a Gaussian mixture model according to the average pixel value matrix, and optimizing the model parameters of the Gaussian mixture model using the maximum expectation algorithm;

Step 14, calculating the probability density of each Gaussian component in the image plane according to the optimized Gaussian mixture model to obtain a receptive field matrix;

Step 15, encoding the enhanced product image according to the receptive field matrix and the average pixel value matrix to obtain the receptive field matrix encoding.

More specifically, step 11 includes:

Input the product image into the autoencoder, which includes an encoder and a decoder;

The product image is compressed through an encoder to obtain a low-dimensional feature vector;

The low-dimensional feature vector is input into the decoder for reconstruction to obtain the denoised product image;

Calculate the grayscale histogram of the denoised product image;

Perform equalization processing on the histogram to obtain a equalized histogram;

Map pixel grayscale according to the equalized histogram;

Contrast enhancement is performed on the denoised product image according to the pixel grayscale to obtain a product image with highlighted defect areas;

The product images with prominent defect areas are marked to obtain defect type labels.

More specifically, step 13 includes:

Construct a Gaussian mixture model based on the mean pixel value matrix;

Initialize the parameters of the Gaussian mixture model;

The average pixel value matrix is vectorized and the objective function of the maximum expectation algorithm is defined as:

in, represents the mean of each Gaussian component, represents the variance of each Gaussian component, Initialize to , the number of Gaussian components is initialized to the number of defect classification types, represents an implicit function, represents the pixel value of the average pixel value matrix;

Repeat the E step of the maximum expectation algorithm to calculate the probability that each pixel value belongs to each Gaussian model;

The objective function is maximized repeatedly through the M-step maximum expectation algorithm to obtain the parameter estimation of the Gaussian model until a Gaussian mixture model that meets the preset optimization requirements is obtained and the calculation is stopped.

Furthermore, the expression of the receptive field matrix encoding is:

in, Represents the normalized average pixel value matrix Middle Row, No. Column elements, Represents the average pixel value matrix with the receptive field matrix added The Row, No. Column elements, , represents the receptive field matrix.

Furthermore, the central server calculates the The time window for round communication with local industrial equipment, the expression of the trade-off function is:

in, Indicates that local industrial equipment is The time window during the round of communication indicates that the local industrial equipment The time window for round communication, , represents the parameters for adjusting the trade-off function, Indicates the actual training time of local industrial equipment. Represents the accuracy of the training set.

Specifically, the expression for backpropagation is:

in, represents the loss function, represents the first branch of the spiking neural network The weight value of the layer, Represents the output value of the activation function ReLU in the artificial neural network branch, Indicates that the spiking neural network branch is The output value of the time step, Linear differential equations representing the branches of the spiking neural network, represents a symbolic function used to map the activation positions of artificial neural network branches on the feature map, represents the branch of the spiking neural network Layer The output value of the time step, Indicates the branch of the artificial neural network The bias term of the neurons in the layer.

Specifically, the design process of the dynamic feedback adaptive threshold mechanism includes:

Definition Local industrial equipment in the average membrane potential strength during the round;

Definition The effective spike value and noise spike value of a round measure the average number of effective spikes and the number of noise spikes, respectively;

According to the average membrane potential intensity, the number of effective spikes, and the number of noise spikes, the The noise activation factor of the round;

The update formula of the dynamic feedback adaptive threshold mechanism constructed according to the noise activation factor is:

;

in, Indicates The threshold in the round, represents the adjustment factor, Indicates The noise activation factor of the round, , Indicates The loss function value in the round, represents the number of noise spikes, represents the number of effective spikes, represents the average membrane potential intensity, is a constant 1e-6, used to prevent the denominator from being zero.

Furthermore, when the global model meets the preset training conditions, the image of the product to be inspected is input into the federated pulse neural network for inspection to obtain the defect detection result.

The above scheme of the present invention has the following beneficial effects:

The present invention obtains product images through multiple local industrial devices, and pre-processes the product images to obtain receptive field matrix codes and defect category labels; the central server sends the constructed initial federal pulse neural network including artificial neural network branches and pulse neural network branches to each local industrial device, and allocates a time window to each local industrial device, wherein the output end of the activation layer of the artificial neural network branch and the output end of each time step in the pulse neural network branch are connected to the input end of the multiplier in the pulse neural network branch, and the multiplier is used to fuse the features output by the artificial neural network branch with the features output at each time step in the pulse neural network branch, and the artificial neural network branches share the network parameters of the pulse neural network branch; the local industrial equipment inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix code into the pulse neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output at each time step in the pulse neural network branch during the forward propagation process of the pulse neural network branch; a loss function is constructed according to the features output at all time steps in the last layer of the pulse neural network branch and the defect category label, and the pulse neural network branch is back-propagated and trained based on the back-propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federal pulse neural network. neural network; the central server aggregates the network parameters of the local federated pulse neural network uploaded by multiple local industrial devices to obtain a global model and updates the time window assigned by the central server to the local industrial device to the time window when the network parameters are currently aggregated; determines whether the global model meets the preset training conditions; if so, the training ends; otherwise, the global model is used as the initial federated pulse neural network in step 2 and the updated time window is transmitted to multiple local industrial devices, and returns to execute step 3; compared with the prior art, the present invention inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix encoding into the pulse neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output at each time step in the pulse neural network branch during the forward propagation process of the pulse neural network branch, thereby effectively alleviating the influence of spike noise on the pulse neural network branch; a loss function is constructed based on the features output at all time steps in the last layer of the pulse neural network branch and the defect type label, and the pulse neural network branch is back-propagated and trained based on the back-propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federated pulse neural network, which effectively solves the disadvantage of low accuracy of the pulse neural network branch when used for defect detection while realizing lightweight distributed collaborative training.

Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of an embodiment of the present invention;

FIG. 2 is a schematic diagram of a model framework according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

In view of the existing problems, the present invention provides a federated learning method of a pulse neural network for defect detection.

In an embodiment of the present invention, the framework of federated learning includes: a central server and multiple local clients;

The federated learning process is as follows:

The local industrial equipment is used as the local client in the federated learning. In the embodiment of the present invention, the local industrial equipment can be a small industrial edge device, such as a camera, sensor or Raspberry Pi on a production line, which is used to collect product images as a local data set. The product can be steel, cloth, hot-rolled strip steel, etc. The local data set is used to train the initial federated pulse neural network sent by the central server to obtain a local federated pulse neural network.

In the embodiment of the present invention, the central server in the federated learning can be a computing device such as a desktop computer, a notebook, a PDA, a server, a server cluster, and a cloud server, which is used to construct an initial federated spiking neural network, allocate time windows to local industrial devices, and aggregate local federated spiking neural networks uploaded by multiple local industrial devices to obtain a global model;

Finally, the global model is used to perform appearance defect detection on the product image to be inspected. For example, when the global model performs appearance defect detection on the steel image, the defect detection result obtained is whether the steel has scratches, cracks, etc.; when the global model performs appearance defect detection on the cloth image, the defect detection result obtained is whether the cloth has stains, pockmarks, or patches.

As shown in FIGS. 1 and 2 , an embodiment of the present invention provides a federated learning method of a pulse neural network for defect detection, including:

Step 1: Obtain product images through multiple local industrial devices, and pre-process the product images to obtain receptive field matrix encoding and defect type labels;

Step 2, the central server sends the constructed initial federated spiking neural network including the artificial neural network branch and the spiking neural network branch to each local industrial device, and allocates a time window to each local industrial device, wherein the output end of the activation layer of the artificial neural network branch and the output end of each time step in the spiking neural network branch are connected to the input end of the multiplier in the spiking neural network branch, the multiplier is used to fuse the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch, and the artificial neural network branches share the network parameters of the spiking neural network branch;

Step 3: The local industrial equipment inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix encoding into the spiking neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch during the forward propagation process of the spiking neural network branch.

Step 4: The local industrial equipment constructs a loss function based on the features and defect category labels outputted at all time steps in the last layer of the spiking neural network branch, and performs back-propagation training on the spiking neural network branch based on the back-propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federated spiking neural network.

Step 5: The central server aggregates the network parameters of the local federated pulse neural network uploaded by multiple local industrial devices to obtain a global model and updates the time window assigned by the central server to the local industrial device to the time window of the current network parameter aggregation;

Step 6, determine whether the global model meets the preset training conditions; if so, the training ends; otherwise, the global model is transmitted to multiple local industrial devices as the initial federated pulse neural network in step 2 and the updated time window, and return to execute step 3.

Specifically, the product image is preprocessed to obtain the receptive field matrix encoding and defect type labels, including:

Step 11, using an autoencoder to perform image denoising on the product image to obtain a denoised product image, and using histogram equalization to perform contrast enhancement on the denoised product image to obtain a product image with highlighted defect areas and mark them to obtain a defect type label;

Step 12, averaging the product images of the prominent defective areas to obtain an average pixel value matrix;

Step 13, constructing a Gaussian mixture model according to the average pixel value matrix, and optimizing the model parameters of the Gaussian mixture model using the maximum expectation algorithm;

Step 14, calculating the probability density of each Gaussian component in the image plane according to the optimized Gaussian mixture model to obtain a receptive field matrix;

Step 15, encoding the enhanced product image according to the receptive field matrix and the average pixel value matrix to obtain the receptive field matrix encoding.

More specifically, step 11 includes:

Input the product image into the autoencoder, which includes an encoder and a decoder;

The product image is compressed through an encoder to obtain a low-dimensional feature vector;

The low-dimensional feature vector is input into the decoder for reconstruction to obtain the denoised product image;

Calculate the grayscale histogram of the denoised product image;

Perform equalization processing on the histogram to obtain a equalized histogram;

Map pixel grayscale according to the equalized histogram;

Contrast enhancement is performed on the denoised product image according to the pixel grayscale to obtain a product image with highlighted defect areas;

The product images with prominent defect areas are marked to obtain defect type labels.

In the embodiment of the present invention, considering that the product image may be affected by factors such as noise and on-site environment, the embodiment of the present invention first pre-processes the product image to improve the quality of the input image;

First, an autoencoder is used for image denoising. The autoencoder is an unsupervised deep learning model consisting of an encoder and a decoder. The encoder and the decoder are usually represented by a multi-layer neural network. The encoder maps the input product image to a low-dimensional feature, which is expressed as:

in, A tensor of pixel values representing a product image, Represents low-dimensional features;

The decoder reconstructs the denoised product image from the low-dimensional features:

in, represents the denoised product image;

The goal of the autoencoder is to minimize the reconstruction loss, which is expressed as:

Through training, the denoised product image is made as close as possible to the original image, thereby removing the noise.

Then histogram equalization is used to enhance the contrast of the image, thereby highlighting the background and defects. The process is as follows:

Calculate the grayscale histogram of the image. The formula is as follows:

Equalize the histogram:

Remap pixel grayscale according to the equalized histogram:

in, Indicates the size of the image pixel value; histogram equalization expands the grayscale range of the image and enhances the contrast. The grayscale distribution in the background area is relatively concentrated. After equalization, the contrast with the defective part is increased, thereby highlighting the defective area.

Specifically, step 12 includes:

Enhanced product images of each local industrial equipment The average pixel value matrix is obtained by averaging, and the expression is as follows:

in, represents the average pixel value matrix, represents the number of enhanced product images, represents the nth enhanced product image, , , ;

Normalize the average pixel value matrix to get:

in, Represents the average pixel value matrix No. Row, No. Column elements, and Represents the average pixel value matrix The maximum and minimum pixel values.

Specifically, step 13 includes:

The Gaussian mixture model is constructed based on the average pixel value matrix:

in, represents the Gaussian function expression, Indicates the number of defect classification types;

From this, the expression of the average pixel value matrix mixed with the receptive field matrix is:

represents the receptive field matrix;

The maximum expectation algorithm is used to optimize the model parameters of the Gaussian mixture model;

It should be noted that the maximum expectation algorithm EM algorithm adopted in the embodiment of the present invention is a conventional iterative optimization algorithm. The EM algorithm performs calculations alternately in two steps. The first step is to calculate the expectation (E), referred to as the E-step, and use the existing estimated values of the hidden variables to calculate their maximum likelihood estimates; the second step is maximization (M), referred to as the M-step, which maximizes the maximum likelihood value obtained in the E-step to calculate the value of the parameter. The parameter estimate found in the M-step is used in the next E-step calculation, and this process is continuously alternated until the model converges.

The embodiment of the present invention uses the maximum expectation algorithm to optimize the model parameters of the Gaussian mixture model. The specific process is as follows:

Parameters of the Gaussian mixture model Initialize;

The average pixel value matrix After vectorization expansion, we get:

The objective function of the maximum expectation algorithm is defined as:

in, represents the mean of each Gaussian component, represents the variance of each Gaussian component, Initialize to , the number of Gaussian components is initialized to the number of defect classification types;

By repeatedly calculating the probability of each pixel value belonging to each Gaussian model through the E step of the maximum expectation algorithm, the hidden parameter The expression is:

in, represents the number of iterations, represents the pixel value of the average pixel value matrix;

By repeatedly maximizing the objective function through the M-step maximum expectation algorithm, the parameter estimate of the Gaussian model is obtained as follows:

The calculation is stopped until a Gaussian mixture model that meets the preset optimization requirements is obtained.

Specifically, step 14 includes:

According to the optimized Gaussian mixture model, the probability density of each Gaussian component in the image plane is calculated to obtain the receptive field matrix , the receptive field matrix strengthens the receptive field of the defective part and weakens the interference caused by background information.

More specifically, step 15 includes:

In the embodiment of the present invention, through the above processing, the normalized average pixel value matrix is obtained: And the receptive field matrix , given For the defect detection task, the embodiment of the present invention mainly focuses on the defect part of the image rather than the background information. Therefore, the enhanced product image is encoded according to the receptive field matrix and the average pixel value matrix, and the defect part in the product image is encoded as 1 and the background part is encoded as 0. The formula for encoding the receptive field matrix is as follows:

in, Represents the normalized average pixel value matrix Middle Row, No. Column elements, Represents the average pixel value matrix with the receptive field matrix added The Row, No. Column elements.

Specifically, in the embodiment of the present invention, the pulse neural network branch score imitates the way of transmitting information between biological neurons, and transmits information in the form of neural pulses, which has the advantages of low energy consumption and small amount of calculation. The pulse neural network branch model adopts the leaky current integrated discharge neuron (Leaky-Integration-and-Fire) model, and its linear differential equation expression is as follows:

in, is the leaky factor, , the number of layers of the pulse neural network branch is L, Representing neurons With Layer of neurons The weight of the connection, Indicates that the spiking neural network branch is Layer of neurons exist The output value at this moment, Neurons in layer exist The output value at the moment can be expressed as:

in, Represents the threshold for the pth iteration round.

Specifically, the activation function of the artificial neural network score in the embodiment of the present invention adopts the ReLU function, whose parameters are shared with the parameters of the pulse neural network branch, and the mapping formula of the middle layer is:

in, Representing neurons With Layer of neurons The weight of the connection, Indicates Layer of neurons The bias term, Represents the neurons in the artificial neural network branch The output value after the activation function is on the local industrial equipment. , The weight parameters of the spiking neural network branch are shared.

In the embodiment of the present invention, a batch normalization method with time step update is used, taking into account that the input of the pulse neural network branch has a time series feature. The specific formula is as follows:

in, represents the learned weights of the batch normalization layer, is a constant that prevents the denominator from being zero. , Indicates Layer The mean and variance of the batch samples at time t, for the update of the batch normalization layer, given , weight The gradient of can be expressed as:

in, , so the time step The learned weights of the batch normalization layer are updated as follows:

in, is the learning rate.

In the embodiment of the present invention, the pulse neural network branch received by the local industrial equipment adopts a pulse-based space-time back propagation algorithm (STBP), and its back propagation formula is:

Among them, for the step signal In order to solve the problem of being unable to derive, the STBP algorithm designed a rectangular function as the pulse gradient for back propagation:

in, Indicates the width of the gradient.

In the embodiment of the present invention, in order to balance the computing power heterogeneity and training accuracy of local industrial equipment, a dynamic time window parameter T is designed so that the local industrial equipment can adaptively adjust the time window parameter T according to its training performance and equipment performance. The specific process is as follows:

(1) Set the initial time window ：The central server assigns an initial time window to each local industrial device ,The initial time window can be allocated based on factors such as global ,performance estimation, device performance, data size, etc.;

(2) Local industrial equipment training process: Local industrial equipment trains the pulse neural network branch in each time window The client needs to record the following data during the training process: the training time of one round , training set accuracy , the time window of the current local industrial equipment ;

(3) Local industrial equipment uploads data: When local industrial equipment communicates with the central server, it uploads the training time of one round. , training set accuracy , the time window of the current local industrial equipment And other parameters;

(4) Adjusting dynamic time window parameters: The central server uses a trade-off function to calculate the time window parameters for the i+1th round of communication based on the uploaded data. To balance training time and training accuracy, the trade-off function is as follows:

in, , is a parameter that adjusts the trade-off and can be adjusted according to specific needs, for example, by increasing Make it more accurate, and by increasing The goal of this trade-off function is to adaptively adjust the time window based on the training performance of local industrial equipment. ,The time window can only be a positive integer, so that local industrial equipment can have appropriate training time under different performance and accuracy requirements;

(5) Reallocate time windows: The central server allocates new time windows based on the calculations. , which can be assigned to local industrial equipment, replacing the previous time window , so that the local industrial equipment will use the new time window parameters in the next round of training ;

(6) Repeat training and adjustment: Repeat the above steps and iterate continuously to gradually adjust the time window parameters so that the training time and accuracy of local industrial equipment are balanced.

The embodiment of the present invention takes into account the spike noise and accuracy issues of the spike neural network branches, and effectively alleviates the impact of the spike noise of the spike neural network branches by integrating the features of the artificial neural network branch outputs, thereby improving the detection accuracy; the specific process is as follows:

Design feature fusion mechanism for network branches:

The local industrial equipment will train two network branches. The model parameters of the two network branches are shared, and the two network branches share the same data set collected by the local industrial equipment. For the pulse neural network branch, the input image data needs to be encoded, and its specific time window is allocated according to factors such as global performance estimation, equipment performance, and data scale. For the artificial neural network branch, it is used as the input of the artificial neural network branch after enhancement processing;

Considering that the spiking neural network branch may activate the originally inactive neuron position in the artificial neural network branch, resulting in the generation of noise spikes, the embodiment of the present invention solves the noise spike problem of the spiking neural network by fusing the feature layer knowledge of the two networks. The expression is as follows:

in, Represents the output value of the activation function ReLU in the artificial neural network branch, It is represented as the output value of the spike neural network branch of the nth layer at the tth time step, is a symbolic function used to convert Map to , used to map the activation positions of artificial neural network branches on the feature map, is the dot product symbol, the noise of the pulse neural network is filtered by the dot product method, and the filtered output value is further forward propagated through the pulse neural network;

Design a back-propagation mechanism for local industrial equipment model training:

First, define the loss function of the overall network. Since the back propagation of the artificial neural network branch is removed in the back propagation design, the loss function expression is:

in, Represents the total number of network layers, represents the output layer of the spike neural network branch The output value of the time step, Indicates The time window size of each local industrial device, Represents the number of neurons in the output layer of the spiking neural network branch, that is, the total number of defect classification categories.

Due to the fusion of the features of the two networks, the back propagation formula is:

in, represents the loss function, Represents the weight value of the nth layer in the spiking neural network branch. Since the training is in the spiking neural network branch, the artificial neural network branch does not participate in the back propagation update of the network. ; Combined with the back propagation formula of the pulse neural network branch, the final back propagation formula is:

in, represents the output value of the spiking neural network branch at the tth time step, Linear differential equations representing the branches of the spiking neural network, represents a symbolic function used to map the activation positions of artificial neural network branches on the feature map, represents the output value of the nth layer of the spiking neural network branch at the tth time step, Represents the bias term of the nth layer of neurons in an artificial neural network branch.

Design of dynamic feedback adaptive threshold mechanism for spiking neural network branches:

Noise spikes are an important factor affecting the accuracy of spiking neural network branch detection. In view of the possible noise spikes in spiking neural network branches, the embodiment of the present invention integrates the feature layer information of the artificial neural network branches and the previous round of loss values to design a dynamic feedback adaptive threshold mechanism:

First define For the Local industrial equipment The average membrane potential intensity of each round is:

in, Indicates The number of product images in local industrial devices, Represents the total number of network layers, Indicates The time window size of each local industrial device, Indicated in Time The membrane potential value of the layer;

According to the feature fusion method described above, the embodiment of the present invention defines the first The effective spike value and noise spike value of the round measure the average number of effective spikes. and the number of noise spikes :

in, represents the set of locations of noisy spikes, Represents the entire feature map location set; by fusing the The loss function value in the round , the embodiment of the present invention defines Noise activation factor for the round:

Among them, the noise activation factor Respectively by , , , Impact, among which and The larger the value is, the smaller the noise factor is, indicating that the threshold does not need to be adjusted significantly in the p+1th round. , The larger the value is, the larger the noise activation factor is, indicating that in the p+1th round, the threshold needs to be increased to reduce the number. Therefore, the dynamic feedback adaptive threshold mechanism update formula can be defined as:

in, Indicates The threshold in the round, represents the adjustment factor, It is used to control the influence of the threshold of the previous round on the threshold of this round. Used to adjust the impact of the noise activation factor on the threshold of this round. Indicates The noise activation factor of the round, Indicates The loss function value in the round, represents the number of noise spikes, represents the number of effective spikes, represents the average membrane potential intensity, is a constant 1e-6, used to prevent the denominator from being zero.

Finally, the central server aggregates the network parameters of the local federated pulse neural networks uploaded by multiple local industrial devices to obtain the global model and updates the time window assigned to the local industrial devices by the central server to the time window when the network parameters are currently aggregated; determines whether the global model meets the preset training conditions; if so, the training ends; otherwise, transmits the global model as the initial federated pulse neural network in step 2 and the updated time window to multiple local industrial devices, and returns to execute step 3.

Specifically, when the global model meets the preset training conditions, the image of the product to be inspected is input into the federated pulse neural network for inspection to obtain the defect detection result; when the global model performs appearance defect detection on the steel image, the defect detection result obtained is whether the steel has scratches, cracks, etc.; when the global model performs appearance defect detection on the cloth image, the defect detection result obtained is whether the cloth has stains, pockmarks, or plaques.

The embodiment of the present invention obtains product images through multiple local industrial devices, and pre-processes the product images to obtain receptive field matrix codes and defect category labels; the central server sends the constructed initial federated spiking neural network including artificial neural network branches and spiking neural network branches to each local industrial device, and allocates a time window to each local industrial device, wherein the output end of the activation layer of the artificial neural network branch and the output end of each time step in the spiking neural network branch are connected to the input end of the multiplier in the spiking neural network branch, and the multiplier is used to fuse the features output by the artificial neural network branch with the features output at each time step in the spiking neural network branch, and the artificial neural network branches share the network parameters of the spiking neural network branch; the local industrial device inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix code into the spiking neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output at each time step in the spiking neural network branch during the forward propagation process of the spiking neural network branch; a loss function is constructed according to the features output at all time steps in the last layer of the spiking neural network branch and the defect category label, and the spiking neural network branch is back-propagated and trained based on the back-propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federated spiking neural network. a local federated spike neural network; aggregate the network parameters of the local federated spike neural network uploaded by multiple local industrial devices to obtain a global model and update the time window assigned by the central server to the local industrial device to the time window when the network parameters are aggregated for the current time; determine whether the global model meets the preset training conditions; if so, the training ends; otherwise, the global model is used as the initial federated spike neural network in step 2 and the updated time window is transmitted to multiple local industrial devices, and return to execute step 3; compared with the prior art, the present invention inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix encoding into the spike neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output at each time step in the spike neural network branch during the forward propagation process of the spike neural network branch, thereby effectively alleviating the influence of spike noise on the spike neural network branch; construct a loss function based on the features output at all time steps in the last layer of the spike neural network branch and the defect type label, and perform back propagation training on the spike neural network branch based on the back propagation mechanism, the dynamic feedback adaptive threshold mechanism, and the loss function to obtain multiple trained local federated spike neural networks, which effectively solves the disadvantage of low accuracy of the spike neural network branch when used for defect detection while realizing lightweight distributed collaborative training.

The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims (9)

1. A federated learning method of a pulse neural network for defect detection, comprising: Step 1: Acquire product images through multiple local industrial devices, and pre-process the product images to obtain receptive field matrix encoding and defect type labels; Step 2, the central server sends the constructed initial federated spiking neural network including an artificial neural network branch and a spiking neural network branch to each of the local industrial devices, and allocates a time window to each of the local industrial devices, wherein the output end of the activation layer of the artificial neural network branch and the output end of each time step in the spiking neural network branch are connected to the input end of the multiplier in the spiking neural network branch, the multiplier is used to fuse the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch, and the artificial neural network branches share the network parameters of the spiking neural network branch; Step 3, the local industrial equipment inputs the product image into the artificial neural network branch for forward propagation, inputs the receptive field matrix encoding into the spiking neural network branch within the time window for forward propagation, and fuses the features output by the artificial neural network branch with the features output by each time step in the spiking neural network branch during the forward propagation process of the spiking neural network branch; Step 4, the local industrial equipment constructs a loss function according to the features of all time step outputs in the last layer of the spiking neural network branch and the defect type label, and performs back propagation training on the spiking neural network branch based on a back propagation mechanism, a dynamic feedback adaptive threshold mechanism, and the loss function to obtain a local federated spiking neural network; Step 5, the central server aggregates the network parameters of the local federated pulse neural network uploaded by the multiple local industrial devices to obtain a global model and updates the time window assigned by the central server to the local industrial device to the time window of the current network parameter aggregation; Step 6, determine whether the federated pulse neural network meets the preset training conditions; if so, the training ends; otherwise, the federated pulse neural network is used as the initial federated pulse neural network in step 2 and the updated time window is transmitted to multiple local industrial devices, and return to execute step 3. 2. The method for federated learning of pulse neural networks for defect detection according to claim 1, characterized in that the preprocessing of the product image to obtain the receptive field matrix encoding and the defect type label comprises: Step 11, using an autoencoder to perform image denoising on the product image to obtain a denoised product image, and using histogram equalization to perform contrast enhancement on the denoised product image to obtain a product image with highlighted defect areas and mark them to obtain a defect type label; Step 12, averaging the product images of the prominent defective areas to obtain an average pixel value matrix; Step 13, constructing a Gaussian mixture model according to the average pixel value matrix, and optimizing the model parameters of the Gaussian mixture model using a maximum expectation algorithm; Step 14, calculating the probability density of each Gaussian component in the image plane according to the optimized Gaussian mixture model to obtain a receptive field matrix; Step 15: Encode the enhanced product image according to the receptive field matrix and the average pixel value matrix to obtain a receptive field matrix code. 3. The method for federated learning of pulse neural networks for defect detection according to claim 2, wherein step 11 comprises: Inputting the product image into the autoencoder, wherein the autoencoder includes an encoder and a decoder; Compressing the product image by the encoder to obtain a low-dimensional feature vector; Inputting the low-dimensional feature vector into the decoder for reconstruction to obtain a denoised product image; Calculating a grayscale histogram of the denoised product image; Performing equalization processing on the histogram to obtain a equalized histogram; Mapping pixel grayscale according to the equalized histogram; Performing contrast enhancement on the denoised product image according to the pixel grayscale to obtain a product image with highlighted defect areas; The product image of the prominent defect area is labeled to obtain a defect type label. 4. The method for federated learning of pulse neural networks for defect detection according to claim 3, wherein step 13 comprises: Constructing a Gaussian mixture model according to the average pixel value matrix; Initializing parameters of the Gaussian mixture model; The average pixel value matrix is vectorized and expanded, and the objective function of the maximum expectation algorithm is defined as: in, represents the mean of each Gaussian component, represents the variance of each Gaussian component, Initialize to , the number of Gaussian components is initialized to the number of defect classification types, represents an implicit function, represents the pixel value of the average pixel value matrix; Repeating the calculation of the probability that each pixel value belongs to each Gaussian model through step E of the maximum expectation algorithm; The objective function is repeatedly maximized by the M-step maximum expectation algorithm to obtain parameter estimates of the Gaussian model until a Gaussian mixture model that meets the preset optimization requirements is obtained, and the calculation is stopped. 5. The federated learning method of pulse neural network for defect detection according to claim 4, characterized in that the expression for encoding the enhanced product image according to the receptive field matrix and the average pixel value matrix is: in, Represents the normalized average pixel value matrix Middle Row, No. Column elements, Represents the average pixel value matrix with the receptive field matrix added The Row, No. Column elements, , represents the receptive field matrix. 6. The federated learning method of pulse neural network for defect detection according to claim 5, characterized in that: The central server calculates the The time window for round communication with the local industrial device, the expression of the trade-off function is: in, Indicates that local industrial equipment is The time window during the round of communication indicates that the local industrial equipment The time window for round communication, , represents the parameters for adjusting the trade-off function, Indicates the actual training time of local industrial equipment. Represents the accuracy of the training set. 7. The federated learning method of pulse neural network for defect detection according to claim 6, characterized in that the expression for back propagation is: in, represents the loss function, represents the first branch of the spiking neural network The weight value of the layer, Represents the output value of the activation function ReLU in the artificial neural network branch, Indicates that the spiking neural network branch is The output value of the time step, The linear differential equations representing the branches of the spiking neural network, represents a symbolic function used to map the activation positions of artificial neural network branches on the feature map, represents the branch of the spiking neural network Layer The output value of the time step, Indicates the branch of the artificial neural network The bias term of the neurons in the layer. 8. The federated learning method of pulse neural network for defect detection according to claim 7, characterized in that the design process of the dynamic feedback adaptive threshold mechanism includes: Definition Local industrial equipment in the average membrane potential strength during the round; Definition The effective spike value and noise spike value of a round measure the average number of effective spikes and the number of noise spikes, respectively; According to the average membrane potential intensity, the number of effective spikes, and the number of noise spikes, define the The noise activation factor of the round; The dynamic feedback adaptive threshold mechanism update formula constructed according to the noise activation factor is: ; in, Indicates The threshold in the round, represents the adjustment factor, Indicates The noise activation factor of the round, , Indicates The loss function value in the round, represents the number of noise spikes, represents the number of effective spikes, represents the average membrane potential intensity, is a constant 1e-6, used to prevent the denominator from being zero. 9. The federated learning method of pulse neural network for defect detection according to claim 8, characterized in that: When the federated pulse neural network meets the preset training conditions, the image of the product to be detected is input into the federated pulse neural network for detection to obtain a defect detection result.