CN112819139A - Optimal conversion method from artificial neural network to impulse neural network - Google Patents

Abstract

The invention discloses an optimal conversion method from an artificial neural network to an impulse neural network. The method includes setting a source artificial neural network ann that meets requirements, setting an upper limit for an activation function; training the source artificial neural network ann, and Record the maximum value of the output of each layer of the neural network; build a spiking neural network snn with the same network structure as the source artificial neural network ann but remove the activation function of each layer, and convert the network weight and bias of each layer in the source artificial neural network ann The bias is copied to each layer of the spiking neural network snn, and the recorded maximum output value a _l of each layer of the source artificial neural network ann is used as the threshold value of the corresponding layer of the spiking neural network snn

Set a desired simulation time T, according to the formula

The bias bias of each layer in the spiking neural network snn increases the offset

to minimize the conversion error. Through the above scheme, the present invention achieves the theoretical minimum value of conversion error, and has high practical value and popularization value.

Description

An optimal conversion method from artificial neural network to spiking neural network

technical field

The invention belongs to the technical field of neural networks, and specifically relates to an optimal conversion method from an artificial neural network to an impulse neural network.

Background technique

Spike Neuronal Networks is known as a new generation of neural network models, which are composed of spiking neural units that simulate biological neurons and are good at processing discrete pulse signals. A spiking neuron only emits a single pulse when activated, the pulse size is constant, and the information is contained in the timing and frequency of the pulse. The unique impulse signal transmission mechanism and event-driven characteristics make the inference running speed of the impulse neural network on the relevant hardware extremely fast and the energy consumption is much lower than that of the traditional artificial neural network1. However, due to the discrete nature of the spiking sequence, the spiking neural network cannot directly use backpropagation for inference training, so how to train a complex and efficient spiking neural network is a difficulty in the field.

There are three main directions of training methods for spiking neural networks, namely biological heuristics 2, 3, instead of gradients 4, 5, 6 (Surrogate Gradient) and conversion from artificial neural networks 7, 8 (ANN-to-SNN). The depth of the network they are suitable for and the length of simulation time required vary: instead of gradients and biological heuristics for shallower and simpler spiking neural networks, network transformations can be used for more complex spiking neural networks; biological heuristics and networks The transformation method requires a long simulation time length (usually greater than 1000), while the replacement gradient requires only a simulation length of less than 100 to complete complex tasks. The performance of the network conversion method on various tasks is the best among the three. It can often achieve results that are very close to the artificial neural network. However, due to the long simulation time required, the running speed and The energy consumption is higher than that of the alternative gradient method. The nearest threshold balance is the most common network conversion method. It first builds a traditional neural network with the same structure as the spiking neural network and the activation function uses the ReLU function. On the training set, the constructed source neural network is first trained, and the network will be recorded every time. The maximum activation value output by the activation function of a layer. After the training is completed, the weights of the source neural network are directly copied to the spiking neural network, and finally the recorded maximum activation value is set as the corresponding layer membrane potential threshold of the spiking neural network, thus completing a reliable spiking neural network. The information passed between the layers of the source network is discretized in the spiking neural network, and the spiking frequency is approximately equal to the value of this information.

Although the network conversion method has the best network performance on various complex tasks, it still has a certain gap from artificial neural networks, and the long simulation time weakens the energy-saving characteristics of spiking neural networks. There are currently a range of approaches to ameliorate the shortcomings of network switching, such as: threshold balancing using 99.9% of the maximum value,7 using the pulse frequency between the two layers to adjust the threshold8, using a soft reset mechanism8 (post-pulse membrane potential minus threshold ) instead of reset mechanisms (membrane potential zeroing after a pulse), hybrid training methods9 (using network transformations followed by alternative gradient training), etc. However, they only focus on increasing the pulse frequency and reducing the residual membrane potential between layers, lacking a theoretical basis to explain the reduction of the conversion error, and the currently known simulation results on complex datasets such as ImageNet are still unsatisfactory (simulation time or task performance), additionally frequency tuning 8 and hybrid training 9 add additional non-negligible overhead on the training or transformation steps. Therefore, how to solve the problems existing in the prior art is an urgent problem to be solved by those skilled in the art.

The references mentioned in the background paper are as follows:

1. Kim S, Park S, Na B, etc. Spiking-YOLO: Spike Neural Networks for Energy-Efficient Object Detection [C]//AAAI Artificial Intelligence Conference Proceedings. 2020, 34(07): 11270-11277.

2. Caporale N, Dan Y. Spike timing-dependent plasticity: Hebbian learning rules [J]. Annu. Rev. Neurosci., 2008, 31: 25-46.

3. Kheradpisheh S R, Ganjtabesh M, Thorpe S J et al. Spiking Deep Convolutional Neural Networks Based on STDP for Object Recognition [J]. Neural Networks, 2018, 99: 56-67.

4. Shrestha S B, Orchard G. Slayer: Timing Pulse Error Redistribution [J]. Conference on Advances in Neural Information Processing Systems, 2018, 31: 1412-1421.

5. Wu Y, Deng L, Li G, etc. Directly training spiking neural networks: faster, bigger, better [C] // AAAI Artificial Intelligence Conference Proceedings. 2019, 33: 1311-1318.

6. Neftci E O, Mostafa H, Zenke F. Alternative gradient learning in spiking neural networks [J]. IEEE Journal of Signal Processing, 2019, 36: 61-63.

7. Rueckauer B, Lungu I A, Hu Y, et al. Converting continuous-valued deep networks into efficient event-driven networks for image classification [J]. Frontiers in Neuroscience, 2017, 11:682.

8. Han B, Srinivasan G, Roy K. RMP-SNN: Residual Membrane Potential Neurons, Enabling Deeper Spiking Neural Networks with High Accuracy and Low Latency [C] // Computer Vision and Pattern Recognition at IEEE/CVF Conference Conference Proceedings. 2020:13558-13567.

9. Rathi N, Srinivasan G, Panda P, et al. Enabling Deep Spike Neural Networks by Hybrid Transformation and Spike Timing-Dependent Backpropagation [C] // International Congress of Learning. 2019.

10. Sengupta A, Ye Y, Wang R, etc. In-depth understanding of spiking neural networks: Vgg and residual architecture [J]. Frontiers in Neuroscience, 2019, 13:95.

SUMMARY OF THE INVENTION

In order to overcome the above deficiencies in the prior art, the present invention provides an optimal conversion method from artificial neural network to spiking neural network, so that the task performance of the target spiking neural network is close to that of the source artificial neural network and only needs to be close to that of the alternative gradient method simulation time.

In order to achieve the above object, the technical scheme adopted in the present invention is as follows:

An optimal conversion method from artificial neural network to spiking neural network, comprising the following steps:

(S1) Set a source artificial neural network ann that meets the requirements, and set an upper limit for the activation function;

(S2) training the source artificial neural network ann, and recording the maximum value of the output of each layer of the neural network;

(S3) constructing a spiking neural network snn which is the same as the network model of each layer of the source artificial neural network ann, and removes the activation function of each layer;

(S4) Copy the network weight weight and bias bias of each layer in the source artificial neural network ann to each layer of the spiking neural network snn, and use the recorded maximum output value a _l of each layer of the source artificial neural network ann as the spiking neural network The threshold of the corresponding layer of snn

脉冲神经网络snn中的每一层的偏差bias增加偏移量

使得转换误差最小化。(S5) Set an expected simulation time T, according to the formula

The bias bias of each layer in the spiking neural network snn increases the offset

to minimize the conversion error.

Specifically, the requirements in the step (S1) include three points, the first is that the activation function has the function of removing the negative value part; the second is that the source artificial neural network ann needs to use an average pooling layer; the third is that the source neural network Batch normalization layers cannot be used on ann.

Compared with the prior art, the present invention has the following beneficial effects:

(1) There is almost no performance gap between the target spiking neural network snn of the present invention and the source artificial neural network ann when performing the same task, and even the use of threshold operations can improve the performance of some artificial neural networks that do not use batch normalization. Since the present invention optimizes the conversion error theoretically, compared with the prior art, the same effect can be achieved by only using about 1/10 of the simulation time. Great help. In addition, compared with the prior art, the present invention derives the conversion error theoretically, and can reach the theoretical minimum value of the conversion error when the test set is unknown.

Description of drawings

FIG. 1 is a schematic diagram of performing threshold and displacement operations on an activation function provided by the present invention.

FIG. 2 is a schematic diagram of converting ANN neurons into SNN neurons through threshold balance in the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and examples. The embodiments of the present invention include but are not limited to the following examples.

Example

The artificial neural network required to convert the spiking neural network has three structural requirements. The first is that the activation function has the function of removing the negative part. As shown in Figure 1, the function of ReLU in the second quadrant of the coordinate axis is directly set as 0, and the image in the first quadrant is similar to the image of the pulse frequency function of the spiking neural network, and has a certain conversion basis; the second is that the source artificial neural network needs to use an average pooling layer instead of the more common max pooling layer, the reason is After converting to a spiking neural network, since the size of the pulse is constant, the max pooling layer has no pooling ability to the pulse, such as two inputs in one kernel of the max pooling layer:

Its output is all 1, which is not discriminative. The third is that batch normalization layers cannot be used on the source neural network, because batch normalization will increase the variance of the activation distribution, making it easier to generate noise spikes early in the simulation.

An optimal conversion method from artificial neural network to spiking neural network in the present invention includes a training step and a conversion step.

Training steps:

(S1) Set up a source artificial neural network ann that meets the requirements of the above three points, and use the activation function h of the ReLU-like function, that is, the output of each layer of the network after receiving the input of the previous layer is x _l =h ( W _l ·x _l-1 ), h is a function of nonlinear transformation, retaining the part of the pre-activation value greater than 0. In the conversion method, the activation function h is set with an upper limit, such as using ReLU6 in MobileNet, that is, setting the upper limit of ReLU to 6, that is, x=clip(x, 0, 6). Since the input of the spiking neural network snn and the source artificial neural network ann are the same image data, it is assumed that the source artificial neural network ann is a network with only one layer, that is, x _l-1 is the image pixel value, so it is hoped that the spiking neural network snn will finally can get the same output as the source artificial neural network ann, so that snn and ann perform the same on this task. Therefore, h' is used to represent the activation function of the spiking neural network, that is, x _l '=h'(W _l ·x _l-1 ). In the case of the same input and weight, if the output is the same, it is necessary to adjust the pulse The activation function h′ is adjusted when the neural network snn principle is changed.

(S2) Train the source artificial neural network ann, and record the maximum output value of each layer of the neural network. For example, the maximum output of the lth layer is a _l , then V _th = a _l in actual FIG. 1 .

Conversion steps:

(S3) constructing a spiking neural network snn which is the same as the network model of each layer of the source artificial neural network ann, and removes the activation function of each layer;

A membrane potential matrix M _l of the same shape as the output is then set to store the neuron's membrane potential v ^l (t). The input of the neuron of the spiking neural network (snn) is a series of pulse sequences. In the present invention, the input of the first layer is the pixel value of the picture, and the other layers are all pulse sequences. For example, after setting the pulse size of the current layer to 1, at each moment, its output can only be 1 or 0. For example, if the simulation time T is 10, and the output of the ann neuron corresponding to a neuron is 0.3, then it will pulse 3 times within T so that its total pulse is expected to be 3/10 = 0.3, the same as the output of ann. The spiking neuron stores a membrane potential internally, it collects the incremental value of the membrane potential (W _l ·x′ _l ), and when it exceeds the threshold potential V _th , it releases a pulse.

这样脉冲神经网络snn就能运行公式(1)，(S4) As shown in Figure 2, copy the network weight weight and bias bias of each layer in ann to each layer of the spiking neural network snn, and use the recorded maximum output value a _l of each layer of the source artificial neural network ann as the corresponding layer of snn the threshold

In this way, the spiking neural network snn can run formula (1),

At each moment, the input is the pixel value of the picture, the output pulse, and the position where the output pulse frequency is the largest is the category of the picture.

超过阈值电位V_th后，神经元会释放大小等于阈值的脉冲信号，使用θ^l(t+1)来表示t+1时刻的脉冲值，它只能等于0或者V_th。The present invention uses the IF model ¹ and the soft reset mechanism ² in the spiking neural network, so that the membrane potential of the 1-layer neuron in the network at time t is v ^l (t), and the received pulse sequence at time t+1 is the value of x _l ′(t+1), then after the neuron receives the impulse signal, the change of its membrane potential is formula (1), where W _l is the connection weight between the neurons in the l layer and the neurons in the l-1 layer, when

After the threshold potential V _th is exceeded, the neuron will release a pulse signal with a magnitude equal to the threshold value, and θ ^l (t+1) is used to represent the pulse value at the time of t+1, which can only be equal to 0 or V _th .

Combined with the soft reset mechanism (post-pulse membrane potential minus threshold V _th ), an updated formula for membrane potential can be obtained:

v ^l (t+1)=v ^l (t)+W _l ·x _l ′(t+1)-θ ^l (t+1) (2)

Then superimpose the above formulas from time 0 to time T, and divide the left and right sides of the equation by T at the same time to obtain formula (3):

来代表网络l层从0到T时刻的平均输入，那么l层的期望输出

同时令网络的初始膜电位v^l(0)为0，可以得到脉冲神经网络的期望输出与平均输入的关系：last use

to represent the average input of network l layer from 0 to T time, then the expected output of l layer

At the same time, let the initial membrane potential v ^l (0) of the network be 0, the relationship between the expected output of the spiking neural network and the average input can be obtained:

Among them, the last part of the formula v ^l (T)/T is the information that still remains in the membrane potential at the end of the simulation, and the first half is similar to the forward process of the artificial neural network, so you can see -v ^l (T)/T as A special activation function is used, and the above equation can be replaced by the following activation function h'( ):

表示向下取整，而clip是截断函数，表示截断小于0和大于T的部分。in,

Indicates rounding down, and clip is a truncation function, which means truncation of parts less than 0 and greater than T.

Correspondingly, using h( ) to represent the activation function of the artificial neural network, formula (6) is obtained:

a _l+1 =h(W _l ·a _l ) (6).

In each layer, our expectation is that the corresponding outputs a' _l+1 on the two networks are closer to a _l+1 , but they have two differences that cause them to always have a certain distance, The first is the difference between the two activation functions. As shown in Figure 1, in the part where the pre-activation value is greater than 0, the change curve of h is a straight line, while the change curve of h' is a stepped curve, and they are always different. Secondly, the input of the current layer ann is not the same as the input of snn. Except that the first layer is input of image pixels, each layer will have errors caused by the activation function. With the accumulation of errors, the gap between the final input will also gradually get bigger.

之间的差值：The loss function is an important function used in network training. It is generally the mean square error or softmax function. For example, the mean square error is (a _L -y) ² , where the former is the output of the last layer of the network, and the latter is what we hope to get For example, in handwritten digit recognition, the input picture is 3, and the value of y is [0,0,1,0,0,0,0,0,0], assuming that the output a _L of the present invention is [0, 0.1, 0.9, 0, 0, 0.1, 0, 0.2, 0.2], the network determines that the neuron in the third position has the highest activation frequency, and recognizes that the number written in this picture is 3. The present invention considers that the performance gap between the converted spiking neural network and the source artificial neural network is caused by the conversion error, which can be considered as the loss function of the network

Difference between:

For the lth layer of the spiking neural network, its activation function can be approximated by the activation function h of the artificial neural network plus an error term: a' _l =h' _l (W _l ·a' _l-1 )= h _l (W _l ·a′ _l-1 )+Δa′ _l , where Δa′ _l is the output error due to the different activation functions of the two networks under the same input condition. However, due to the accumulation of errors, the input of each layer is different except the input layer. The present invention can decompose the output error of each layer into the part Δa′ _l caused by the activation function and the part Δa _{l- 1} ,:

Δa _l :=a′ _l −a _l =Δa′ _l +[h _l (W _l ·a′ _l-1 )-h _l (W _l ·a _l-1 )]≈Δa′ _l +B _l ·W _l Δa _l-1

Among them, the last approximation represents the first-order Taylor expansion, and B _l is the matrix of the first-order derivative of the activation function h _l on the diagonal. At the same time, the present invention uses the second-order Taylor formula to expand the expression of the conversion error:

是在源神经网络ann的损失函数，在训练过程中被最小化了，所以第一项可以认为非常接近0而被忽略。所以转换误差主要存在于第二项中，将输出误差Δa_l的表达式(7)代入(8)的第二项中得到：where H _aL is the Hessian matrix, since

is the loss function of the source neural network ann, which is minimized during training, so the first term can be considered very close to 0 and ignored. So the conversion error mainly exists in the second term, and the expression (7) of the output error _Δal is substituted into the second term of (8) to get:

并替代上式(9)最后一项得：where the intermediate interaction term can be ignored by the decoupling assumption, and from previous work3 it can be obtained

And replace the last term of the above formula (9) to get:

Finally, the forward layer is gradually progressed, and the complex network conversion error is decomposed into the sum of different layers E[Δa′ _l ^T H _al Δa′ _l ]. The equations (10) are independent of each other. A E[Δa′ _l ^T H _al Δa′ _l ] is optimized to the minimum, then the overall network conversion error can be minimized.

In Figure 1, the stepped segmented dotted line is the expected activation function h' of the spiking neural network, and the straight line is the ReLU activation function h used by the source neural network. The upward trend of the two has a certain similarity, but due to h' The maximum value cannot exceed V _th , so when the pre-activation value is much larger than V _th , the error will be very large, so the present invention first considers the threshold operation, that is, an upper bound y _th is set for the ReLU function, and when the activation value is too large, the output is constant is y _th , and the derivative gradient is 0, which limits the translation error when the activation value is too large. The second is the displacement operation. The present invention considers that h' or h can be shifted by a certain distance δ, that is, the offset term b is increased or decreased in the network. Considering that H _al is a constant in formula (10), it is necessary to minimize Δa′ _l ^T Δa′ _l ], that is, the error caused by the activation function under the same input. The present invention considers using a displacement transformation δ to achieve the purpose, in layer l, only the following formula needs to be minimized:

E _z [h′ _l (z-δ)-h _l (z)] ² (11)

Among them, the pre-activation value z=W·a′ _l-1 is within one cycle [(t-1)V _th /T, tV _th /T] (t=1, 2, 3...T) in FIG. 1 are uniformly distributed, that is, the present invention needs to minimize the area they enclose in one cycle, that is:

/2T，使得转换误差最小化。(S5) Set an expected simulation time T, according to formula (12), the bias bias of each layer in the spiking neural network snn increases the offset

/2T, which minimizes the conversion error.

Therefore, the derivation shows that the conversion error is the smallest when the offset V _th /2T (the bold line in Figure 1), and the conversion error is:

Through theoretical derivation, the expression of the minimum conversion error is obtained. The minimum error is proportional to the square of the threshold V _th and inversely proportional to the simulation time T, which is consistent with the law obtained in previous work: the larger the simulation time, the smaller the artificial The neural network activation values are consistent with the result that the transformed spiking neural network performs better.

In order to verify the above-mentioned conversion method of the present application, the applicant has carried out simulation verification, and the specific simulation steps are as follows:

(A1) Convert the input of snn to a picture pixel value or a pulse sequence after the picture is pulsed by methods such as Poisson pulses.

输入通过当前层网络模型计算出一个输出值a′_l，膜电位矩阵M_l增加a′_l，然后判断M_l中有哪些位置超过了阈值电位

超过的部分会减小

存放下来用于下一是时刻，并且发放一个大小为

的脉冲。最后记录输出层输出的脉冲矩阵O_t。(A2) Forward process (formula (1)), at each time t, the network model of the lth layer will receive an impulse input matrix, and the value at each position of it can only be 0 or

The input calculates an output value a' _l through the current layer network model, the membrane potential matrix M _l increases by a' _l , and then determines which positions in M _l exceed the threshold potential

The excess part will be reduced

Store it for the next time, and issue a size of

pulse. Finally, the pulse matrix O _t output by the output layer is recorded.

(A3) Repeat step (A2) T times, accumulate the output pulse matrix O _t at all times, and divide by T, then we can get the output frequency of snn, which is the result of formula (5), which is very close to The output of the source ann (Equation (6)), so the same task as ann can be performed.

(A4) By verifying the classic networks such as VGG16 and ResNet20, the applicant can achieve almost the same performance as the source ann using only about 400 simulation lengths on the datasets Cifar10, Cifar100 and ImageNet, which is far smaller than the currently known Other methods.

Through the above operations, the present invention obtains a spiking neural network whose task performance is very close to the source artificial neural network. By shifting the spiking neural network to a corresponding hardware platform, the corresponding tasks can be completed accurately, quickly and with low energy consumption. In the present invention, the key points are the threshold operation in step (S1) and the offset operation in step (S5). If the threshold operation is not performed, the maximum output value of each layer of the network will vary greatly, resulting in an excessively large threshold of the converted spiking neural network, resulting in the phenomenon that many neural sources cannot be activated; the offset operation can make the maximum conversion error The expectation is reduced by half, which can actually reduce the output gap between the target spiking neural network and the source artificial neural network under the same input conditions.

The above-mentioned embodiments are only the preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any changes made by adopting the design principles of the present invention and non-creative work on this basis shall belong to the protection of the present invention. within the range.

The references mentioned in the embodiment of the present invention are as follows:

1. Barbi M, Chillemi S, Di Garbo A, et al. Stochastic resonance in a sinusoidal forced LIF model with noise threshold [J]. Biosystems, 2003, 71(1-2):23-28.

2. Han B, Srinivasan G, Roy K. RMP-SNN: Residual Membrane Potential Neurons for Deeper Spiking Neural Networks with High Accuracy and Low Latency [C] // Computer Vision and Pattern Recognition at IEEE/CVF Conference Conference Proceedings. 2020:13558-13567.

3. Botev A, Ritter H, Barber D. Practical Deep Learning Gauss-Newton Optimization [C] // Proceedings of the 34th International Conference on Machine Learning Quantities 70.2017:557-565.

4. Sengupta A, Ye Y, Wang R, et al. In-depth understanding of spiking neural network: Vgg and residual architecture [J]. Frontiers in Neuroscience, 2019, 13:95.

Claims (2)

1. An optimal conversion method from an artificial neural network to a spiking neural network, comprising the steps of:

(S1) setting a source artificial neural network ann that meets the requirement, and setting an upper limit to the activation function;

(S2) training the source artificial neural network ann and recording the maximum value of each layer output of the neural network;

(S3) constructing a spiking neural network snn that is identical to the source artificial neural network ann for each layer of the network model, and removing the activation function for each layer;

(S4) copying the network weight and the deviation bias for each layer in the source artificial neural network ann into each layer of the spiking neural network snn, and recording the maximum output value a of each layer of the source artificial neural network ann_lAs thresholds for the corresponding layer of the spiking neural network snn

(S5) setting a desired simulation time T according to the formula

The deviation bias per layer in the impulse neural network snn increases the offset amount

So that the conversion error is minimized.

2. The method for transforming an artificial neural network into an impulse neural network as claimed in claim 1, wherein the requirement in the step (S1) includes three points, the first is that the activation function has an effect of removing a negative value portion; second, the source artificial neural network ann requires the use of an average pooling layer; third, batch normalization layers cannot be used on the source neural network ann.

Images