TW201331855A - High-speed hardware-based back-propagation feedback type artificial neural network with free feedback nodes - Google Patents

Abstract

The present invention discloses a high-speed hardware-based back-propagation feedback type artificial neural network with free feedback nodes, which has a learning mechanism, and its use of cyclic serial architecture is suitable for implementing the parallel computing features of artificial neural networks, calculations of all computing units in a same layer are performed simultaneously by means of single- instruction multiple-bus. Through an architecture design of segmented back-propagation network computing technology, different combinations of network sizes can be obtained under a fixed hardware resource and sufficient memory capacity; it not only can implement three- and four-layer back-propagation artificial neural network, but also can add feedback layers to the feedback-type neural network, and utilize multiplexers to control nodes of the feedback layer so that the network can freely choose the nodes for feedback, the activation function between layers can choose from sigmoid functions, hyperbolic tangent functions and linear functions, so that the network can be configured into different types of feedback and feed-forward type artificial neural network to further broaden the range of hardware applications.

Description

High-speed hardware back-transfer and feedback-like neural network with free feedback nodes

The invention relates to an artificial neural network, in particular to a node for controlling a feedback layer by using a multiplexer, so that the network can freely select a node to be fed back, and the activation function between the levels also has a double bending function, double The curve tangent function and the linear function can also be selected so that the network can construct different types of feedback-type and feed-forward neural networks, so that the hardware application range is more extensive and the high-speed hardware of the free feedback node is inverted. Transfer and feedback type neural networks.

Prior to Technology 1, please refer to Figure 1. D. Hammerstrom proposed the X1 Array architecture, also known as CNAPS (Connected Network of Adapted Processors System). The advantage of CNAPS is that each operation node has built-in addition and multiplier, and can be used by The instruction bus input instruction controls its operation. Each node is equivalent to a simple arithmetic unit (Arithmetic Unit), so the algorithm for computing different types of neural networks is very flexible. Applying this architecture to the hardware development of neural networks, the number of nodes can be easily adjusted because the computing nodes use the same architecture. Unfortunately, the control instructions used in this architecture are so complicated that it is necessary to compile the instructions with the software, and there is no hardware architecture dedicated to the activation function. Because of the universal architecture, there is a speed of operation. Sacrifice.

Prior art 2, the architecture proposed by US005087826A uses a multiplier operation input product and key value ( x‧w ) for each neuron chain to form a two-dimensional array architecture, which is fast but expensive. Hardware costs and a large number of bus bars are not conducive to design.

Although the architecture proposed by US005091864 is not as flexible as CNAPS, its architecture is simpler and easier to design and use, which not only improves the speed of calculation but also reduces the development cost, and also simplifies the complexity of the control unit and shortens the development cycle. The input data is transmitted in a serial connection, that is, the data is first transmitted to the first operation unit, and the second operation unit is passed after two cycles, but because the data arrives at each processing unit at different times, It also adds to the difficulty of designing the control unit. The preferred part of this is the number of reduced activation functions. This architecture considers the characteristics of the one-dimensional array. The data is input and output with the pen pulse cycle. Therefore, the operation unit does not need to perform the calculation of the activation function at the same time, and then the activation function is taken out from the neuron and placed independently. In the backhaul part of the array, only one activation function needs to be designed to complete the operation, and it will not delay the operation. In addition, the architecture is designed with a set of shift registers, which can store the calculated data first and then transfer it back and forth. All the arithmetic units can immediately perform the next data calculation, which saves time. The disadvantage of this architecture is that there is no mechanism for learning parts, and only the retrieving operation can be performed for the network that is trained to complete.

The architecture proposed by US005799134 is similar to the concept of US005091864, but the input data is connected in parallel, that is, all the arithmetic units receive the same input signal at the same time. And by adding a subtractor and a multiplexer to the arithmetic unit, the change of the operation is more flexible. But there is also no mechanism for learning part, only the echo function of the neural network can be performed.

WO 2008/067676 A1 proposes an adjustable artificial neural network system and architecture. The system performs forward and reverse operations of the artificial neural network through segmentation, and the segmentation method can be adjusted according to requirements. However, the hardware of the system must input the actual hardware chip, the number of segments, and the logic component usage limit into the software program, and then the hardware description program generates the required hardware description language code. Then download it to the hardware chip. Therefore, once the wafer is planned, the number of segments cannot be changed.

WO 2010/106116 proposes an analog type artificial neural network circuit composed of a nano device, a CMOS circuit and a column-arranged conductor. The small chip area of the architecture can combine high-density complex circuits with low power consumption. However, the system has no concept of segmentation. Building a neural network of different nodes requires redesigning the architecture of the system.

It can be seen that there are still many shortcomings in the above-mentioned household items, which is not a good designer and needs to be improved.

In view of the shortcomings derived from the above-mentioned conventional technologies, the inventor of the present invention has improved and innovated, and after years of painstaking research, he finally successfully developed and completed this high-speed hardware reverse transfer and feedback type neural network with free feedback nodes. road.

Using the multiplexer to control the nodes of the feedback layer allows the network to freely select the nodes to be fed back. The activation function between the levels also has a double bending function, a hyperbolic tangent function and a linear function, so that the network can be constructed. Different types of feedback and feedforward neural networks are the focus of the present invention.

3.1 Hardware Architecture and Principle

3.1.1 Forward computing hardware architecture

Both the feedback type neural network and the inverse transfer type neural network are multi-layer perceptron architectures, and each layer of neurons has a hierarchy of operations. If the hardware architecture is designed with a hierarchical concept, the network size will be proportional to the hardware cost and logic components consumed. Therefore, the data of the previous layer can be stored in the memory first, and then transmitted through the input bus when needed. The result of the later layer operation is also stored in the memory, and is read from the memory and placed on the hierarchical input busbar in the next layer of operation. This approach can reduce the amount of hardware used.

The invention is based on a ring-and-column architecture. This hardware uses a single-layer network architecture to segment the calculation of the input layer to the hidden layer and the calculation of the hidden layer to the output layer to complete a forward operation. This design can reduce costs. However, the forward operation is composed of many arithmetic units. Under the constraints of the actual hardware logic components, it will affect the size of the network that can be established, which reduces the practicability of this architecture. Therefore, the present invention further calculates the forward operation by using the segmentation calculation method, and the segmentation calculation is performed by the controller, and the number of the operation units is fixed and the capacity of the memory is utilized, so that the system can complete the calculation with a limited number of operation units. Large neural network.

Please refer to Figure 2, which is the forward operation architecture diagram. After the hardware is initialized, the forward operation of the network is started. The input layer data is transferred from the input bus to the operation unit of the next layer, and the weight value in the memory of the operation unit. The multiply-add operation is performed, and the result of the operation is further calculated by the activation function block, and the activation function block outputs the activation function value and its differential value, and is accessed by the register. Then, the shift signal is enabled, and the result of each operation unit operation is transmitted to the next operation unit through the bus bar until the last operation unit outputs the result of each operation unit to the external memory. The value stored in the external memory can be used as the input of the next level, and the above-mentioned action is repeated through the data bus to calculate the output of the next level. This design allows multiple layers of forward operations to be performed on a single layer of network architecture.

When the number of preset arithmetic units is insufficient to establish the size of the network, the present invention adopts a segmentation calculation method to synthesize a larger network. Please refer to Figure 3 for the segmentation calculation flow chart. Assuming that the maximum processing unit of the hardware architecture is two, but the number of neurons that need to be operated is five, the ring string architecture will calculate the first two neurons first. The output value is stored in memory and the remaining neurons are continued.

3.1.2 Reverse Computing Hardware Architecture

Please refer to Figure 4, which is the reverse reverse operation architecture diagram. The hardware architecture mainly calculates the reverse operation flow in the inverse transfer algorithm. The hardware architecture is divided into four blocks, which are respectively calculated for the δ block of the output layer and calculated hidden. The delta block of the layer, the Δw block is calculated, and the hardware block with the updated weight is calculated. When the reverse operation starts, the system first passes the target value and the output value and the differential value obtained through the forward operation to the δ block of the calculation output layer. This block is designed according to the inverse transfer learning method to calculate the δ method of the output layer. The target value is subtracted from the result of the forward operation, and then multiplied by the differential value, the error δ of the output layer can be obtained.

Waiting for the error δ of the output layer to be calculated, the operation of hiding the layer error will continue. In the error expression of the hidden layer, there is a multiplication and addition operation similar to the concept of the forward operation. The difference is that the calculation of the activation function and the bias value of the neuron are not required. The present invention refers to the prior art, reusing the loop-like architecture and parallel processing of the forward operation, and excluding the portion of the activation function calculation in the forward architecture. The δ of the output layer and the weight of the processing unit are multiplied and accumulated by the ring string architecture, and the result is transmitted to the δ block of the hidden layer with the differential value of the hidden layer neuron in the forward operation. This block only needs a multiplier, and the product of the two can be used to obtain the error of the hidden layer. Similarly, the four-layer inverted-transitive neural network has the same error calculation method for the hidden layer, and the same hardware architecture and process can be used to calculate the neuron error between the second and first hidden layers.

After the errors of the output layer and the hidden layer are all calculated, the instruction of the Δw calculation block receiving system management unit starts to calculate the correction amount of the weight value. The input to the Δw calculation block contains the amount of gaps, learning rates, and input bus bars for each layer of delta memory storage. The data transmitted via the input bus includes the bias value of each neuron and the output value of the previous level. The input bus in the Δw calculation block will transfer the offset value to the input bus of the Δw calculation block to calculate the correction amount of the partial weight each time the value in the δ memory is read. Then, the output value of the previous level is transmitted to the Δw calculation block via the input bus to calculate the correction amount of the partial weight. The Δw calculation block first calculates the weight correction amount between the output layer and the hidden layer. The result of the calculation is finally transmitted to the hardware block of the update weight together with the weight value of the previous iteration, and the updated value is obtained by adding the added weights of the hardware block of the update weight to complete the reverse operation.

3.1.3 arithmetic unit

Please refer to Figure 5 for the internal hardware architecture diagram of the unit. The ring string architecture of the present invention is formed by a series of arithmetic units. The function of the arithmetic unit is like a neuron in a neural network. The arithmetic unit obtains the input value and the weight value via the input bus and the weight bus, calculates the multiplicative accumulated value of the two, and performs nonlinear conversion through the activation function block. The internal structure of the arithmetic unit is mainly four blocks of memory, multiply accumulator, shift register and activation function. These four blocks operate independently and use pipeline design to speed up the computation of the overall computing unit.

The queue memory inside the operation unit is used to store the weight value required for the weight calculation. The weighting order of the forward and reverse operations of the network is different, so it is necessary to store the weight indicators in the forward and reverse operations by different memories. In order to read the weight values in the arithmetic unit in the correct order, at the time of system initialization, the memory address of the weight value needs to be arranged in the weight memory address management unit for the forward operation, the reverse operation, and the update weight. Each unit has its own addressing number, which is used to store the data of the weight bus in the correct unit. That is, the memory of the arithmetic unit needs to be matched with the signal on the address bus when writing the signal. When the arithmetic unit reads the weight value in the memory, the weight value is passed to the internal multiply accumulator. Since all the arithmetic units receive input data from the input bus at the same time for multiplication and accumulation calculation, there is no need to perform address analysis. The multiply accumulator is responsible for the multiplication and addition of the input data and the weight value. In the forward operation, the input message needs to be converted by the activation function. The activation function hardware block in the arithmetic unit is divided into a double bending function and its differential block, and a double ˊ curve tangent function and its differential block. The result of the multiply accumulator operation is passed to the activation function block for conversion.

Each arithmetic unit has a shift control signal inside, and this signal controls whether the result calculated by the current operation unit is output to the next operation unit. After the data is calculated by the arithmetic unit, the results calculated by each arithmetic unit are transferred to the next arithmetic unit by means of shifting. The last arithmetic unit passes the result to the memory storage via the bus.

3.1.4 Numerical system

Referring to FIG. 6 , for the fixed-point decimal representation, the data calculation and processing of the present invention are implemented by using binary signals. For the numerical operation, the prior art proposes an error comparison between an integer with a finite precision and a floating point number, and the difference between the two training errors is not large. While the prior art proposes to use integer operations, relative floating point operations can reduce the computation time and do not require too many valid bit numbers. Based on the above theoretical considerations and hardware resources such as logic gates and bus bars, the present invention uses a 32-bit binary code fixed-point numbered decimal number as the numerical encoding method of the architecture. However, the Nios II microprocessor decimal type uses the floating point fraction of the IEEE 754 format. Therefore, when the hardware receives the instructions and network parameters issued by the Nios II, it must first be converted by the floating point to fixed point converter. The integer number of digits and the number of decimal places of the present invention can be changed in accordance with the user.

3.1.5 Stacking, Queue, and Random Location Access Memory

The system needs memory access data during the operation to record the input and output values required by each layer of the network. The invention uses three different hardware architectures, such as stack, queue (FIFO) and random access memory, to store data. Stacking and stacking do not need to know the location of the memory when storing, which can reduce the design burden, but it is less flexible in use. The data is stored in order and can only be read in a specific order. The random access memory needs to be matched with the memory location for storage or reading, and the address of the read memory can be read. Table 1 describes the memory functions provided by the present invention. The stacking and array hardware architecture signal functions are described as follows:

Clear: In the stack architecture, the write position returns to the lowest position, and the read position returns to the highest position. In the queue structure, both the write and read positions return to the lowest position.

Hold: Indicates the current memory location in the stack and queue architecture.

Restart: This signal needs to be matched with the hold signal. In the stack and queue structure, the read position is returned to the position set by the previous hold.

Rd_addr: the memory location of the read data

3.1.6 Activation Function Hardware Architecture

The activation function is implemented in a hardware architecture. The common methods in the literature are look-up table method, table-cutting interpolation method and fragment linear method. The look-up table method is one of the most commonly used methods to implement the activation function hardware architecture. In the prior art, the table look-up method only needs three layers of hardware operations to obtain the activation function data. The advantages of the look-up table method are simple design and fast calculation speed. The disadvantage is that the hardware cost of the cost is proportional to the accuracy. When the conversion of the activation function requires more accurate data, the hardware needs more memory storage. Adding more information will also take more time to execute. The table cut line interpolation method is a straight line cutting curve, and the length of the cut varies with the turning point. The best turning point is called when the Euclidean distance between the turning point and the previous turning point is less than the specified error. For a schematic diagram, please refer to FIG. 7 , which is a schematic diagram of a conventional interpolation method and a line interpolation method. The segment linear method [48] uses the Center Linear Approximation (CRI) algorithm to calculate the approximation of the double bending function curve. The algorithm is shown in Table 2. The Δ of Table 2 represents the interpolation depth (Interpolation Depth), x represents the input value, and q represents the number of iterations. The prior art indicates that there is an optimum solution when q is 2 and Δ is set to 0.28094. This algorithm can only calculate the part of the double-bend function x≦0, and the other half can be realized by the property of the function symmetry. This method can reduce the usage of logic elements. Its hardware architecture diagram is shown in Figure 8. The principle of the fragment linear method is to find the turning point first, then the error will gradually increase. When the error reaches a limit, it will find a new turning point. Figure 9 is a schematic diagram of the segment linear method. It can be seen from the figure that the segment linear method uses the interpolation method to find the value of the double bending function.

Table 2 Fragment linear algorithm

3.1.7 Double Bend Function Hardware Architecture

The present invention calculates activation function values using prior art activation function hardware architectures. The prior art modifies the architecture of the fragment linear method, utilizing the hardware architecture reusable features to reduce the resources required by the hardware architecture. Taking the table 2 segment linear algorithm as an example, the initial value of g(x) is 0 in the first calculation of i =0. Equation (3.2) requires an adder and two shift registers for hardware implementation, and three adders and three shift registers for equation (3.4). Change equation (3.1) to equation (3.2), propose a quarter to equation (3.3) and then do the calculation. This approach can achieve the same result, but in terms of the use of logic components, the original equation (3.1) saves two shift registers. And because the initial value of g(x) is equal to zero. Therefore, in equation (3.3), the design with 0 added can be omitted directly, and the equation (3.2) is substituted into equation (3.4). Compared with the original design, you can save an additional adder and speed up the clock required for the operation.

In the algorithm of Table 3.2, the larger the value of the iteration number q, the closer the result is to the value of the double bending function, but the longer the operation time. The result of the experiment is the best number of times for the overall benefit of the network. Therefore, in the case where the number of iterations is fixed at 2, the equation (3.5) of Table 3.2 is changed to a register storing a constant, reducing the hardware architecture required by equation (3.5). Instead, equation (3.3) is changed to equation (3.6) in the second iteration, and the original Δ shifting of 2 bits is omitted. In the first and second iterations, the constant register is changed by Δ ₁ = 0.979721, Δ ₂ = 0.197876, and the equation (3.4) and equation (3.6) are added. The result of this hardware architecture design is shown in Figure 10.

3.1.8 Hyperbolic Tangent Function Hardware Architecture

The hyperbolic tangent function is similar to the graph of the double bending function, except that the range of the two ranges is different. The range of the double bending function is between 0 and 1, and the hyperbolic tangent function is between -1 and 1. The segment linear method is interpolated into a pattern in which a plurality of line segments are interlaced. The delta size and the slope of the equation in the algorithm affect the curve function graph synthesized by the segment linear method.

Adjust the value of the constant register in the previous section of the hardware architecture and the hyperbolic tangent function of the shift register. The resulting hardware architecture is shown in Figure 11. The delta values for the two iterations are determined by the software test, Δ ₁ =0.476807 and Δ ₂ = 0.188034, respectively.

3.1.9 Symmetric properties of the activation function

Both the hyperbolic tangent function and the double bending function realized by the segment linear method have symmetrical properties. See Figure 12 and Figure 13. The other half of the function can be obtained by using the judgment of the subtractor and the multiplexer according to equations (3.7) and (3.8). Such an approach can reduce the usage of nearly half of the logic components. The hardware architecture in the first two sections is shown in Figure 14 and Figure 15.

3.2 Management Unit

The management unit controls the system flow and the ordering of important indicators in the management system. The system architecture of the present invention comprises five different functional management units, which are mainly for controlling the ordering of the weight memory addresses, managing the operation unit address, arranging the access order of the target values, and outputting the neuron nodes of the decision feedback layer. And control of the overall system flow. The management unit performs the behavior that the management unit should operate on the program in accordance with the current state in the design mode of the finite state machine, and enters the next specified state. In addition to controlling the flow of the overall system, the system management unit also handles the mechanism of the segmentation operation and the timing control of the system.

3.2.1 Weight Memory Address Management Unit

In a complete training of the inverse transfer error learning method, the forward and reverse angles are observed, and the arrangement of the weight values can be divided into three types. The first is the order in which the weight values are arranged in the forward operation, the second is the order in which the weight values are arranged in the reverse operation, and the third is the order in which the weight values are updated. In order to facilitate the system to find the memory address of the weight, the weight memory address management unit is designed for each of the three sequences, and the address of the weight memory is first stored in the random access memory. In the body. When the system is executing one of the states, the instruction of the central management unit requests the weight memory address management unit to sequentially output the address of the weight memory, and the system finds the required weight value according to the address in the specific memory. .

The hardware block diagram of the weight memory address management unit is shown in FIG. 16. The three random access memories are individually responsible for accessing an array of weight memory addresses. The management unit receives the instruction of the system management unit. When the intialize pin is enabled, the LearningAddr_FIFO and the UpdatingAddr_FIFO block will start to perform the weight memory address arrangement of the above three cases according to the input parameters. The following is an important signal description of the weight memory address management unit:

1. Initialize: This is the enable pin of the weight memory address management unit. When the network parameters required by the hardware are set, the block will be required to perform the arrangement and storage of the weight memory address.

2. clear: When this signal is enabled, the memory read and write addresses stored in the first order are returned to the lowest position.

3. LA_FIFO_clear: This signal is responsible for storing the second order of memory read and write addresses back to the lowest position.

4. UA_FIFO_clear: When this signal is enabled, the third-order memory read/write address is returned to the lowest position.

5. Done: When the weight memory address of the weight management unit is arranged, this signal will generate a high potential, indicating that the work of this block is completed.

6. kind: record the current network type signal, when the signal is "000", it is a three-layer reverse transmission type neural network. When the signal is "001", it means a four-layer inverted transmission type neural network, "010" The three-layer feedback type back-transfer-like neural network, "011" is an Elman-like neural network, and the signal "110" represents a three-layer feedback-type inverted-transitive neural network with a hidden layer added to the feedback layer, signal "111" It represents the Elman-like neural network that adds the feedback layer to the hidden layer. Based on this signal, the weight management unit determines whether it is necessary to make a feedback layer weighting arrangement.

7. Rd_opcode: This signal controls which weights are output by the weight management unit. When the signal is "00", the order of the weight value output from the memory is the first type, when it is "01", it is the second reverse weight order, and the "10" is the time when the network updates the weight. Three orders.

The first order of the weight memory addresses is stored sequentially using the queue memory. When the system starts executing, the system management unit will first execute the instruction to request the initialization of each block. Then the weight control block will start to operate, and the address of the first, second and third cases will be sorted into the queue memory. .

The ordering of the second weight memory address is for the case of reverse operation in the inverse transfer learning algorithm. The calculation of the hidden layer error by the inverse operation has the same weight as the forward operation. Therefore, the calculation of this part of the equation can be performed by using the structure of the ring string by means of hardware co-construction. The inverse operation is calculated from the output layer to the direction of the hidden layer. Therefore, the arrangement of the weighted addresses is arranged from the output layer to the hidden layer. The design of the hardware architecture is mainly considered that the partial weight values are not arranged, and the weights of the hidden layer to the input layer need not be arranged. The operation unit of the ring string may read the weight value required for the next neuron operation when reading the memory. To prevent this from happening, add the same number of weight values to the memory location on the same layer. FIG. 18 is a reverse operation arrangement state diagram of the weight memory address management unit, and Table 3 is a description of FIG.

The third weighting address arrangement is used when the neural network is updating the weight value. This part of the arrangement is used to calculate the weight correction amount in the reverse operation, which is convenient for the system to find the weight that needs to be updated in the memory. Because the weight of the inverse transfer network algorithm is modified from the output layer at the end of the network architecture to the hidden layer layer by layer. Therefore, the ordering of the weighted addresses is stored in the memory in reverse order.

Take the modified Elman neural network as an example, the number of input layer neurons is 1, the number of feedback neurons in the input layer is 3, the number of hidden layer neurons is 3, and the number of feedback neurons in the hidden layer is 2 The number of neurons in the output layer is two, and the maximum processing unit of the hardware is preset, as shown in Figure 17. The weight memory address map is shown in Figure 19. Before the ring serial architecture begins to operate, the weight value is first stored in the memory of the arithmetic unit, as shown in Figure 20. When the input value is transmitted to the ring string architecture through the input data bus, the operation unit array sequentially reads out the weighted values stored in the operation units 1 and 2 and multiply and accumulate the input values. For example, when calculating the output of the hidden layer, the arithmetic unit 1 sequentially reads W04, W14, W24, W34, and W44, and the arithmetic unit 2 simultaneously outputs W05, W15, W25, W35, and W55 and operates with the input value in the multiply accumulator. The result of the operation is stored in the hidden layer memory. Then the system will judge whether the neurons in the same layer have been calculated. If not, the segmentation will be processed. The weight values W06, W16, W26, W36, W66 and X, X, X, X, and X stored in the arithmetic unit 1 and the arithmetic unit 2 are continuously read. X is a preset value used by the weight management unit to fill the number of weights. When the number of preset arithmetic units is not a factor of the total number of network neurons in the same layer, an error occurs in which the arithmetic unit reads the weight, and the weight value required for the next neuron is read early. Therefore, in the arithmetic unit 2, it is necessary to arrange the same weight number as the arithmetic unit 1 in the weight memory.

3.2.2 Address Management Unit

The address management unit is a controller that is responsible for managing the address of the arithmetic unit. The number of the arithmetic unit address is controlled according to the type of the network and the maximum number of arithmetic units preset by the hardware. The ring serial architecture is a parallel arrangement of arithmetic units, each of which has a separate address. When the weight is importantly written to the memory in an operation unit, it is written into the correct operation unit according to the address on the address bus.

Due to different network types, number of network neurons, and maximum processing unit for hardware presets. The weight value must be stored in the specified unit before the operation is performed using the ring string architecture. Taking FIG. 21 as an example, if weights W04, W14, W24, W34, and W44 are to be written into the memory of the first arithmetic unit 1. When the weights W04, W14, W24, W34, and W44 are written to the weight bus, the signal on the address bus must be 1, and the weights W04, W14, W24, W34, and W44 are written into the arithmetic unit 1. Then, the weights W05, W15, W25, W35, and W55 are written into the arithmetic unit 2 through the weight bus, and the address bus needs to be adjusted to 2, as shown in FIG.

3.2.3 Target value management unit

The inverse transfer learning algorithm adopted by the present invention belongs to supervised learning, and the purpose of learning is to reduce the gap between the network output value and the target value. In order to reduce the gap, it is necessary to adjust the weight value between the neurons through the reverse operation. Since the reverse operation is to calculate the output layer error from the last neuron in the output layer, the order of storage should be considered in order to store the target value of the memory. Otherwise, when calculating the error of the last neuron in the output layer, the first target value output from the target value memory will not be the target value required for this calculation. When the number of target values is the same as the number of neurons in the output layer, the above problem can be solved by using the stacked memory to solve the last target value in an advanced sorting manner, but it is the target value required to calculate the first output layer error. problem. However, when the number of target values is different from the number of neurons in the output layer, that is, when there is more than one set of training samples, the method of using stacked memory will not be applicable. The last set of target value data will become the first set of target values used to calculate the output layer error. Considering the above two sorting problems, the present invention designs a controller for managing the memory address of the target value storage, so that the target value can be stored in the memory only by following the first group of the first neuron, first The sequential input of the second neuron of the group. The controller performs the storage of the target value memory address suitable for the reverse operation according to the number of nodes of the output layer and the total number of target values. The target value stored at the initialization is read to the target value stack memory storage according to the sorted target value memory address. Completing the stored target value The internal memory target value of the stacked memory can be correctly calculated with the output layer error of the reverse operation.

If the network output layer has two neurons, the problem has four sets of target value data. The target value and the memory address stored in the target value queue memory are shown in Fig. 23. The inverse operation of the inverse transfer learning algorithm first calculates the output layer error starting from t1 of the memory address 1. However, the queue memory will first read t0 from the memory address 0, and then read t1. After processing by the target value management unit, the target value memory address is rearranged and written into the stacked memory as shown in FIG. When the stacked memory of FIG. 24 is read, the target value of the last neuron t1 of the first group is output first, and the target value of the first neuron t0 is output to the inverse hardware block. When the inverse hardware block calculates the second set of output layer errors, the stacked memory then reads the target value of the last neuron t3 of the second group and then reads the target value t2 of the first neuron. Table 4 is the target value algorithm. The algorithm first defines the largest group number. Then the group written to the memory address will start from the largest number, and calculate the smallest address of the memory in the group. The address will be Write one by one to the target value stack memory. Write a set of memory addresses, and continue to program the next set of memory addresses and write them to the target stack memory until all target value memory locations have been written.

3.2.4 Feedback Node Control Unit

The biggest difference between the network architecture of the feedback type neural network and the feedforward neural network is that the feedback type neural network has a feedback processing layer. The input signal of the feedback processing layer comes from the feedback of the hidden layer or the output layer neurons, and provides the function of local memory of the network. The effect of the time delay (Time-Delay) caused by the feedback action shows that the feedback-like neural network is suitable for dealing with dynamic problems related to time. The feedback-type neural network designed by the invention not only provides a self-hiding layer feedback feedback type neural network, but also adds a feedback type neural network from the output layer to the hidden layer. The neuron node of the feedback layer is controlled by the feedback node control unit to control whether the neuron output of the feedback layer is transmitted to the neurons of the next layer. When the neuron output of the feedback layer is not transmitted to the neurons of the next layer, it means that the neurons connected to the neurons of the feedback layer do not perform feedback. The feedback control unit receives the neuron signals that need to be fed back from the specific pin of the hardware, and each signal is combined into a set of hidden layer or output layer neurons of the feedback feedback type neural network via the shift accumulator in the feedback control unit. The vector signal that is fed back.

3.2.5 System Management Unit

The system management unit controls the flow of the entire neural network. The data between each block must be matched with the timing control to correctly calculate the result of the algorithm. The controller uses the design of the finite state machine to decode the current state of the system, and the command begins with the circuit block associated with the next state, as shown in Figure 25. In addition to controlling the timing between each block, the system management unit also controls the architecture of the entire ring string, segmentation calculation, accumulator, multiplexer signal selection line, and control of other management units.

The main flow and status of the controller are shown in Figure 26. The flow chart is divided into three major blocks: initialization, forward operation and reverse operation. When the system starts running, first enter the initialization process, zeroing the scratchpad and memory parameters in the hardware, to avoid the operation error caused by the wrong data. The system then receives network parameters, training data, and target data and writes them to the scratchpad and memory. When the data is stored, the management unit generates the start signal request weight, the address, the feedback node, and the target value controller to calculate according to the network parameters. After waiting for the completion signal of the weight, address, feedback node and target value controller, the system writes the data needed for the forward operation and the reverse operation to the memory specified by the system according to the result of each management unit. in. Complete the initialization work.

When the system completes the initialization action, it begins the part of the forward operation. The system controller adjusts the signal selection line of the multiplexer according to the type of the first hidden layer activation function. The training data is then transferred from the input queue memory to the ring string architecture for weighting operations. In the case of a feedback neural network, after the training data is written, the system controller determines whether the data of the feedback layer is transmitted to the ring structure according to the output signal of the feedback node controller. After the value of the first hidden layer neuron is calculated and stored, the controller determines whether the hardware needs to perform segmentation calculation according to the network size until the operation of the first layer hidden layer is completed. Adjust the signal selection line of the multiplexer according to the type of network. The operation of the hidden layer and the output layer is performed by using the value of the previous layer calculated by the activation function block in the ring string and storing it in the memory, and then returning it to the ring string architecture to complete the entire forward operation. .

In the learning mode, after the system performs the forward operation, the reverse operation starts. The controller instructs the associated memory output target value, and the differential activation function and the result of the forward operation are passed to the inverse operation block to calculate the error of the output layer. The controller calculates the error of the output layer and the weight of the weight management unit through the ring serial architecture, and then inputs the hidden layer error block to obtain the hidden layer error. After the hidden layer error is calculated, the controller compares the error, the learning rate, and the input value of the previous level of each neuron through the weight management unit with the clock required for the Δw block operation, and synchronizes the design in the pipeline in the reverse operation area. The block calculates the weight correction amount and updates the weight and stores the new weight. Finally, it is judged whether the training materials are all trained, and the set number of trainings is reached, and the operation of the entire algorithm is completed.

3.3 Software Planning

The system transmits training data and commands to the Avalon bus via the Nios II microprocessor, and then provides design of hardware network parameters and operations through the design of the Avalon bus register, such as network type and activation function selection. , training materials, network size, weight data, target value, learning rate, etc. The advantage of this design is that when you want to modify the type and size of the network, you can set it through the software without re-setting and compiling the hardware. Figure 27 is a block diagram of the system used in the present invention and the Nios II. Since the hardware operation uses fixed-point decimals, the decimal type used by the Nios II software is a floating-point fraction in IEEE754 format. In order to reduce the load on the NIOS II processor and increase the speed of the overall system, the converters for floating-point conversion and fixed-point conversion floating-point are placed on the hardware. When the software transmits the training data to the hardware, it will first convert to a fixed-point decimal format for the hardware operation through the converter of the floating-point conversion fixed point. After the hardware operation is completed, the new weight is converted to a floating point decimal by the converter of the fixed point conversion floating point to the memory storage waiting software to read.

Figure 28 shows the software flow chart. In addition to the hardware design, the system must be operated by software. When the system is first started, the hardware resets the instructions, then defines the network type, network test or training, weight data, target data, input data, learning rate, training loop times, and activation function types and networks. The architecture, when the network settings and data are passed to the hardware, the command to start training is released, and the hardware will start computing until the signal is completed by the Avalon interface. Upon receiving the completed signal, the software can execute the instruction to read the result. Table 5 is the register function on the Avalon bus that is responsible for communicating the controlled end hardware and the host software.

The hardware design of the present invention verifies the function of each controller through ModelSim, and whether the hardware-like neural network runs the reverse-transfer network algorithm correctly. In order to simulate different hardware-like networks of hardware-transformable parameters, two different test codes were designed, and a 1*3*4 inverted transmission network was synthesized, and a 1*4*1 feedback-like neural network was synthesized. The network, the two networks are shown in Figure 29 and 30. The number in the neuron in the figure represents the number of the neuron, and wi j represents the weight between the i-th neuron and the j-th neuron. Bk represents the bias value of the kth neuron.

4.2.1 Simulation of three-layer inverted transfer neural network

When designing the hardware, each block must be able to receive and output the correct value at the correct clock. This section, along with the flow of Figure 3.25, uses the waveforms of the ModelSim simulation to illustrate each important process, memory and The situation where the signal line changes. Table 6 is the initial value of the network parameters. During initialization, the network parameters are passed to the weight, address, and target value management units to perform the operations responsible for each unit, and the initialization action is completed. Then, in the forward operation flow, the weight value is written into the memory of the operation unit according to the address of the operation unit, and then the training data stored in the input memory is read. After the operation unit performs the multiplication and accumulation operation, the activation function value and the differential value are output through the calculation of the activation function block. The address of the 22-weighted arithmetic unit is shown in Figure 31, 32, and 33. The activation function block outputs both the activation function value and the activation function differential value. The lettersig, tansig, and linear pins in Figure 34 are the output signal lines for the double-bend function, the hyperbolic tangent function, and the integration function, respectively. And d_logsig, d_tansig, and d_linear are the differential values of the function. In the training mode, the forward operation is followed by the reverse operation. The inverse operation first calculates the error of the output layer. Figure 35 The system management unit reads the target value stack memory, the output layer neuron output, and the activation function differential value to the output layer error block calculation error in the reverse operation block. The output layer error is then passed to the array of arithmetic units, and the resulting results are compared to the hidden layer output differential values to calculate the hidden layer error. After the error calculation of the output layer and the hidden layer is completed, the error calculation process will continue to calculate the correction weight and the revised new weight in the correction weight block and the update weight block in the reverse operation block. Read the memory stored in the new weight, and observe the values in the pin new_weight_stack_output as shown in Figure 36, 37, 38, and 39.

4.2.2 Simulation of Elman Neural Network

The simulated Elman-like neural network of the present invention is different from the general Elman-like neural network except that the free feedback node function of the present invention is applied, and the feedback layer of the output layer is additionally increased. The experimental parameters are shown in Table 7. In the initialization part, the signal output by the feedback node control unit in the waveform diagram of the feedback type neural network controls whether the multiplexer of the feedback layer node is the data of the output feedback layer node or 0, and the present invention controls the hidden layer first and third. The neurons are not fed back, and the vector signals output by the feedback node management unit to the system management unit are stored by the judge1_signal and the judge2_signal register respectively, 0000001010 and 0000000001, as shown in FIG. When the arithmetic unit array sequentially inputs the data of the feedback layer, the system management unit synchronizes a bit to read the signal for judging whether the neuron is fed back. If it is 1, the feedback data is transmitted to the arithmetic unit array for calculation; If it is 0 then no. This input mode affects the output signal of the multiply accumulator of the arithmetic unit. Under continuous input, the value of the arithmetic unit integrated function register will change once every clock, but if the input is 0, it will be maintained. The value of the previous clock, while the hidden layer of the present invention only feeds back the output values of the second and fourth neurons, and the value of the integrated function register maintains the same value of two clocks in each of the four clocks, such as Figure 41 shows. Calculate the activation function of the first two neurons in the hidden layer, because the maximum unit of operation preset by the hardware is two. Therefore, the calculation of the latter two neurons is performed in a piecewise calculation manner, and the activation function of the last two neurons in the hidden layer obtained by re-entering the data of the training sample and the feedback layer is shown in Fig. 42 and Fig. 43. . The forward operation is completed, and then the flow of the reverse operation is entered to calculate the error of the output layer and the hidden layer. The feedback-type neural network includes the Elman-like neural network, and the process of calculating the error is the same as that of the feedforward neural network. In calculating the correction weight, Elman differs from the feedback type neural network in that the weight of the connection with the feedback layer is a fixed value and does not change due to training. Therefore, the system management unit will first determine whether it is an Elman-like neural network before calculating the correction weight.

4.3 Experimental results

4.3.1 Curve fitting

The invention implements a curve fitting (Curve Fitting) on a sinusoidal function by using a three-layer inverted transmission network architecture with different numbers of hardware computing units, and the fitting equation is (4.2), and the present invention analyzes the execution of the hardware by this experiment. efficacy. Training samples from between 0 and 1 randomly sampled pen 50, as shown in FIG red circle that 44 corresponding to x. The training data input is self-variable x , y is the number of strains and the target value of the experiment. The experimental parameters are set as shown in Table 8. The invention adopts a 1*4*1 network architecture to train the relationship between the input and output of the sine function of the network mapping. The initial weight of the experiment is between random numbers [-0.5, 0.5], and the training data x ranges from 0 to 3.14. The experimental results are shown in Figures 44 and 45. The red circle in the figure represents the output value obtained by inputting the training data to the neural network after training, the green symbol represents the value of the test data in the neural network, and the blue line is the target value. In the experiment, the influence of the number of arithmetic units of hardware synthesis on the operation speed is compared under the same initial weight. The experimental results are shown in Table 9. Hardware-based neural networks with fewer hardware units need to segment the output values of neurons more times. It takes more time to calculate. The hardware network with three hardware units in the experiment is more time-consuming in training time than the hardware network with two hardware units. The reason is the weighting problem of the segmentation algorithm, so that the hardware network with three arithmetic units actually synthesizes a 1*6*1 hardware network to calculate the 1*4*1 class neural network. The controller needs to do more processing for the extra arithmetic unit, so that the hardware network with three hardware units will be the same as the hardware network with two hardware units under the same number of processing steps. Training time is long. With the same initial weight and number of trainings, the hardware network will receive the same update weight after training, and will not have different results due to the number of hardware units. Therefore, the error obtained by the experimental results is the same.

y ( x )=sin( x ) (4.2)

4.3.2 Voltage prediction when the battery is discharged

In order to verify that the system proposed by the present invention can implement a feedback type neural network, the present invention performs a voltage prediction experiment by using battery discharge data provided by Baisuo Company. This battery data records the discharge data of a group of batteries every 10 seconds. Battery power is maintained near -29.1 watts. In this paper, the voltage, current and battery temperature in the battery data are used as the neural network training data, and the normalized conversion is performed according to equations (4.3) to (4.5) before inputting the network so that the input value is [0.2, 0.8). In the case, as shown in Figures 47 to 49, the graph on the left is the data graph of the original data, and the graph on the right is the data graph after normalization. Table 10 describes the network parameters of the experiment. The initial weight is a random number randomly selected between [-0.5, 0.5]. The present invention uses the online learning method to input a set of data and start training the network until the set conditions. The training stop condition set by the present invention is training 1000 times. After the training is completed, the network inputs a second group of data to predict the voltage value of the third group. The hardware network then uses the second set of data training networks to input the third set of data to predict the voltage value of the fourth group. The experiment is not completed until the training samples are trained. Figure 50 is a normalized discharge curve for hardware network prediction. The predicted voltage value in the figure is compared with the actual voltage map. It can be observed that the predicted voltage value is slightly larger than the actual voltage value. Figure 51 is the error map of the two discharge curves. The more the prediction of the voltage is, the later the error is. small. Table 11 is the result data of the experiment. The training time of each group of training materials takes an average of 0.0047 seconds and then predicts the voltage value after 10 seconds.

ν=1.7 e -4*(ν-8856)+0.2, (4.3)

i =7.41656 e -4*( i +3200)+0.2, (4.4)

t = 4.1958 e -2*( t -26.25) + 0.2. (4.5)

4.3.3 Pansy classification experiment

The invention uses the four kinds of data represented by real numbers of the butterfly, the length of the flower bud, the width of the flower bud, the length of the flower petal and the width of the flower petal to train the hardware neural network proposed in the paper, and the learning goal is to distinguish the three subspecies of the butterfly flower (Iris). Setosa, Iris Versicolor, and Iris Virginica), the target data of the experiment, represented by {0,0,1}, {0,1,0}, {1,0,0} represent three categories. Through the NIOS II instructions, the hardware is synthesized into a 4*4*5*3 four-layer inverted transfer neural network. The experimental parameters are shown in Table 12. There are a total of 75 groups of information on the pansies. 35 sets of data are used as training materials, and 40 sets of data are used as test data. The hardware network adjusts weights and bias values via successive training and reverse transfer learning algorithms. Table 13 is an analysis table of experimental results. In the recall mode, the recognition rate of the training data is 100%, the identification rate of Iris Setosa and Iris Versicolor of the test data is also 100%, but the recognition rate of Iris Virginica is only 92.86%. The main reason is that Iris Virginica is very similar to some of Iris Versicolor's data, so that the system misidentified Iris Virginica as Iris Versicolor.

The detailed description of the preferred embodiments of the present invention is intended to be limited to the scope of the invention, and is not intended to limit the scope of the invention. The patent scope of this case.

In summary, this case is not only innovative in terms of space type, but also can enhance the above-mentioned multiple functions compared with the customary items. It should fully meet the statutory invention patent requirements of novelty and progressiveness, and apply for it according to law. This invention patent application, in order to invent invention, to the sense of virtue.

Figure 1 Schematic diagram of prior art

Figure 2 Forward operation architecture diagram

Figure 3 Segmentation calculation flow chart

Figure 4 Reverse reverse operation architecture diagram

Figure 5 Internal hardware diagram of the arithmetic unit

Figure 6 Fixed-point decimal representation

Figure 7. Schematic diagram of traditional interpolation and intercept interpolation

Figure 8 Fragment linear method hardware architecture diagram

Figure 9 Schematic diagram of the fragment linear method

Figure 10: Double-bend function hardware architecture diagram of improved segment linear method

Figure 11 Hyperbolic tangent function hardware architecture diagram

Figure 12 Symmetrical nature of the double bending function

Figure 13 Symmetrical nature of the hyperbolic tangent function

Figure 14 Double-bending function symmetrical nature hardware architecture diagram

Figure 15 Hyperbolic tangent function symmetry property hardware architecture diagram

Figure 16 Weight memory address management unit hardware block diagram

Figure 17 Improved Elman-like neural network 1-3-2 architecture

Figure 18: Reverse operation weight memory address management unit state diagram

Figure 19 Weight memory data configuration diagram

Figure XX Operation unit memory stored value map

Figure 21: Operation unit weight value configuration diagram (1)

Figure 22: Operation unit weight value configuration diagram (2)

Figure 23 Target value queue memory configuration diagram

Figure 24 Quad Value Stack Memory Configuration Diagram

Figure 25: Control unit finite state machine diagram

Figure 26 System Management Unit Flowchart

Figure 27 Hardware and Nios II architecture diagram

Figure 28 Software planning flow chart

Figure 29: Network size 1*3*4 three-layer inverted transmission neural network diagram

Figure 30 Network size 1*4*1 feedback type neural network diagram

Figure 31. Waveform of the weight writing operation unit (1)

Figure 32. Waveform of the weight writing operation unit (2)

Figure 33. Waveform of the weight writing operation unit (3)

Figure 34 Results of the output of the activation function block

Figure 35. Signal Source Diagram for Calculating Output Layer Error

Figure 36: New weight value waveform (1)

Figure 37. New weight value waveform (2)

Figure 38: New weight value waveform (3)

Figure 39: New weight value waveform diagram (4)

Figure 40: Signals that determine whether the hidden and output layer neurons are fed back.

Figure 41 Schematic diagram of the integrated function of the output of the arithmetic unit

Figure 42. Results of the output of the activation function block (1)

Figure 43. Results of the output of the activation function block (2)

Figure 44: Sinusoidal function actual curve and hardware fitting curve comparison chart

Figure 45. Error analysis of network output value and target value of training data

Figure 46 Error analysis of network output value and target value of test data

Figure 47: Battery voltage diagram and voltage data after normalization

Figure 48 Battery current diagram and normalized current data

Figure 49. Battery temperature map and normalized temperature data

Figure 50 Comparison of battery discharge curves predicted by feedback-like neural networks

Figure 51: Feedback-type neural network predicts battery discharge curve error map

Claims (11)

A high-speed hardware reverse transfer and feedback type neural network with a free feedback node, wherein the network operation mode includes: a four-layer reverse transfer type neural network mode; and a free choice feedback node neural network mode. For example, in the operation mode described in claim 1, the content of the software is determined by the content of the software update register. A high-speed hardware reverse transfer and feedback type neural network with a free feedback node, the hardware structure of which includes: an input device, which is a device or a microprocessor capable of generating a digital signal; a programmable hard The programmable hardware is interfaced with the input device, receives data of the input device and performs arithmetic processing, the programmable hardware is an element programmable logic gate array; and an output device, the output The device is interfaced with the programmable hardware and receives data calculated by the programmable hardware. The output device is a device or a microprocessor that can receive digital signals. For example, the operation mode described in claim 1 wherein the freely-selective feedback node neural network mode comprises: a connection input layer; a first feedback processing layer; a multiplexer; a hidden layer; and an output The first feedback processing layer receives the hidden layer output of the previous iteration when inputting the data, and then transmits the received input data to the hidden layer of the iteration; the weight value between the feedback processing layer and the hidden layer is fixed. The multiplexer controls the nodes in the inner layer of the feedback, so that the network can freely select the node to be fed back, or choose not to give feedback, and the node that does not give feedback has a link weight of 0. The operating mode of claim 1, wherein the freely-selective feedback node neural network mode comprises: a second feedback processing layer; a multiplexer; a hidden layer; and an output layer; The feedback processing layer receives the output layer output of the previous iteration and inputs the data, and then transmits the received input data to the output layer of the iteration; the weight value between the feedback processing layer and the output layer is fixed; the multiplexer control The nodes in the inner layer of the feedback system enable the network to freely select the nodes to be fed back, or choose not to give feedback. The nodes that do not give feedback have a link weight of 0; the input data of the hidden layer comes from the first feedback processing layer. After connecting with the input layer, the multiplication of the weight values is performed, and the result is output to the output layer through the activation function of the hidden layer; the output layer receives the output value of the hidden layer and the second feedback processing layer, and the multiplication of the weight values is added. The output of the output layer is obtained through the activation function of the output layer. The operation mode described in claim 1, wherein the four-layer reverse transfer type neural network mode includes: an input layer; a first hidden layer; a second hidden layer; and an output layer; Receiving the external signal, inputting the data, and then transmitting the received input data to the first hidden layer; the input data of the first hidden layer is multiplied by the weight value, and then outputting the result through the activation function of the first hidden layer Giving the second hidden layer; the second hidden layer receives the multiplication of the output value of the first hidden layer by the weight value, and outputs the result to the output layer through the activation function of the second hidden layer; After the multiplication of the weight values of the layers, the output of the output layer is obtained through the activation function of the output layer. As described in claim 4 or 5, in any of the freely-selective feedback node neural network modes, the content of the control software update register can be used to determine the activation function used by the hidden layer and the output layer, respectively. For example, in claim 4 or 5, in any of the freely-selective feedback node neural network modes, the content of the control software update register may be used to determine the use between the feedback processing layer and the hidden layer, respectively. Initial weight value. For example, the four-layer inverted transfer neural network mode described in claim 6 can use the content of the control software update register to determine the activation function used by the first hidden layer, the second hidden layer, and the output layer, respectively. . The freely selective feedback node neural network mode as described in claim 7, wherein the activation function is a double bending function or a double bending tangent function. The four-layer inverted transfer neural network mode as described in claim 9 wherein the activation function is a double bending function or a double bending tangent function.