CN106203617B - A kind of acceleration processing unit and array structure based on convolutional neural networks - Google Patents

Abstract

The invention discloses an accelerated processing unit based on a convolutional neural network, which is used for performing convolution operations on local data, the local data includes a plurality of multimedia data, and the accelerated processing unit includes a first register, a second register, and a second register. Three registers, a fourth register, a fifth register, a multiplier, an adder, and a first multiplexer and a second multiplexer. A single accelerated processing unit controls the first multiplexer and the second multiplexer, so that the multiplier and the adder can be reused, so that an accelerated processing unit only needs one multiplier and one adder to complete Convolution operation reduces the use of multipliers and adders. When implementing the same convolution operation, reducing the use of multipliers and adders will increase processing speed and reduce energy consumption. At the same time, the single acceleration processing unit has a smaller on-chip area .

Description

An accelerated processing unit and array structure based on convolutional neural network

technical field

The invention relates to a convolutional neural network, in particular to an accelerated processing unit and an array structure in a convolutional layer of the convolutional neural network.

Background technique

Compared with shallow learning, deep learning refers to the machine learning rules from historical data through algorithms, and intelligently identifying and predicting things.

Convolutional Neural Network (CNN) is a kind of deep learning. It was invented in the early 1980s and consists of artificial neurons arranged in multiple layers. The convolutional neural network reflects the way the human brain processes vision. As Moore's Law drives computer technology to become more and more powerful, convolutional neural networks can better imitate the actual operation of biological neural networks, avoid complex pre-processing of images, and can directly input original images, thus obtaining better results. For a wide range of applications, it has been successfully used in handwritten character recognition, face recognition, human eye detection, pedestrian detection and robot navigation.

The basic mechanism of convolutional neural network includes multiple convolutional layers, each layer is composed of multiple two-dimensional planes, and each plane is composed of multiple independent neurons. Each neuron is used to perform convolution operations on the local data of the multimedia data, and one of its input terminals is also connected to the local receptive field of the previous convolutional layer. product operation to extract the features of the local receptive field.

In the prior art, an accelerated processing unit is usually used as a neuron to perform convolution operations on partial data of multimedia data. The existing accelerated processing unit is designed with an adder and a multiplier for each input multimedia data. When the accelerated processing unit needs to process multiple local data, it means that each accelerated processing unit includes multiple additions. and multiple multipliers, this design leads to a larger area of the accelerated processing unit chip, a large power consumption, and a processing speed that needs to be improved.

Contents of the invention

The present application provides an accelerated processing unit based on a convolutional neural network, which is used to perform convolution operations on local data, the local data includes a plurality of multimedia data, and the accelerated processing unit includes a first register, a second register, and a second register. Three registers, a fourth register, a fifth register, a multiplier, an adder, and a first multiplexer and a second multiplexer;

The first register is used for inputting multimedia data, and its output end is connected with the input end of multiplier, and multimedia data is sent to multiplier;

The second register is used to input the filter weight value, and its output terminal is connected with the input terminal of the multiplier, and the filter weight value is sent to the multiplier;

The multiplier is used to multiply the multimedia data and the filter weight, and its output terminal is connected to the third register, and the multiplied result is sent to the third register;

The output end of the third register is connected with the first end of the first multiplexer;

The second end of the first multiplexer is connected to the adder, and the third end is part and input of the previous acceleration processing unit, and the first multiplexer connects the third register and the adder through state switching , or connect the part of the previous accelerated processing unit with the input end and the adder;

The adder is also connected with the fifth register and the fourth register, and is used for adding the multiplied result transmitted by the first multiplexer or the partial sum of the previous acceleration processing unit to the data in the fifth register , and output the added result to the fourth register;

The first terminal and the second terminal of the second multiplexer are respectively connected to the fourth register and the fifth register, and the fourth register is connected to the fifth register through the second multiplexer.

Preferably, the first multiplexer maintains the first state when the acceleration processing unit has not completed the multiplication and addition operation of the local data, and connects the third register to the adder, and after the acceleration processing unit completes the multiplication and addition operation of the local data To switch to the second state, connect the part and the input of the previous accelerated processing unit to the adder.

Preferably, the second multiplexer remains in the first state when the acceleration processing unit has not completed the multiplication and addition operation of the local data, connects the fourth register to the fifth register, and completes the multiplication and addition of the local data in the acceleration processing unit switch to the second state after the operation to clear the fifth register.

Preferably, the third terminal of the second multiplexer is a reset terminal, and the second multiplexer switches to the second state after the acceleration processing unit completes the multiplication and addition operation of local data, and the reset terminal connected to the fifth register.

Preferably, it also includes a first memory, a second memory and a third memory, the first memory is connected to the input end of the first register, and is used to input and store the local data that needs to be convoluted, and convert the local data into A plurality of multimedia data are sent to the first register in turn; the second memory is connected to the input of the second register for inputting and storing the filter weight, and sending the filter weight to the second register; The third memory is connected to the input terminal of the fourth register, and is used for inputting and storing the added result output by the adder, and sending the added result to the fourth register.

Preferably, it is characterized in that the adder also outputs the added result to the next accelerated processing unit.

The present application also provides an array structure based on a convolutional neural network, including multiple accelerated processing units, and the multiple accelerated processing units are in the form of a matrix with M rows and N columns, where M and N are integers greater than or equal to 1 , the accelerated processing units of each column are connected back and forth.

Preferably, in each column, the output terminal of the adder of the preceding accelerated processing unit is connected to the third terminal of the first multiplexer of the subsequent accelerated processing unit.

Preferably, in the accelerated processing units in the same row, the input filter weights are the same; in the accelerated processing units located in the same diagonal line, the input local data are the same.

Preferably, in the accelerated processing units of different rows, input filter weights are different.

The beneficial effects of the present invention are: a single accelerated processing unit controls the first multiplexer and the second multiplexer, so that the multiplier and the adder can be reused, so that one accelerated processing unit only needs one multiplier The convolution operation can be completed with one adder, reducing the use of multipliers and adders. When implementing the same convolution operation, reducing the use of multipliers and adders will increase processing speed and reduce energy consumption. At the same time, a single Accelerated processing units have smaller on-chip areas.

Description of drawings

FIG. 1 is a structural block diagram of an accelerated processing unit based on a convolutional neural network provided by an embodiment of the present invention;

2 is a schematic diagram of a convolution operation process of an accelerated processing unit based on a convolutional neural network provided by an embodiment of the present invention;

3 is a schematic diagram of column distribution of an array structure based on a convolutional neural network according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of row distribution of an array structure based on a convolutional neural network according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a diagonal distribution of an array structure based on a convolutional neural network according to an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below through specific embodiments in conjunction with the accompanying drawings. Apparently, the described embodiments are only some of the embodiments of the present invention, rather than all of them.

Embodiment one:

Please refer to FIG. 1 , the present embodiment provides an accelerated processing unit based on a convolutional neural network. The accelerated processing unit 61 includes a first register 21, a second register 22, a third register 23, a fourth register 24, and a fifth register 25. , a multiplier 41 , an adder 51 , and the first multiplexer 31 and the second multiplexer 32 .

The first register 21 is connected to an input end of the multiplier 41 , and the first register 21 is used for inputting multimedia data and sending the multimedia data to the multiplier 41 . The second register 22 is connected to the other input terminal of the multiplier 41 , and the second register 22 is used to input the filter weight and send the filter weight to the multiplier 41 . The output terminal of the multiplier 41 is connected to the third register 23 for multiplying the multimedia data and the filter weight, and sending the multiplied result to the third register 23 .

The first terminal of the first multiplexer 31 is connected to the output terminal of the third register 23 , the second terminal is connected to an input terminal of the adder 51 , and the third terminal is part and input terminal of the previous accelerated processing unit. When the first multiplexer 31 is switched to the first state (such as setting 0), the first multiplexer 31 connects the third register 23 and the adder 51, and sends the data in the third register 23 to the adder 51; when the first multiplexer 31 is switched to the second state (for example, set to 1), the first multiplexer 31 connects its third terminal to the adder 51, and sends the part sum of the previous accelerated processing unit to adder 51.

The other input end of adder 51 is connected with the 5th register 25, and the output end of adder 51 is connected with the 4th register 24, and adder 51 inputs the data in the 3rd register 23 and the 5th register 25, with the data in the two registers. The data is added, and the added result (also called internal partial sum) is output to the fourth register 24 .

The first terminal and the second terminal of the second multiplexer 32 are connected to the fourth register 24 and the fifth register 25 respectively, and the third terminal of the second multiplexer 32 is a reset terminal. When the second multiplexer 32 is switched to the first state (for example, set to 0), the second multiplexer 32 will connect the fourth register 24 and the fifth register 25, and send the internal part in the fourth register 24 to To the fifth register 25; when the second multiplexer 32 switches to the second state (for example, setting 1), the second multiplexer 32 connects its third terminal to the fifth register 25, and resets the fifth register 25. Clear the data in the fifth register 25 to zero.

In some embodiments, for the convenience of sending data to the registers, the accelerated processing unit 61 further includes a first memory 11, a second memory 12, and a third memory 13, the first memory 11 is connected to the input end of the first register 21 for It is used to input and store the local data that needs to be convoluted, and send multiple multimedia data in the local data to the first register 21 in sequence; the second memory 12 is connected to the input end of the second register 22 for input and storage filter weight, and send the filter weight to the second register 22; the third memory 13 is connected to the input end of the fourth register 24, and is used to input and store the internal part sum of the adder 51 output, and the internal part and sent to the fourth register 24.

The acceleration processing unit 61 is used for performing convolution operation on partial data, the partial data includes a plurality of multimedia data, and the multimedia data may be video data, image data, or audio data. When the multimedia data is video data, it can be considered that each multimedia data corresponds to one pixel.

Taking image data as an example, the convolution operation process of the accelerated processing unit 61 will be described below.

1 and 2, the working process of a single convolutional neural network-based accelerated processing unit 61 is as follows:

Step 10, read the video data and filter weights that need to be convolved. If the image data is not 0, the image data is stored in the first memory 11, and sent to the first register 21 to extract the image data when needed, and if the image data is 0, the image data 0 is directly sent to the first register 21 without extracting, skipping or gating strategy is adopted to avoid unnecessary reading and calculation; the filter weight value is stored in the second memory 12, and is sent to the second register 22 for filter weight when needed Value data, wherein, the data extraction method is serial extraction in sequence, that is, in the first cycle, the first image data in the local data that is convoluted by the acceleration processing unit 61 is sent to the first register 21; In the second cycle, the second image data is sent to the first register 21, and then the image data is sequentially read in. The filter weights are generated by the processor according to the requirements of the convolution algorithm.

Step 20, multiplication operation. The image data in the first register 21 and the filter weight in the second register 22 are sent to the multiplier 41 for multiplication, and the multiplied result by the multiplier 41 is output to the third register 23 .

Step 30, addition operation. Since the multiplication and addition operation in the acceleration processing unit 61 has not yet ended, the first multiplexer 31 is set to 0, and when the first multiplexer 31 is set to 0, the image data in the third register 23 is sent to the addition In the register 51, the adder 51 adds the image data to the previous internal partial sum in the fifth register 25. For the first internal convolution operation, the fifth register 25 is zero, and for the subsequent internal convolution operation, the fifth register 25 is the internal partial sum after the previous convolution operation. The result of the addition in this convolution operation (ie, the internal partial sum) is output to the fourth register 24, and an internal convolution operation is completed at this time, and finally the first image data and the part of the filter weight are obtained and. Since the multiplication and addition operation in the acceleration processing unit 61 has not yet ended, the second multiplexer 32 is set to 0 at this moment, and when the second multiplexer 32 is set to 0, the internal part sum is sent to the first by the fourth register 24 Five Register 25.

Step 40, the acceleration processing unit 61 judges whether the internal convolution operation of all local data is completed, if the internal convolution operation is not completed, step 10, step 20 and step 30 will be repeated in sequence to extract the second image data, and Input to the first register 21, the second filter weight is input to the second register 22, the image data in the first register 21 and the filter weight in the second register 22 are all sent in the multiplier 41, the image data Multiplied with the filter weight, the result obtained is sent to the third register by the multiplier 41. Since the multiplication and addition operation in the acceleration processing unit 61 has not yet ended, the first multiplexer 31 is set to 0 at this moment, and the first multiplexer 31 is set to 0. The data in the third register 23 is sent to the adder 51 through the first multiplexer, and summed with the data in the fifth register 25 to finally obtain the partial sum of the second image data and the filter weight. The partial sum in the adder 51 is sent in the fourth register 24. At this time, because the multiplication and addition operation in the accelerated processing unit has not yet ended, the second multiplexer 32 is set to 0 at this time, and the data in the fourth register 24 sent to the fifth register 25 through the second multiplexer 31. Thus completing the inner convolution operation on the second image data. Until the last image data of the local data is extracted, the image information and the filter weights undergo the above-mentioned multiplication and addition operations to obtain the partial sum of this accelerated processing unit, and this partial sum is finally entered into in the fifth register 25. When all inner convolution operations have been completed, then step 50 will proceed.

Step 50, after the multiplication and addition operation on the local data in the accelerated processing unit 61 ends, the first multiplexer 31 and the second multiplexer 32 are set to 1, and when the first multiplexer 32 is set to 1 , the part sum in the previous acceleration processing unit is sent to the adder 51 by the first multiplexer 31, and the fifth register 25 sends the final part sum in the acceleration processing unit 61 to the adder 51, and finally, the previous The partial sum of one accelerated processing unit is summed with the partial sum of this accelerated processing unit 61 to obtain the superimposed partial sum of the two accelerated processing units, and the output of the superimposed partial sum is sent to the next accelerated processing unit. When the second multiplexer 32 will be set to 1 from state 0, the fourth register 24 will no longer send data to the fifth register 25, and the data in the fifth register 25 will be cleared.

In this embodiment, a single accelerated processing unit controls the first multiplexer 31 and the second multiplexer 32 to make the multiplier 41 and the adder 51 reusable, so that one accelerated processing unit only needs one A multiplier and an adder can complete the convolution operation, reducing the use of multipliers and adders. When implementing the same convolution operation, reducing the use of multipliers and adders will increase processing speed and reduce energy consumption. At the same time, the area on a single accelerated processing unit chip is smaller.

Embodiment two:

Please refer to Figures 3 to 5, which show an array structure based on a convolutional neural network, including multiple accelerated processing units, and the multiple accelerated processing units are in the form of a matrix with M rows and N columns, where M and N is an integer greater than or equal to 1, and the accelerated processing units of each column are connected successively.

In this embodiment, multiple accelerated processing units are in the form of a matrix of 3 rows and 3 columns. In each column, the output end of the adder of the previous accelerated processing unit is connected to the first multiplexer of the next accelerated processing unit. Three ends.

In the accelerated processing units in the same row, the input filter weights are the same; in the accelerated processing units located in the same diagonal line, the input local data are the same.

In the accelerated processing units of different rows, the input filter weights are different.

The following describes the convolutional layer operation process of multiple acceleration processing units with reference to the accompanying drawings.

Combining Figures 1 to 5, the operation process of the convolutional neural network-based array structure is as follows:

As shown in Figure 3, the adder 51 of the preceding acceleration processing unit is connected to the first multiplexer 31 of the subsequent acceleration processing unit, and the partial sum output by each row is moved vertically, so that the partial sums of the two accelerated processing units before and after are moved vertically. Accumulations, which can be read out on the top row at the end of a computation pass, are buffered into the bottom row of the array at the beginning of the next computation pass.

For example, the accelerated processing unit PE1.1, the accelerated processing unit PE2.1, and the accelerated processing unit PE3.1 first perform internal convolution operations respectively, and the final results are stored in their respective fifth registers 25, and then the accelerated processing unit PE3. The partial sum output in 1 and the partial sum of the fifth register 25 in the accelerated processing unit PE2.1 are summed and accumulated again in the adder 51 of the accelerated processed unit PE2.1 to obtain the first accumulated partial sum. The part of one accumulation is sent to the acceleration processing unit PE1.1 by the acceleration processing unit PE2.1, and the part of the fifth register 25 in the acceleration processing unit PE1.1 and in the adder 51 of the acceleration processing unit PE1.1 Sum again, and finally output the partial sum of all accelerated processing units 1 in this column.

It should also be pointed out that, as shown in Figure 4 and Figure 5, in the accelerated processing units in the same row, the input filter weights are the same, and in the accelerated processing units located on the same diagonal line, the input image data are the same, but different In the accelerated processing units of the row, the input filter weights are different. Since the entire image data has several rows, and each accelerated processing unit only processes a single row of data in the entire image data, the accelerated processing unit needs to process each row of data separately and then accumulate the convolution results of each row of data. The input data on the same diagonal are the same, and the input image data on different diagonals are different, which means that the input image data on different diagonals are different rows of image data. Processing different rows of image data requires different filter weights. For example, a filter weight is only used to process the first row of image data. When processing the second row of image data, a new filter weight is required. device weight. Therefore, accelerated processing units in the same row can use the same filter weight, and accelerated processing units in different rows can use different filter weights.

For example, the filter weights in the accelerated processing unit PE1.1, the accelerated processing unit PE1.2 and the accelerated processing unit PE1.3 are the same, and the input image data in the accelerated processing unit PE2.1 and the accelerated processing unit PE1.2 are the same, The filter weights in the accelerated processing unit PE1.1, the accelerated processing unit PE2.2 and the accelerated processing unit PE3.1 are different.

In this way, one line of multimedia data is processed at the same time, and then different filter weights are used for different lines of multimedia data. After each line of data is processed separately, the multimedia data of the previous and subsequent lines is accumulated, so as to quickly and reliably Handle all multimedia data.

The above uses specific examples to illustrate the present invention, which is only used to help understand the present invention, and is not intended to limit the present invention. For those skilled in the technical field to which the present invention belongs, some simple deduction, deformation or replacement can also be made according to the idea of the present invention.

Claims (10)

1. a kind of acceleration processing unit based on convolutional neural networks, for carrying out convolution algorithm, the part to local data Data include multiple multi-medium datas, it is characterised in that are posted including the first register, the second register, third register, the 4th Storage, the 5th register, multiplier, adder and the first multiple selector and the second multiple selector；

First register sends out multi-medium data for inputting multi-medium data, the input terminal connection of output end and multiplier It is sent to multiplier；

Second register is used for input filter weights, and filter weights are sent out in the input terminal connection of output end and multiplier It is sent to multiplier；

Multiplier is for multi-medium data to be multiplied with filter weights, and output end is connect with third register, after multiplication Result be sent to third register；

The output end of third register is connect with the first end of the first multiple selector；

The second end of first multiple selector connects adder, and third end is part and the input of previous acceleration processing unit End, first multiple selector is switched by state connects third register with adder, or previous acceleration is handled list The part of member is connected with input terminal with adder；

The adder is also connect with the 5th register and the 4th register, after the multiplication for transmitting the first multiple selector Result or previous acceleration processing unit part and carry out add operation with the data in the 5th register, and after will add up As a result it is output to the 4th register；

The first end and second end of second multiple selector is separately connected the 4th register and the 5th register, and the described 4th Register is connected to the 5th register by the second multiple selector.

2. acceleration processing unit as described in claim 1, which is characterized in that first multiple selector is accelerating processing single First state is kept when the multiply-add operation of the unfinished local data of member, third register is connected to adder, is handled in acceleration It is switched to the second state after the multiply-add operation of unit completion local data, the part of previous acceleration processing unit and input terminal are connected It is connected to adder.

3. acceleration processing unit as described in claim 1, which is characterized in that second multiple selector is accelerating processing single First state is remained when the multiply-add operation of the unfinished local data of member, the 4th register is connected to the 5th register, is being added It is switched to the second state after the multiply-add operation of fast processing unit completion local data, the 5th register is reset.

4. acceleration processing unit as claimed in claim 3, which is characterized in that attach most importance at the third end of second multiple selector End is set, second multiple selector is switched to the second state after the multiply-add operation that acceleration processing unit completes local data, Resetting end is connected to the 5th register.

5. acceleration processing unit according to any one of claims 1 to 4, which is characterized in that further include first memory, Two memories and third memory, the first memory are connect with the input terminal of the first register, are needed for inputting and storing The local data of convolution algorithm is carried out, and multiple multi-medium datas in local data are sent to the first register successively； The second memory is connect with the input terminal of the second register, is weighed for inputting and storing filter weights, and by filter Value is sent to the second register；The third memory is connect with the input terminal of the 4th register, for inputting and storing addition Device output after being added as a result, and the result after will add up be sent to the 4th register.

6. acceleration processing unit according to any one of claims 1 to 4, which is characterized in that the adder also will add up Result afterwards is output to latter acceleration processing unit.

7. a kind of array structure based on convolutional neural networks, which is characterized in that including multiple as any in claim 1 to 6 Acceleration processing unit described in, multiple acceleration processing units are rendered as the matrix shape of M rows N row, wherein M and N be more than or Integer equal to 1 is connected before and after the acceleration processing unit of each row.

8. array structure as claimed in claim 7, which is characterized in that in each row, the adder of previous acceleration processing unit Output end connect latter acceleration processing unit the first multiple selector third end.

9. array structure as claimed in claim 8, which is characterized in that in the acceleration processing unit of a line, the filtering of input Device weights are identical；In the acceleration processing unit on same diagonal line, the local data of input is identical.

10. array structure as claimed in claim 9, which is characterized in that in the acceleration processing unit that do not go together, the filtering of input Device weights are different.