TW201818264A - Buffer device and convolution operation device and method - Google Patents

Abstract

A buffer device includes input lines, an input buffer unit and a remap unit. The input lines are coupled to a memory and configured to be inputted with data from the memory in a current clock. The input buffer unit is coupled to the input lines and configured to buffer one part of the inputted data and output the part of the inputted data in a later clock. The remap unit is coupled to the input lines and the input buffer unit, and configured to generate remap data for convolution according to the data on the input lines and the output of the input buffer unit in the current clock.

Description

 Buffer device and convolution operation device and method  

The present invention relates to a buffer device and an arithmetic device, and more particularly to a buffer device and a convolution operation device for convolution operations.

Convolution is a mathematical operation that generates a third function through two functions. Convolution is widely used in science, engineering, and mathematics, such as image processing and electronic engineering and signal processing.

The Convolutional Neural Network (CNN) is also an application of convolution operations that includes one or more convolutional layers and associated weights and pooling layers. The convolutional neural network is a feedforward neural network, and its artificial neurons can respond to a part of the coverage area, which has a good performance for large image processing.

Convolutional neural networks are also used in deep learning, and convolutional neural networks produce better results in terms of image and speech recognition than other deep learning architectures. In addition, models using convolutional neural networks can also be trained using backpropagation algorithms. Compared with other deep learning architectures and feedforward neural networks, convolutional neural networks require fewer estimated parameters, so they become an important development trend of the current deep learning architecture.

However, convolution operations are very cost-effective operations. In convolutional neural network applications, convolution operations will dominate most processors. Therefore, how to provide a buffer device and a convolution operation device, which can improve the performance of convolution calculation, is one of the current important issues. In addition, multimedia applications such as streaming media in the Internet are becoming more and more widespread. Therefore, how to provide a buffer device and a convolution computing device, and also capable of processing data streams, is one of the current important issues.

In view of the above, it is an object of the present invention to provide a buffer device and a convolution operation device capable of processing a data stream.

SUMMARY OF THE INVENTION An object of the present invention is to provide a buffer device and a convolution operation device which can improve the performance of convolution calculation.

A buffer device is coupled to a memory device, the buffer device including an input line, an input buffer unit, and a remapping unit. The input line is coupled to the memory, configured to allow data to be input from the memory under a current clock; the input buffer unit is coupled to the input line, configured to buffer a portion of the input at the current clock to output the output at a subsequent clock. The remapping unit is coupled to the input line and the input buffer unit, configured to generate a plurality of remapping data according to the data on the input line and the output of the input buffer unit under the current clock, and remap the data for the input of the convolution operation .

In one embodiment, the input line inputs W data from the memory at the current clock, and the remapping unit generates W sets of remapping data for input of W convolution operations, respectively.

In one embodiment, the input line inputs W data from the memory at the current clock, and the input buffer unit buffers the last K data from the W data to output it at a subsequent clock, and outputs the output to the buffer unit. The system is arranged in front of the input line.

In an embodiment, each set of remapping data includes M remapping data, and the convolution calculation is a M x M convolution calculation.

In one embodiment, the remapping unit retrieves M data as a set of remapping data from the start end to the end per J step of the output of the input buffer unit and the input line.

In one embodiment, the stride between convolution operations is 1, each set of remapping data includes three remapping data, the convolution operation is a 3x3 convolution operation, and the input buffer unit is configured to be under the current clock. The last 2 input data buffers are output at subsequent clocks.

In an embodiment, the buffering device further includes a control unit coupled to and controlling the remapping unit.

In one embodiment, the data stored in the memory is two-dimensional array data, the buffer device is used as a column buffer, and the input buffer unit is used as a partial row buffer.

In an embodiment, the remapping unit is operable in a first convolution mode and a second convolution mode. When operating in the first convolution mode, the remapping unit generates remapping data according to the data on the input line and the output of the input buffer unit under the current clock, and remaps the data for input of the convolution operation; In the second convolution mode, the remapping unit outputs the data on the input line for input to the convolution operation at the current clock.

In one embodiment, the first convolution mode is a 3x3 convolution mode of operation and the second convolution mode is a 1x1 convolution mode of operation.

A convolution operation device includes a memory, a convolution operation module, and a buffer device. The buffer device is coupled to the memory and the convolution operation module. The buffer device includes an input line, an input buffer unit, and a remapping unit. The input line is coupled to the memory, configured to allow data to be input from the memory under a current clock; the input buffer unit is coupled to the input line, configured to buffer a portion of the input data at the current clock to be subsequently applied to the subsequent clock The remapping unit is coupled to the input line and the input buffer unit, configured to generate a plurality of remapping data according to the data on the input line and the output of the input buffer unit under the current clock, and remap the data input to the convolution operation Module.

In one embodiment, the input line inputs W data from the memory at the current clock, and the remapping unit generates W group remapping data to the convolution operation module, and the convolution operation module performs remapping data according to the W group. Convolution operation.

In one embodiment, the input buffer unit buffers the last K data from the W data for output at subsequent clocks, and the output of the input buffer unit is arranged in front of the input line.

In an embodiment, each set of remapping data includes M remapping data, and the convolution calculation is a M x M convolution calculation.

In one embodiment, the remapping unit retrieves M data as a set of remapping data from the start end to the end per J step of the output of the input buffer unit and the input line.

In one embodiment, the stride between convolution operations is 1, each set of remapping data includes three remapping data, the convolution operation is a 3x3 convolution operation, and the input buffer unit is configured to be at the current clock. Two input data buffers are used to output them at subsequent clocks.

In an embodiment, the buffering device further includes a control unit that couples and controls the remapping unit.

In one embodiment, the data stored in the memory is two-dimensional array data, the buffer device is used as a column buffer, and the input buffer unit is used as a partial row buffer.

In one embodiment, the amount of input data at the same clock is equal to the number of convolution operations performed by the convolutional computing module.

In one embodiment, the convolutional computing module and the buffering device are operable in a first convolution mode and a second convolutional mode. When operating in the first convolution mode, the remapping unit generates a plurality of remapping data according to the data on the input line and the output of the input buffer unit under the current clock, and the remapping data is input to the convolution operation module; When operating in the second convolution mode, the remapping unit outputs the data on the input line to the convolution operation module at the current clock.

In one embodiment, the first convolution mode is a 3x3 convolution mode of operation and the second convolution mode is a 1x1 convolution mode of operation.

A convolution operation method for data stream, comprising: obtaining data that has been input by a previous round of convolution operation from a buffer; and obtaining data that has not been input by a previous round of convolution operation from a data stream in a memory; The data obtained by the buffer and the data stream generates a plurality of sets of remapping data; the current convolution operation is performed based on a filter and each group of remapping data; and part of the data of the current convolution operation is reserved in the buffer for the next round Convolution operation.

In one embodiment, the data that has not been input by the W previous convolution operations is taken from the data stream in the memory, and the generated remapping data is the W group for the input of the W convolution operations.

In one embodiment, the last K data in the data that has not been input by the previous W convolution operations is retained in the buffer for the next round of convolution operations.

In an embodiment, each set of remapping data includes M remapping data, and the convolution calculation is a M x M convolution calculation.

In one embodiment, the stride between the convolution operations is 1, and each set of remapping data includes three remapping data, and the convolution operation is a 3×3 convolution operation, and the previous round of convolution operations is not yet input. The last two data are reserved in the buffer for the next round of convolution operations.

In one embodiment, the buffer is a scratchpad inside the processor, and the memory is a cache memory inside the processor.

As described above, in the buffer device and the convolution operation device, the input buffer unit of the buffer device can buffer part of the input data buffer as the input data of the next clock, even if the convolution operation requires more input data than The amount of memory that can be read at one time, the remapping unit can still obtain the missing data from the input buffer unit, thus providing sufficient remapping data for convolution operations, thereby improving the overall convolution operation performance. In addition, since the amount of data input from the memory and the number of outputs from the convolution operation are equal, this architecture is also well suited for processing data streams.

1‧‧‧ memory

2‧‧‧buffering device

21‧‧‧Input line

22‧‧‧Input buffer unit

23‧‧‧Remapping unit

24‧‧‧Control unit

3‧‧‧Convolutional computing module

30~37‧‧‧Convolution unit

4‧‧‧buffer unit

5‧‧‧Control unit

6‧‧‧Convolution unit

61‧‧‧ address decoder

62‧‧‧Adder

Conv_size, Stride, num_row, num_col‧‧‧ signals

PE0~PE8‧‧‧Processing unit

Data[47:0]‧‧‧ lines

Fc_bus[47:0]‧‧‧ lines

Psum[35:0]‧‧‧ output

Pm_0[31:0], pm_1[31:0], pm_2[31:0]‧‧‧ output

1 is a block diagram of a convolution operation device in accordance with an embodiment of the present invention.

2 is a schematic diagram of a convolution operation of the two-dimensional data by the convolution operation device of FIG. 1.

3 is a block diagram of a memory, a buffer device, and a convolution operation module according to an embodiment of the invention.

4 is a schematic view showing the operation of the shock absorber of FIG. 2.

Figure 5 is a block diagram of a convolution unit in accordance with an embodiment of the present invention.

Hereinafter, a buffer device and a convolution operation device according to an embodiment of the present invention will be described with reference to the accompanying drawings, wherein the same elements will be described with the same reference numerals, and the drawings are for illustrative purposes only and not for limitation. invention.

1 is a block diagram of a convolution operation device in accordance with an embodiment of the present invention. Referring to FIG. 1 , a convolution operation device includes a memory 1 , a convolution operation module 3 , a buffer device 2 , a control unit 5 , and a buffer unit 4 . The convolutional computing device can be used in the application of the Convolutional Neural Network (CNN).

The memory 1 stores data to be convoluted, such as images, video, audio, statistics, data of one layer of a convolutional neural network, and the like. In the case of image data, for example, pixel data; in the case of video data, for example, pixel data of a video frame or a motion vector, or audio in a video; In terms of data, it is usually a two-dimensional array of data, and often an image data.

All or most of the data may be stored elsewhere, for example, in another memory, to be fully or partially loaded into the memory 1 when the convolution operation is to be performed, and then the data is input to the volume through the buffer device 2. The product calculation module 3 performs a convolution operation. If the input data is streamed from the data stream, the memory 1 will write the latest data from the data stream for convolution at any time.

For frame data of image data or video, the order of processing is to read multiple columns at the same time, so in one clock, the buffer device 2 inputs the same line from the memory 1. The data on the different columns, that is, the buffer device 2 is buffered as a column.

The buffer device 2 is coupled to the memory 1 and the convolution operation module 3, and has a limited memory access width with the memory 1. The convolution operation module 3 can actually perform convolution operations and memory. The body access width is related. If the memory input has a frequency neck, the performance of the convolution operation will be affected by the impact.

In fact, the input required by the convolution operation module 3 has coefficients in addition to the data. Due to the parallel processing relationship, the convolution operation module 3 does not perform only one convolution operation, and he performs multiple adjacent operations at the same time. Convolution of data to improve performance. Since the size of the stride does not exceed the size of the sliding window or the convolution size, adjacent convolution operations tend to have overlapping data. In the general application of convolutional neural networks, the commonly used moving windows are 1×1, 3×3, 5×5, 7×7, etc., and 3×3 is more commonly used.

For example, the convolution operation module 3 has a plurality of convolution units, each convolution unit performs a convolution operation based on a filter and a plurality of current data, and retains part of the current data after the convolution operation. The buffer device 2 acquires a plurality of new data from the memory 1, and inputs the new data to the convolution unit, and the new data is not overlapped with the current data, for example, the previous round convolution operation is not used but the current convolution operation The data to be used. The convolution unit of the convolution operation module 3 performs a second round convolution operation based on the filter, the retained current data, and the new data.

In one embodiment, the convolution operation device is, for example, a processor, the memory 1 is, for example, a cache memory in the processor, and the convolution operation module 3 may include one or more convolution units. The array, the convolution unit array has a plurality of convolution units, and each convolution unit can separately process convolution operations of different sets of input data, and the different sets of input data may partially overlap with the previous or next set of input data. data. A plurality of functional units are included in the buffer device to increase the efficiency of parallel processing of convolution operations. The units of the control unit 5, the buffer unit 4, the convolution unit array, and the buffer device are constituted by digital logic circuits, and the inside of each unit may include a plurality of logic elements to perform their functions. The memory 1, the convolution operation module 3, the buffer device 2, the control unit 5, and the buffer unit 4 can be integrated in the same integrated circuit.

In other embodiments, the memory 1 may be a general random access memory (dram), and the convolution operation module 3, the buffer device 2, the control unit 5, and the buffer unit 4 may be integrated in the same integrated circuit.

The control unit 5 may include an instruction decoder and a controller, and the instruction decoder obtains an instruction from the controller and decodes the instruction, thereby obtaining the current input data size, the number of rows of the input data, the number of columns of the input data, and the moving window ( The sliding window) or the number called the convolution size and the starting address of the input data in the memory 1. In addition, the command decoder also obtains the moving window type information and the output feature number from the controller, and outputs an appropriate vacant signal to the buffer device 2, and the buffer device 2 operates according to the signals, and also controls the convolution unit array 3 and the buffer unit 4. The operation, for example, the timing of data input from the memory 1 to the buffer device 2 and the convolution operation module 3, the scale of the convolution operation of the convolution operation module 3, and the read position of the data from the memory 1 to the buffer device 2. The address, the data from the buffer unit 4 to the write address of the memory 1, the convolution operation module 3, and the convolution mode operated by the buffer device 2.

For example, the buffer device 2 and the convolution operation module 3 can operate in a first convolution mode and a second convolution mode, and the convolution mode is determined by the control unit in the buffer device 2, That is, the control unit in the buffer device 2 controls the buffer device 2 and the convolution operation module 3 to operate in the first convolution mode or the second convolution mode, and the different convolution modes are different sizes of sliding windows or This is called convolution size for convolution operations. For example, the first convolution mode is a 3x3 convolution operation mode, and the second convolution mode is a 1x1 convolution operation mode.

For example, the control unit 5 can receive a control signal or a mode command, and according to the control signal or mode command, determine which module and other mode operations the other modules are to operate on. This control signal or mode command can be derived from other control units or processing units.

When the buffer device 2 and the convolution operation module 3 operate in the first convolution mode, the buffer device 2 saves part of the input data of the previous clock to the current clock, and inputs data and current time according to the part of the previous clock. The input data of the pulse generates a plurality of remapping data, and the remapping data is input to the convolution operation module 3. These remapping data includes data required for multiple sets of moving windows, such as three remapping data for a 3x3 convolution mode of operation. Since the convolution operation module 3 has a parallel operation and a shift function, in the case of a stride of 1, three remapping data is provided under each clock, that is, the convolution operation module 3 can perform a 3×3 volume. Product operation.

When operating in the second convolution mode, the buffer device 2 outputs the input data under the current clock to the convolution operation module 3. At this time, it is not necessary to retain part of the input data of the previous clock to the current clock.

The buffer unit 4 temporarily stores the result of the convolution calculation, and if necessary, performs a pooling operation. The calculation result is stored in the memory 1 through the buffer device 2, and then output to the other places through the memory 1.

Please refer to FIG. 2. FIG. 2 is a schematic diagram of the convolution operation of the two-dimensional data by the convolution operation device of FIG. 1. The two-dimensional data has a plurality of rows and columns, which is, for example, an image, and only 5×4 of the pixels therein are schematically shown. A 3×3 matrix size filter is used for convolution of two-dimensional data. The filter has coefficients FC0~FC8, and the moving step of the filter is smaller than the shortest width of the filter. The size of the filter is comparable to a sliding window or a convolution window. The moving window can be moved on the 5×4 image, and each time the movement, the corresponding data P0~P8 in the window is subjected to a 3×3 convolution operation, and the result of the convolution operation can be referred to as a feature value. The interval at which the moving window S moves each time is called stride, and since the stride does not exceed the size of the sliding window or the convolution size, With the moving window stride of this embodiment, it will be less than the moving distance of 3 pixels. Moreover, adjacent convolution operations tend to have overlapping data. In the case where the stride is equal to 1, the data P2, P5, and P8 are new data, and the data P0, P1, P3, P4, P6, and P7 are data that has been input by the previous round of convolution operations. For the application of the general convolutional neural network, the commonly used moving window size is 1×1, 3×3, 5×5, 7×7, and the moving window size of the embodiment is more commonly used (3) ×3).

3 is a block diagram of a memory, a buffer device, and a convolution operation module according to an embodiment of the invention. Referring to FIG. 3, the buffer device 2 includes an input line 21, an input buffer unit 22, and a remapping unit 23. The input line 21 is coupled to the memory 1 and is configured to allow data to be input from the memory 1 under a current clock. The input buffer unit 22 is coupled to the input line 21 and is configured to buffer a portion of the input data under the current clock to output it at a subsequent clock. The remapping unit 23 is coupled to the input line 21 and the input buffer unit 22, and is configured to generate a plurality of remapping data according to the data on the input line 21 and the output of the input buffer unit 22 under the current clock, and remap the data input. The convolution operation module 3 is input for convolution operations.

For example, the input line 21 inputs W data from the memory 1 under the current clock, and the remapping unit generates W group remapping data to the convolution operation module 3, and the convolution operation module performs remapping data according to the W group. For each of the W convolution operations, each set of remapping data can be input to each convolution unit of the convolution operation module 3 to perform convolution operations, respectively. The input buffer unit 22 buffers the last K data from the W data to output it at a subsequent clock, and the output of the input buffer unit 22 is arranged in front of the input line 21. Each group of remapping data includes M remapping data, and the convolution calculation is M×M convolution calculation. The remapping unit 23 acquires M pieces of data as a set of remapping data from the start end toward the end every J step interval in the output of the input buffer unit 22 and the input line 21. In some implementations, the amount of input data at the same clock is equal to the number of convolution operations performed by the convolutional computing module.

In the present embodiment, W is 8, K is 2, M is 3, and J is 1. These values are merely illustrative and not limiting. In other embodiments, W, K, M, J may be other values.

In the present embodiment, the input line 21 inputs 8 data from the memory 1 under the current clock, the unit of the data is 1 byte, and the input buffer unit 22 is configured to input the last 2 input data under the current clock. Buffer to output it at subsequent clocks. For example, in the first clock, data R0 to R7 are input to the remapping unit 23, and data R6 to R7 are also input to the input buffer unit 22; in the second clock, data R8 to R15 are input to the second. The mapping unit 23, the data R14~R15 are also input to the input buffer unit 22; in the i+1th clock, the data Ri*8~Ri*8+7 is input to the remapping unit 23, and the data Ri*8+ 6~Ri*8+7 is also input to the input buffer unit 22.

In one clock, there will be 10 data input to the remapping unit 23. For example, in the first clock, the data R0 to R7 on the input line 21 are input to the remapping unit 23; in the second clock, the output data R6 to R7 of the input buffer unit 22 are input to the remapping unit. 23, the data R8~R15 on the input line 21 is input to the remapping unit 23; in the i+1th clock, the output data Ri*8-2~Ri*8-1 of the input buffer unit 22 is input to the re The mapping unit 23 inputs the data Ri*8~Ri*8+7 on the input line 21 to the remapping unit 23.

The remapping unit 23 generates 8 sets of remapping data to the convolution operation module 3, each group of remapping data includes 3 remapping data, each remapping data is 1 byte, and each convolution unit 30~37 The convolution operation is a 3×3 convolution operation, and the stride between convolution operations is 1. The generation of remapping data will be explained with reference to FIG.

Since the convolution operation is a 3x3 convolution operation and the stride is 1, the remapping unit 23 needs at least 10 data to generate 8 sets of remapping data for 8 convolution operations. If there is no input buffer unit 22, the remapping unit 23 can only obtain 8 data from the input line 21, and thus can only generate 6 sets of remapping data for 6 convolution operations, and each time the loading action from the memory 1 must be performed. Loading the two data that has been loaded the previous time, and the number of data input from the memory 1 at the same clock is not equal to the number of convolution operations performed by the convolution operation module 3, which will result in the whole Performance is declining.

In contrast, in the present case, by the setting of the input buffer unit 22, the remapping unit 23 can obtain 10 data in one clock, and thus can generate 8 sets of remapping data for 8 convolution operations, so that the amount of data input from the memory 1 The number of convolution operations performed by the convolution operation module 3 is equal.

4 is a schematic view showing the operation of the shock absorber of FIG. 3. Referring to FIG. 3 and FIG. 4, for example, the data stored in the memory 1 is two-dimensional array data, the buffer device 2 is used as a column buffer, and the input buffer unit 22 is used as a partial row buffer. .

The sliding window is a column that sequentially reads fixed 8 columns of data from the memory 1. In each clock, the eight columns of data of the memory 1 are taken as eight input data.

In the clock Clk n, the data R0 to R7 are input from the memory 1 through the input line 21 to the remapping unit 23, and the data R6 to R7 are also input to the input buffer unit 22. The remapping unit 23 is in the input line 21, that is, among the data R0 to R7, three data are acquired as a group of remapping data from the start end (data R0) toward the end (data R7) at intervals of one step. Each group of remapping data is a group of data R0~R2, a group of R1~R3, and a group of R5~R7. However, since the data input to the remapping unit 23 is insufficient, the remapping unit 23 can only obtain 8 data and thus can only generate 6 sets of remapping data for efficient convolution operations, and will generate 2 sets of unconservable convolution operations. Data, but this happens only at the beginning.

In the clock Clk n+1, the data R6 to R7 are input from the input buffer unit 22 to the remapping unit 23, and the data R8 to R15 are input from the memory 1 through the input line 21 to the remapping unit 23, and the data R14 to R15. It is also input to the input buffer unit 22. The remapping unit 23 is in the output of the input buffer unit 22 and the input line 21, that is, from the data R6 to R15, three data are obtained as one from the start end (R6) toward the end (R15) at intervals of one step. The group remaps the data, and each group remaps the data into a group of R6~R8, R7~R9 as a group, and R13~R15 as a group. There are already 10 data input to the remapping unit 23, and data of 8 sets of effective convolution operations will be generated.

By analogy, in the clock Clk n+i, the data Ri*8-2~Ri*8-1 will be input from the input buffer unit 22 to the remapping unit 23, and the data Ri*8~Ri*8+7 will be The memory 1 is input to the remapping unit 23 via the input line 21, and the data Ri*8+6~Ri*8+7 is also input to the input buffer unit 22. The remapping unit 23 is in the output of the input buffer unit 22 and the input line 21, that is, from the data Ri*8-2~Ri*8+7, from the start end (Ri*8-2) toward the end (Ri *8+7) Get 3 data per block as a set of remapping data, and remap the data in each group. Ri*8-2~Ri*8 are a group, Ri*8-1~Ri*8 +1 is a group, ...Ri*8+5~Ri*8+7 as a group. There are already 10 data input to the remapping unit 23, and 8 sets of data of the effective convolution operation will be generated. In the same clock, reading 8 data from the memory 1 will correspondingly generate 8 convolution operation results. .

In addition, please refer to FIG. 3 again, the buffer device 2 further includes a control unit 24, the control unit 24 is coupled to and controls the remapping unit 23, and the control unit 5 controls the convolution operation module 3, the buffer device 2, and the buffer unit 4. The operation, for example, the timing of data input from the memory 1 to the buffer device 2 and the convolution operation module 3, the scale of the convolution operation of the convolution operation module 3, and the read position of the data from the memory 1 to the buffer device 2. The address, the data from the buffer unit 4 to the write address of the memory 1, the convolution operation module 3, and the convolution mode operated by the buffer device 2. The signal conv_size, the signal Stride, the signal num_row, and the signal num_col are input to the control unit 24, the signal conv_size indicates the current convolution size, the signal Stride indicates the stride between the convolution operations, and the signal num_row indicates the number of columns of the current data. The signal num_col is the number of lines of the current data. The control unit 24 controls the operation mode of the convolution operation of the remapping unit 23 and the convolution operation module 3 according to the signal conv_size, and controls the read displacement according to the signal Stride.

For example, the remapping unit 23 and the convolution operation module 3 can operate in the first convolution mode and the second convolution mode, and the convolution mode is determined by the control unit 24, that is, the control unit 24 controls. The remapping unit 23 and the convolution operation module 3 operate in a first convolution mode or a second convolution mode, and different convolution modes are different sizes of a sliding window or a convolution size. ) to perform convolution operations. For example, the first convolution mode is a 3x3 convolution operation mode, and the second convolution mode is a 1x1 convolution operation mode.

When the remapping unit 23 and the convolution operation module 3 operate in the first convolution mode, the remapping unit 23 generates remapping data according to the data on the input line 21 and the output of the input buffer unit 22 under the current clock. The remapping data is input to the convolution operation module 3 for input of the convolution operation. The input buffer unit 22 retains part of the input data of the previous clock to the current clock, and the remapping unit 23 generates more data based on the previous clock portion input data output by the input buffer unit 22 and the input data on the current clock input line 21. The remapping data is input to the convolution operation module 3. These remapping data includes data required for multiple sets of moving windows, such as three remapping data for a 3x3 convolution mode of operation. Since the convolution operation module 3 has a parallel operation and a shift function, in the case of a stride of 1, three remapping data is provided under each clock, that is, the convolution operation module 3 can perform a 3×3 volume. Product operation.

When operating in the second convolution mode, the remapping unit 23 outputs the input data on the current clock input line 21 to the convolution operation module 3, at which time the remapping unit 23 does not need to acquire the retention from the input buffer unit 22. Part of the data from the previous clock.

Figure 5 is a block diagram of a convolution unit in accordance with an embodiment of the present invention. Hereinafter, a 33-convolution unit will be described as an example. Referring to FIG. 5, the convolution unit 6 includes nine processing units PE0 to PE8 (process engine), a bit address decoder 61, and an adder 62. The input data to be convoluted is input to the processing unit PE0~PE2 via the line data[47:0], and the processing units PE0~PE2 input the input data of the current clock to the processing unit PE3~PE5 in the next clock. PE3~PE5 will input the current clock input data to the processing unit PE6~PE8 at the next clock. The filter coefficients are input to the processing units PE0 to PE8 through the line fc_bus[47:0]. When the convolution operation is performed, the processing unit PE multiplies the filter coefficients of the selected address and the input data input to the processing units PE0 to PE8 through the address decoder 61. When the convolution unit 6 performs a 3 × 3 convolution operation, the adder 62 adds the results of the multiplication operations to obtain the result of the convolution operation as the output psum[35:0]. When the convolution unit 6 performs a 1×1 convolution operation, the adder 62 directly takes the result of the convolution operation of the processing units PE0 to PE2 as the output pm_0[31:0], pm_1[31:0], pm_2[31 :0].

In addition, a convolution operation method for data stream includes: obtaining data that has been input by a previous round of convolution operation from a buffer; and obtaining data that has not been input by a previous round of convolution operation from a data stream in a memory; The data obtained from the buffer and the data stream generates a plurality of sets of remapping data; the current convolution operation is performed based on a filter and each group of remapping data; and part of the data of the current convolution operation is reserved in the buffer for the second time A round of convolution operations.

In one embodiment, the data that has not been input by the W previous convolution operations is taken from the data stream in the memory, and the generated remapping data is the W group for the input of the W convolution operations. The last K data in the data that has not been input by the previous W convolution operation is reserved in the buffer for the next round of convolution operations. Each group of remapping data includes M remapping data, and the convolution calculation is M×M convolution calculation.

In one embodiment, the stride between the convolution operations is 1, and each set of remapping data includes three remapping data, and the convolution operation is a 3×3 convolution operation, and the previous round of convolution operations is not yet input. The last two data are reserved in the buffer for the next round of convolution operations.

In one embodiment, the buffer is a temporary memory inside the processor, and the memory is a cache memory inside the processor.

The convolution operation method can be applied or implemented in the convolution operation device of the foregoing embodiment, and related changes and implementations will not be described again. The convolution operation method can also be applied or implemented in other computing devices. For example, a convolution operation method of data stream can be used in a processor capable of executing an instruction, and an instruction configured to perform a convolution operation method is stored in a memory, and the processor couples the memory and executes the instruction to perform convolution. Algorithm. For example, the processor includes a cache memory, a math operation unit, and an internal register, the cache memory stores the data stream, the math operation unit can perform a convolution operation, and the internal register can retain the portion of the current convolution operation. The data is used in the convolutional computing module for the next round of convolution operations.

In summary, in the buffer device and the convolution operation device, the input buffer unit of the buffer device can buffer part of the input data buffer as the input data of the next clock, even if the convolution operation requires more input data than The amount of memory that can be read at one time, the remapping unit can still obtain the missing data from the input buffer unit, thus providing sufficient remapping data for convolution operations, thereby improving the overall convolution operation performance. In addition, since the amount of data input from the memory and the number of outputs from the convolution operation are equal, this architecture is also well suited for processing data streams.

The above-mentioned embodiments are not intended to limit the invention, and any equivalent modifications and variations of the present invention are intended to be included within the scope of the appended claims.

Claims (17)

 The buffer device is coupled to a memory, and includes: an input line coupled to the memory, configured to input data from the memory under a current clock; an input buffer unit coupled to the input line, Configuring to buffer a portion of the input data at the current clock to output it at a subsequent clock; and a remapping unit coupled to the input line and the input buffer unit configured to be under the current clock A plurality of remapping data is generated by inputting data on the line and the output of the input buffer unit, the remapping data being input to a convolution operation.    The buffer device of claim 1, wherein the input line inputs W data from the memory under the current clock, and the remapping unit generates W groups of the remapping data for respectively The input of a convolution operation.    The buffer device of claim 2, wherein the input buffer unit buffers the last K data from the W data to output it at a subsequent clock, and the output of the input buffer unit is arranged in The front of the input line.    The buffer device of claim 3, wherein each of the sets of the remapping data comprises M of the remapping data, the convolution calculation being a M x M convolution calculation.    The buffer device of claim 4, wherein the remapping unit acquires M data as a group of information from the start end to the end of each J-step interval in the output of the input buffer unit and the input line. Map data.    The buffer device of claim 3, wherein the stride between the convolution operations is 1, and each set of remapping data comprises three such remapping data, the convolution operation being a 3×3 convolution Operation, the input buffer unit is configured to buffer the last 2 input data at the current clock to output it at a subsequent clock.    The buffer device of claim 1, further comprising: a control unit coupled to and controlling the remapping unit.    The buffer device according to claim 1, wherein the data stored in the memory is two-dimensional array data, the buffer device is used as a column buffer, and the input buffer unit is used as a partial row buffer. .    The buffer device of claim 1, wherein the remapping unit is operable in a first convolution mode and a second convolution mode, the remapping unit when operating in the first convolution mode The remapping data is generated at the current clock according to the data on the input line and the output of the input buffer unit, the remapping data being an input for a convolution operation; when operating in the second convolution mode The remapping unit outputs the data on the input line for input to the convolution operation at the current clock.    The buffer device according to claim 9, wherein the first convolution mode is a 3×3 convolution operation mode, and the second convolution mode is a 1×1 convolution operation mode.    A convolution operation device, comprising: a memory; a convolution operation module; and the buffer device according to any one of claims 1 to 10, wherein the remapping data is input to the volume Product computing module.    A convolution operation method for data stream, comprising: obtaining data that has been input by a previous round of convolution operation from a buffer; and obtaining data that has not been input by a previous round of convolution operation from the data stream in a memory; Generating multiple sets of remapping data from the buffer and the data obtained by the data stream; performing a current convolution operation based on a filter and each group of remapping data; and retaining part of the data of the current convolution operation in the buffer For the next round of convolution operations.    The convolution operation method according to claim 12, wherein the data that has not been input by the W previous convolution operations is obtained from the data stream in the memory, and the generated remapping data is W. Groups are input for W convolution operations, respectively.    The convolution operation method according to claim 13, wherein the last K data of the data that has not been input by the previous W convolution operations is retained in the buffer for the next round of convolution operations.    The convolution operation method of claim 12, wherein each of the sets of the remapping data comprises M such remapping data, and the convolution calculation is M×M convolution calculation.    The convolution operation method according to claim 12, wherein the stride between the convolution operations is 1, and each group of remapping data includes three such remapped data, and the convolution operation is 3×3. Convolution operation, the last 2 data in the data that has not been input in the previous round of convolution operation is retained in the buffer for the next round of convolution operations.    The convolution operation method of claim 12, wherein the buffer is a temporary memory inside a processor, and the memory is a cache memory inside the processor.