WO2021088563A1 - Convolution operation circuit, apparatus and method - Google Patents

Abstract

Disclosed in the embodiments of the present disclosure are a convolution operation circuit, an apparatus and a method. The convolution operation circuit comprises: a convolution operation control circuit; an operation unit array, which comprises a plurality of operation units; the convolution operation control circuit is configured to receive a convolution operation instruction, and to transmit, according to an operation order indicated by the convolution operation instruction, inputted data and weight data one by one to operation units, participating in the operation, among the plurality of operation units; and the operation units participating in the operation performs convolution operation on the inputted data and the weight data according to the indication of the instruction, the instruction being a single instruction. Said method solves the technical problems in the prior art of low calculation efficiency and high power consumption during convolution calculation.

Description

Convolution operation circuit, device and method Technical field

The present disclosure relates to the field of neural network computing, and in particular to a convolution operation circuit, device and method.

Background technique

With the development of science and technology, human society is rapidly entering the era of intelligence. The important feature of the intelligent age is that people are getting more and more types of data, the amount of data is getting bigger and bigger, and the speed of processing data is getting higher and higher. Chips are the cornerstone of data processing, it is fundamentally determined The ability of people to process data. From the perspective of application fields, there are two main routes for chips: one is a general-purpose chip route, such as CPU (Central Processing Unit, central processing unit), etc. They can provide great flexibility, but they are effective in processing algorithms in specific fields. The power is relatively low; the other is a dedicated chip route, such as TPU (Tensor Processing Unit, tensor processor), etc. They can exert higher effective computing power in some specific fields, but they are more versatile in the face of flexible and changeable In the field, their processing power is relatively poor or even unable to handle. Due to the wide variety and huge amount of data in the intelligent era, the chip is required to have extremely high flexibility, capable of processing different fields and rapidly changing algorithms, and extremely strong processing capabilities, which can quickly process extremely large and rapidly increasing data. the amount.

Convolution calculations are often required in artificial intelligence calculations. In the existing schemes for realizing convolution calculations, there are usually two schemes:

(1) CPU scheme: In this scheme, if it is a single-core CPU, the matrix involved in the convolution calculation will be disassembled into scalars for operation, and the convolution operation will be realized by combining scalar instructions; if it is a multi-core CPU, it may pass Multiple cores execute their scalar instructions in parallel, and combine them to implement convolution operations. However, the use of this solution has the following disadvantages: the underlying program is complex, and generally requires multi-layer loops to implement convolution operations; convolution operations are implemented through general calculation instructions, which is inefficient and require multiple branch jumps; CPU cache is limited, and the implementation is relatively large Convolution operation needs to move data from off-chip multiple times, which affects efficiency; CPU needs to access data multiple times, which will increase the calculation time for convolution operation; CPU needs to access data multiple times, which will increase the realization of convolution The computational power consumption added by the calculation; if it is a multi-core parallel calculation, the communication between the cores is complicated, and the communication performance may become a bottleneck.

(2) GPU (Graphics Processing Unit) solution: In this solution, the GPU will disassemble the convolution operation into multiple instruction operations. These instructions are mainly vector instructions, and the convolution operation is realized by combining and executing vector instructions. However, the use of this solution has the following disadvantages: the underlying program is complex, and generally requires multiple layers of loops to implement convolution operations; the convolution operation is achieved through multiple combinations of vector instructions, which is inefficient; GPU requires multiple data access, which will increase the implementation The calculation time of convolution operation; GPU needs to access data multiple times, which will increase the calculation power consumption of convolution operation; GPU has limited cache, and the realization of relatively large convolution operation requires multiple transfers from outside the chip. effectiveness.

Summary of the invention

The content of the invention is provided to introduce concepts in a brief form, and these concepts will be described in detail in the following specific embodiments. The content of the invention is not intended to identify the key features or essential features of the technical solution required to be protected, nor is it intended to be used to limit the scope of the technical solution required to be protected.

In order to solve the technical problems of low calculation efficiency and high power consumption when performing convolution calculations in the prior art, the embodiments of the present disclosure propose the following technical solutions:

In the first aspect, an embodiment of the present disclosure provides a convolution operation circuit, including:

Convolution operation control circuit;

An arithmetic unit array, the arithmetic unit array includes a plurality of arithmetic units;

The convolution operation control circuit is configured to receive a convolution operation instruction, and transmit the input data and the weight data one by one to those involved in the operation in the plurality of operation units according to the operation sequence indicated by the convolution operation instruction. An arithmetic unit, where the arithmetic unit participating in the operation performs a convolution operation on the input data and the weight data according to the instruction of the instruction, wherein the instruction is a single instruction.

Further, the convolution operation instruction includes an instruction name, a first address of input data, a first address of weight data, and a first address of output data.

Further, the arithmetic unit includes an arithmetic unit, and the arithmetic unit includes at least a multiplier and an adder;

The operation unit is configured to combine the multiplier and the adder according to the convolution operation instruction to perform a convolution operation.

Further, the operation of the operation unit participating in the operation to perform a convolution operation on the input data and the weight data according to the instruction of the instruction includes:

Calculating the product of the input data and the weight data by a multiplier;

Calculating the accumulated value of the product through an adder;

The accumulated value is output.

Further, the input data is the row vector data of the first input matrix; the weight data is the row vector data of the convolution kernel.

In the second aspect, an embodiment of the present disclosure provides a convolution operation device, including:

Memory, used to store convolution operation instructions, input data, weight data and output data;

An instruction fetching module, connected to the memory, and configured to acquire the convolution operation instruction from the memory;

A decoding module, connected to the instruction fetching module, and configured to decode the convolution operation instruction acquired by the instruction fetching module;

Register for storing the attribute data of the input data, the attribute data of the weight data, and the attribute data of the output data;

The execution module is connected to the decoding module, the memory and the register, and includes the convolution operation circuit according to claims 1-6, which is used to execute the decoded convolution operation instruction.

Further, the execution module obtains the decoded convolution operation instruction from the decoding module;

The execution module obtains the attribute data of the input data, the attribute data of the weight data, and the attribute data of the output data from the register;

The execution module obtains the input data and the weight data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data;

The execution module calculates the input data and weight data according to the decoded convolution operation instruction to obtain output data;

The execution module stores the output data in the memory according to the attribute data of the output data.

Further, the input data is data of a first input matrix, the weight data is data of a convolution kernel, and the output data is data of an output matrix obtained by convolution; the attribute data of the input data includes the The row number, column number, depth, and depth storage interval of the first input matrix; the attribute data of the weight data includes the row number, column number, step length of the convolution kernel, and row storage interval of the convolution kernel; The attribute data of the output data includes the number of rows, the number of columns, the depth, and the depth storage interval of the output matrix.

Further, the execution module acquiring the input data and the weight data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data includes:

The execution module reads the data of the first input matrix according to the preset first reading method and the attribute data of the input data;

The execution module reads the data of the convolution kernel according to the preset second reading method and the attribute data of the weight data.

Further, the first reading mode is row reading or column reading; the second reading mode is row reading or column reading.

In a third aspect, embodiments of the present disclosure provide a matrix operation method, which is based on the matrix operation method of the convolution operation circuit described in any one of the foregoing first aspects, and is characterized in that it includes:

Fetch the convolution operation instruction from the memory;

Decoding the convolution operation instruction, and sending the decoded convolution operation instruction to the convolution operation circuit;

Based on the decoded convolution operation instruction, the convolution operation circuit obtains input data and weight data from the memory and performs operations, and stores the operation result in the memory after the operation is completed.

In a fourth aspect, an embodiment of the present disclosure provides an electronic device, including: a memory, configured to store computer-readable instructions; and one or more processors, configured to execute the computer-readable instructions to cause the processor to run The convolution operation method described in any one of the foregoing third aspects is realized at the time.

In a fifth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the aforementioned third aspect Any of the convolution operation methods described above.

In a sixth aspect, embodiments of the present disclosure provide a computer program product, which is characterized in that it includes computer instructions, and when the computer instructions are executed by a computing device, the computing device can execute any of the foregoing third aspects. The convolution operation method.

In a seventh aspect, an embodiment of the present disclosure provides a chip, which is characterized by comprising the matrix operation circuit described in any one of the first aspect.

In an eighth aspect, an embodiment of the present disclosure provides a computing device, which is characterized by including the chip described in any one of the seventh aspects.

The embodiments of the present disclosure disclose a convolution operation circuit, device and method. Wherein the convolution operation circuit includes: a control circuit; a convolution operation control circuit; an arithmetic unit array, the arithmetic unit array includes a plurality of arithmetic units; the convolution operation control circuit is used to receive a convolution operation instruction, according to the The operation sequence indicated by the convolution operation instruction transmits the input data and the weight data one by one to the operation unit participating in the operation among the plurality of operation units, and the operation unit participating in the operation performs the operation according to the instruction of the instruction. The input data and the weight data perform a convolution operation operation, wherein the instruction is a single instruction. Through the above method, the technical problems of low calculation efficiency and high power consumption in the prior art when performing convolution calculations are solved.

The above description is only an overview of the technical solutions of the present disclosure. In order to understand the technical means of the present disclosure more clearly, they can be implemented in accordance with the content of the specification, and to make the above and other objectives, features and advantages of the present disclosure more obvious and understandable. In the following, the preferred embodiments are cited in conjunction with the drawings, and the detailed description is as follows.

Description of the drawings

The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent in conjunction with the accompanying drawings and with reference to the following specific implementations. Throughout the drawings, the same or similar reference signs indicate the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

FIG. 1 is a schematic structural diagram of a matrix operation circuit provided by an embodiment of the disclosure;

FIG. 2 is a schematic structural diagram of an arithmetic unit provided by an embodiment of the disclosure;

FIG. 3 is a further structural schematic diagram of an arithmetic unit provided by an embodiment of the disclosure;

Figures 4a-4b are overall schematic diagrams of convolution operations in an embodiment of the present disclosure;

5-9 are the decomposition calculation process of the convolution operation provided by the embodiments of the disclosure;

FIG. 10 is a schematic diagram of a specific example of a convolution operation in an embodiment of the disclosure.

FIG. 11 is a schematic structural diagram of a matrix calculation device provided by an embodiment of the disclosure;

Figures 12a-12e are schematic diagrams of the storage formats of input data, weight data, and output data in this disclosure;

FIG. 13 is a schematic diagram of block calculation of convolution operation in the present disclosure;

14a-14f are schematic diagrams of examples of performing convolution operations using the convolution operation circuit disclosed in an embodiment of the present disclosure.

Detailed ways

Hereinafter, embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided for Have a more thorough and complete understanding of this disclosure. It should be understood that the drawings and embodiments of the present disclosure are only used for exemplary purposes, and are not used to limit the protection scope of the present disclosure.

It should be understood that the steps recorded in the method embodiments of the present disclosure may be executed in a different order, and/or executed in parallel. In addition, method implementations may include additional steps and/or omit to perform the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "including" and its variations as used herein are open-ended includes, that is, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related definitions of other terms will be given in the following description.

It should be noted that the concepts of “first” and “second” mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units. Or interdependence.

It should be noted that the modifications of “a” and “a plurality of” mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as “one or Multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

FIG. 1 is a schematic structural diagram of a convolution operation circuit provided by an embodiment of the disclosure. As shown in FIG. 1, the convolution operation circuit 100 includes a convolution operation control circuit 101 and an arithmetic unit array 102, wherein the arithmetic unit array includes a plurality of arithmetic units (PU) 103, and the arithmetic unit 103 is shown in FIG. 2 , Including a first input register (Rin1) 104, a second input register (Rin2) 105, and an output register (Rout) 106, wherein the first input register 104 is used for receiving input data, and the second input register 105 is used for Receiving weight data; the convolution operation control circuit 101 is configured to receive a convolution operation instruction, and transmit the input data and the weight data to the multiple operations one by one according to the operation sequence indicated by the convolution operation instruction The arithmetic unit 103 that participates in the operation in the unit 103, and the arithmetic unit 103 that participates in the operation performs a convolution operation on the input data and the weight data according to the instruction of the instruction, wherein the instruction is a single instruction; The output register 106 is used to store the operation result of the operation operation, that is, output data.

The operation order indicated by the convolution operation instruction refers to that during the convolution operation, the convolution operation is divided into multiple steps to be executed, and the context between the steps is the operation order indicated by the convolution operation instruction. For the specific operation sequence, please refer to the description of the instructions below.

The arithmetic unit involved in the operation refers to at least one arithmetic unit that performs a specific operation among a plurality of arithmetic units.

In the present disclosure, the convolution operation instruction includes the instruction name, the first address of input data, the first address of weight data, and the first address of output data. The following table shows an exemplary command format:

The name of the instruction corresponds to the meaning, format, and operation of the instruction. The first address of the input data and the first address of the weight data respectively define the read addresses of the two source operands of the instruction, and the first address of the output data defines the purpose of the instruction. The storage address of the operand. The convolution instruction includes a multiplication operation and an addition operation. It first performs a multiplication operation on the input data and the weight data, and then accumulates the result of the multiplication operation to obtain the convolution result of the input data and the weight data, that is, the output data . Among them, the calculation process of an output data is as follows:

among them

FIG. 3 is a further structural schematic diagram of an arithmetic unit provided by an embodiment of the disclosure. As shown in FIG. 3, in addition to the first input register, the second input register, and the output register, the arithmetic unit also includes an arithmetic unit. The arithmetic unit includes at least a multiplier 301 and an adder 302, wherein the arithmetic unit For combining the multiplier and the adder according to the convolution operation instruction to perform a convolution operation.

Specifically, the operation unit participating in the operation performs a convolution operation operation on the input data and the weight data according to the instruction of the instruction, including: calculating the input data and the weight through the multiplier The product of data; the accumulated value of the product is calculated by the adder; and the accumulated value is stored in the output register. Wherein, the input data in the first input register is the data sequentially read into the first input register according to the first address of the input data; the weight data in the second input register is according to the The first address of the weight data is sequentially read into the data in the second input register.

For example, after the convolution operation control circuit transmits the input data and weight data to the arithmetic unit, the arithmetic unit calculates the product of the input data in the first input register and the weight data in the second input register through a multiplier, as in the above example, In one clock cycle, if the data received in the first input register is a ₁ and the data received in the second input register is b ₁ , the multiplier in the arithmetic unit calculates the product of _{a 1} *b _{1, and then} Send the product of a ₁ *b _{1 to} the adder for accumulation. At this time, the accumulator has no input in the previous clock cycle, so the accumulation result of the accumulator is a ₁ *b ₁ ; then in the next clock cycle, continue The above calculation operation, at this time the data received in the first input register is a ₂ , and the data received in the second input register is b ₂ , then the multiplier in the arithmetic unit calculates the product of _{a 2} *b _{2, and then} The product of a ₂ *b ₂ is sent to the adder for accumulation. The input of the previous clock cycle of the accumulator is a ₁ *b ₁ , so calculate a ₁ *b ₁ +a ₂ *b ₂ in this clock cycle; continue Do the above operations until one row of input data and one column of weight data are calculated, and the final accumulated value is obtained, and then the accumulated value is stored in the output register. The convolution operation control circuit stores the accumulated value in the output register in the system memory according to the first address of the output data in the instruction.

An exemplary convolution operation is as follows:

4a-4b are schematic diagrams of convolution operations in an embodiment of the disclosure. Figure 4a is the overall schematic diagram of the convolution operation process, and the diagram is explained as follows:

Win: Enter the width of the feature map (Feature Map);

Hin: input the height of the feature map;

Cin: The number of channels of the input feature map, collectively referred to as the depth of the input feature map later;

S_Cin: the interval of physical storage on the depth of the input feature map;

Wout: the width of the output feature map;

Hout: the height of the output feature map;

Cout: The number of channels of the output feature map, collectively referred to as the depth of the output feature map later;

S_Cout: the interval of physical storage in the depth of the output feature map;

Kw: the width of the convolution kernel;

Kh: the height of the convolution kernel;

S_kernel: the interval between the convolution kernel on the physical storage;

N_dilation: the dilation value of the convolution kernel;

S_K: the step size of convolution sliding;

N_pad_u: add 0 number above;

N_pad_d: add 0 number below;

N_pad_l: add 0 to the left;

N_pad_r: Add 0 to the party.

The feature points on the input feature map constitute the first input matrix; the output feature map is the output matrix, and a feature point on the output feature map is a piece of data on the output matrix. During the convolution operation, the convolution kernel will slide on the input feature map. Each time it slides, it will multiply and accumulate data with the corresponding data in the input feature map to extract an output feature point, that is, a data on the output matrix. .

Figure 4b is a schematic diagram of the calculation of an input feature point with depth. As shown in Figure 4b, the convolution kernel slides on the input feature map. When it stops at a position, it will multiply and accumulate the corresponding data with the feature points in the input feature map at that position to obtain the output feature corresponding to the position. Points; there are Cout convolution kernels, and each convolution kernel will multiply and accumulate data with the feature points in the input feature map at the same position to obtain Cout output feature points in the depth direction; Cout output feature points are composed A feature point with depth on the entire output feature map, the depth of this point is Cout; the convolution kernel will slide the entire input feature map to obtain the entire output feature map.

For a certain convolution kernel at depth l (1<=l<=Cout), its feature extraction formula is as follows:

Dout is a point with depth in the output feature map, and its superscript l corresponds to the depth of the output; Din refers to the data corresponding to the convolution kernel in the input feature map, and its superscript i corresponds to the depth of the input feature map, j And k respectively correspond to the width and height of the convolution kernel; w is the convolution kernel, and its superscripts l and i correspond to the depth of the output feature map and the depth of the input feature map, respectively, and j and k correspond to the width of the convolution kernel. And height.

最后将这Kh个部分结果相加即得到最终结果

For the convolution kernel of size Kh*Kw*Cin, it can be divided into Kh Kw*Cin partial convolution kernels, and partial feature extraction is performed. Each time, 1/Kh of the entire feature extraction is realized, which is Kw*Cin Part of the features corresponding to the convolution kernel, and the partial results obtained are:

Finally, add these Kh partial results to get the final result

又可以分成Kw步，每一步实现

然后将Kw个部分结果相加即得到最终结果

的实现，是一个一行的输入数据矩阵(即部分的第一输入矩阵)和一个一列的权重矩阵(即部分的卷积核)相乘，其实现如图5所示。 among them,

It can be divided into Kw steps, each step is realized

Then add the Kw partial results to get the final result

The realization of is to multiply a row of input data matrix (that is, part of the first input matrix) and a column of weight matrix (that is, part of the convolution kernel), and the realization is shown in Fig. 5.

的实现，同样是一个一行的输入数据矩阵和一个一列的权重矩阵相乘，只是此时的行和列中数据的个数是

中数据的个数的Kw倍，其实现如图6所示。

The realization of the same is the multiplication of a row of input data matrix and a column of weight matrix, but the number of data in rows and columns at this time

Kw times the number of data in the middle, its realization is shown in Figure 6.

的实现，还是一个一行的输入数据矩阵和一个一列的权重矩阵相乘，只是此行和此列中元素的个数，是

中元素的个数的Kh倍，其实现如图7所示。 And for the convolution of the points to get a complete output feature map, that is

The realization is that a row of input data matrix is multiplied by a column of weight matrix, but the number of elements in this row and this column is

Kh times the number of elements in the middle, the realization is shown in Figure 7.

The number of convolution kernels is Cout. Therefore, the depth of the output feature point is Cout. The input data matrix of one row can be multiplied by the convolution kernel matrix of the Cout column composed of Cout convolution kernels, that is, the weight matrix. A feature point with depth, this feature point is a vector, the length of the vector is the depth Cout of the output feature point, and its realization is shown in Figure 8.

And because the process of convolution or partial convolution of the neural network is the sliding process of the convolution kernel on the input feature map, it can be regarded as the process in which the data of the input feature map changes with the sliding, while the weight remains unchanged. The process of convolution by the network becomes the convolution kernel sliding Wout*Hout times on the output feature map. The input data matrix of the Wout*Hout row and the weight matrix of the Cout column are multiplied to obtain the Wout*Hout row and Cout. The output data matrix of the column, its realization is shown in Figure 9.

The above-mentioned convolution operation in this disclosure only needs to use a single instruction, that is, the above-mentioned convolution operation instruction can complete the entire convolution process. It only requires input data, weights, and output data to be stored in a predetermined manner. The entire convolution can be realized by a single convolution operation circuit.

Specifically, in the convolution operation circuit in the foregoing embodiment: the input data is the row vector data of the first input matrix; the weight data is the row vector data of the convolution kernel. In other words, the data in the first input register and the data in the second input register both need to be row vector data of the matrix, so that the calculated data is the convolution result. Figure 10 is a schematic diagram of the above convolution calculation. Take the first input matrix as a 3*3*2 matrix and the convolution kernel as two 2*2*2 matrices as an example, the step size is 1, the output The matrix is a 2*2*2 matrix. As shown in FIG. 10, when performing convolution, the convolution kernel slides on the first input matrix, and the data of the first input matrix read is as shown in 1001, and each row is a convolution kernel 1002. The data of the first input matrix corresponding to the position includes 4 numbers with a depth of 2, a total of 8 data; one column in 1002 is the data read by a convolution kernel in a depth-first manner, including the depth of The 4 numbers of 2, a total of 8 data,; a row of data in 1001 and a column of data in 1002 are multiplied and accumulated to obtain a data in 1003, and a data in 1003 is a depth of 2 on the output matrix One of the two values in the data, each row of the output data, corresponds to a data with a depth of 2 in the output matrix.

Combined with the PU array shown in Figures 1-3, briefly explain how the PU array implements the calculation of convolution.

As shown in Figure 10, 1001 is the storage structure of the first input matrix, 1002 is the storage structure of the convolution kernel, 1003 is the storage structure of the output matrix, and 1004 is the operation sequence of the convolution. In the read data, each row of data represents the data block corresponding to when the convolution kernel slides on the first input matrix. After the arithmetic unit reads out the data, it sends the data to the arithmetic units participating in the calculation one by one. For example: the first value 1 in the first row of data 1004 of the first input matrix in Figure 10 is sent to Rin1 of PU11 and Rin1 of PU12, and the first value of data 1004 in the second row of the first input matrix is 3 Rin1 of PU21 and Rin1 of PU22, the first value 7 of the third row of the first input matrix data 1004 is sent to Rin1 of PU31 and Rin1 of PU32, the first of the fourth row of the first input matrix data 1004 The value 9 is sent to Rin1 of PU41 and Rin1 of PU42; the first value 0.1 in the first column of convolution kernel 1002 is sent to Rin2 of PU11, Rin2 of PU21, Rin2 of PU31, and Rin2 of PU41, convolution kernel 1002 The first value of 0.9 in the second column in is sent to Rin2 of PU12, Rin2 of PU22, Rin2 of PU32, and Rin2 of PU42; PU11, PU12, PU21, PU22, PU31, PU32, PU41, and PU42 perform data multiplication operations respectively, And send the result to the output register to save; in the next clock cycle, the second value 2 of the first row of the data 1004 of the first input matrix is sent to Rin1 of PU11 and Rin1 of PU12, and the data of the first input matrix 1004 is the first The second value 4 of the two rows is sent to Rin1 of PU21 and Rin1 of PU22, the second value 8 of the third row of the first input matrix data 1004 is sent to Rin1 of PU31 and Rin1 of PU32, and the first input matrix data 1004 The second value 10 in the fourth row of the is sent to Rin1 of PU41 and Rin1 of PU42; the second value of 0.2 in the first column of the convolution kernel 1002 is sent to Rin2 of PU11, Rin2 of PU21, Rin2 of PU31, and Rin2 of PU41, the second value 1.0 in the second column of convolution kernel 1002 is sent to Rin2 of PU12, Rin2 of PU22, Rin2 of PU32 and Rin2 of PU42, PU11, PU12, PU21, PU22, PU31, PU32, PU41 , PU42 performs this data multiplication operation respectively, and sends the result to the output register and accumulates the result saved last time.

By analogy, the result of multiplication and accumulation of convolution is finally obtained.

FIG. 11 is a schematic structural diagram of a matrix calculation device provided by an embodiment of the disclosure. As shown in FIG. 11, the convolution operation device 1100 includes: a memory 1101 for storing convolution operation instructions, input data, weight data, and output data; and an instruction fetching module 1102, which is connected to the memory 1101, and is used to store convolution operation instructions, input data, weight data, and output data; The memory 1101 obtains the convolution operation instruction; the decoding module 1103 is connected to the instruction fetching module 1102 and is used to decode the convolution operation instruction obtained by the instruction fetching module 1102; the register 1104 , Used to store the attribute data of the input data, the attribute data of the weight data, and the attribute data of the output data; the execution module 1105 is connected to the decoding module 1103, the memory 1101 and the register 1104, It includes the convolution operation circuit in the above embodiment, and is used to execute the decoded convolution operation instruction.

In one embodiment, the execution module obtains the decoded matrix operation instruction from the decoding module; the execution module obtains the attribute data of the input data and the attribute of the weight data from the register Data and attribute data of the output data; the execution module obtains the input data and the weight data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data; The execution module calculates the input data and weight data to obtain output data according to the decoded convolution operation instruction; the execution module stores the output data in the memory according to the attribute data of the output data in. Wherein, the input data is data of a first input matrix, the weight data is data of a convolution kernel, and the output data is data of an output matrix obtained by convolution; the attribute data of the input data includes the first The number of rows, the number of columns, the depth, and the depth storage interval of an input matrix; the attribute data of the weight data includes the number of rows, the number of columns, the step length of the convolution kernel, and the storage interval between the convolution kernels; The attribute data of the output data includes the number of rows, the number of columns, the depth, and the depth storage interval of the output matrix. Wherein, the number of rows and the number of columns defines the size of the matrix, the input data storage interval defines the storage address difference of the input data in depth, and the output data storage interval defines the storage address of the output data in depth Difference, the storage interval of the convolution kernel defines the storage address difference between the two convolution kernels. For example, the first input matrix has 10 int8 matrix elements in the depth direction. If it is stored continuously, the row storage interval is 10Byte. If it is stored at a certain interval, such as 20Byte, then 10Byte is a matrix element, and the other 10Byte is The content does not belong to this matrix, it may be invalid data, or it may be data for other purposes.

Optionally, the execution module obtains the input data and the weight data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data, including: the execution module Read the data of the first input matrix according to the preset first reading method and the attribute data of the input data; the execution module reads the data of the first input matrix according to the preset second reading method and the attribute data of the weight data Take the data of the convolution kernel. Optionally, the first reading mode is row reading or column reading; the second reading mode is row reading or column reading.

For example, if the attribute data defines that the number of rows of the first input matrix is 5 rows, the number of columns is 5 columns, the depth is 2, and the depth storage interval is 4, the preset reading method is reading by row, Then read the first row of the first input matrix according to the first address, number of columns, depth, and depth storage interval of the first input matrix in the instruction. You can know that the first row has 5 matrix elements through the number of columns. The depth of the element is 2, then after reading 2*5=10 data, one row of data of the first input matrix is obtained, and then the first address plus 5 depth storage intervals are used as the new first address, and the second row is read, There are also 5 matrix elements, so read 5 times in sequence, the execution module can obtain all the data of the first input matrix.

Similarly, the data of the convolution kernel is obtained in the same way. Optionally, the execution module storing the data of the output matrix in the memory according to the attribute data of the output matrix includes: the execution module according to a preset storage mode and the attribute data of the output matrix The data of the output matrix is stored in the memory. Wherein, the predetermined storage mode is row storage or column storage, and the specific storage mode is similar to reading, but the direction is opposite, so it will not be repeated here.

Fig. 12a is a schematic diagram of the storage order and format of the first input matrix in the present disclosure. As shown in Figure 12a, it is an example of the first input matrix in the above embodiment. When it is stored in the memory, it is stored in a manner of depth Cin first, then width Win, and finally height Hin. The depth interval is S_Cin. . Taking Cin=2, Win=3, Hin=3, S_Cin=4 as an example, first store the first point of the first line in Hin=3 rows. Due to the depth, the one point contains 2 data. S_Cin=4, so add 4 data storage addresses to the first address of storage, and then store the second point, which also contains 2 data, and then add 4 data storage addresses to the first address of the point , And then save the third point, which also contains 2 data, until the third point is stored, there are a total of 3*2=6 data in the first row, including 3 points in the Win direction; after that, Hin=3 The first point of the second line in the row is stored in this way until all the points are stored. An example of the storage order and format of the first input matrix is shown in Figure 12b.

Fig. 12c is a schematic diagram of the storage order and format of the convolution kernel of the present disclosure. As shown in Fig. 12c, it is an example of the convolution kernel in the above-mentioned embodiment. The number) priority is stored in rows, and each column stores a convolution kernel. In the column direction, the depth Cin of the convolution kernel is prioritized, then the width Kw, and then the height Kh. Taking Cout=2, Cin=2, Kw=2, Kh=2, S_kernel=2 as an example, a total of Cin*Kw*Kh=8 rows of data will be stored, and each row will store 2 data. First store the first data in the depth 2 point in the first row and first column of the first convolution kernel, and then store the depth 2 point in the first row and first column of the second convolution kernel Complete the storage of the first row of data of the convolution kernel; then calculate the first address of the second row by adding S_kernel to the first address of the convolution kernel, and store the first row of the first convolution kernel. The second data in the point with a depth of 2 in the column, and then the second data in the point with a depth of 2 in the first row and the first column of the second convolution kernel is stored to complete the second row of the convolution kernel The data is stored in this way, until all the points are stored. An example of the storage order and format of the convolution kernel is shown in Figure 12d.

Because the output matrix can be used as the first input matrix for the next convolution calculation, in one embodiment, the storage order and format of the output matrix are the same as the first input matrix, as shown in FIG. 12e, which is the output of the present disclosure. Schematic diagram of the storage order and format of the matrix.

As shown in Figure 12a, rs1 is the first address of the first input matrix. The data of the matrix that is read in can be controlled by setting the storage format and the setting of the reading method. For example, according to the order of the depth direction in the above example, with the row reading and the depth interval, you can read the first Enter the row vector data of the matrix. As shown in Figure 12c, rs2 is the first address of the convolution kernel. The data of the read matrix can be controlled by setting the storage format and the setting of the reading mode. For example, store the data in the order in the above example. According to the row reading and the row interval of the convolution kernel, the row vector data of the convolution kernel can be read. After the data of the first input matrix and the convolution kernel are read, the convolution operation of the single instruction is completed by using the convolution operation instruction. The rd shown in Figure 12e is the first address of the output matrix. The output data calculated by the above convolution operation instruction is stored from rd in the manner in FIG. 12e until all the output data is stored.

Before calling the above-mentioned convolution operation instruction, it is necessary to define the parameters of the convolution operation. In the present disclosure, a register 1104 is set to store the parameters of the convolution. The first input matrix, the convolution kernel, and the output The attribute data of the matrix is set. An example configuration of register 1104 is shown in the following table:

Register name Register function description (31:0) Conv_FM_in 31:16 (the width of FM_in Win); 15:0 (the height of FM_in Hin) Conv_Depth_in 31:16 (the interval on the depth of FM_in is S_Cin); 15:0 (the depth of FM_in is Cin) Conv_FM_out 31:16 (width Wout of FM_out); 15:0 (height Hout of FM_out) Conv_Depth_out 31:16 (the interval on the depth of FM_out is S_Cout); 15:0 (the depth of FM_out is Cout) Conv_S_kernel 15:0 (the line interval between the convolution kernel in the physical storage is S_kernel) Conv_kernel 31:24 (Kernel width Kw); 23:16 (Kernel height Kh); 15:8 (Kernel N_dilation); 7:0 (convolution sliding window step size S_K) Conv_padding 31:24 (upper complement 0 number N_pad_u); 23:16 (lower complement 0 number N_pad_d); 15:8 (left complement 0 number N_pad_l); 7:0 (right complement 0 number N_pad_r)

Among them, FM_in is the first input matrix, FM_out is the output matrix, and Kernel is the convolution kernel

In the convolution operation, the convolution operation control circuit calculates the addresses of input data, weight data and output data in the order of logical operation; the arithmetic unit array reads the input data and weight data from the memory according to the address, and performs calculations. Then the calculation result is stored in the memory according to the address of the output data, and the entire convolution operation is completed.

However, due to the limited number of arithmetic units in the arithmetic unit array, such as only M*N, and the number of elements in the output matrix is much more than M*N, the input data and weights are usually divided into convolution operations. Block read in, and block output and storage for output data. The schematic diagram of the convolution operation is shown in Figure 13. The input data and weight data are divided into multiple M*N data blocks into the arithmetic unit array, and the calculated The result is also divided into multiple M*N data blocks and output separately.

Figures 14a-14f show an example of the convolution operation circuit performing the convolution operation according to the convolution operation instruction. In this example, the parameters of the convolution operation are as follows:

The arithmetic unit array M*N is 2*2;

The attribute data of the input data (the first input matrix) is: Win=Hin=Cin=4, S_Cin=8;

The attribute data of the output data (output matrix) is: Wout=Hout=Cout=4, S_Cout=8;

The attribute data of the weight data (convolution kernel) is: S_kernel=8, Kw=Kh=1, N_dilation=1, S_K=1;

The parameter of Padding is: N_pad_u=N_pad_d=N_pad_l=N_pad_r=0.

The entire convolution diagram is shown in Figure 14a: 4*4*4 input data and 4 1*1*4 weight data are convolved to obtain 4*4*4 output data.

The storage format of the input data is shown in FIG. 14b. Each row in the input data storage format stores a point in the input data with a depth of 4, and the interval between two adjacent points is 8. The storage format of the weight data is shown in Figure 14c. Each column in the storage format of the weight data stores 4 points of a convolution kernel, and each row stores the points of 4 convolution kernels at the same depth. Is 8. The storage format of the output data is shown in Figure 14d. The storage format of the output data is the same as the storage format of the input data. Each row in the output data storage format stores a point in the output data with a depth of 4. The interval between is 8.

The actual calculation process of convolution is shown in Figure 14e and Figure 14f. Specifically, in one clock cycle, the convolution operation control circuit reads input data from the first input matrix, and sends the input data to the first input register of the arithmetic unit, and the convolution operation control circuit reads from the convolution core Take the weight data and send the weight data to the second input register of the arithmetic unit. In the next clock cycle, the arithmetic unit in the arithmetic unit uses the input data in the first input register and the weight data in the second input register for multiplication , And accumulate with the accumulated value of the multiply-accumulate operation in the previous cycle. As shown in Fig. 14e, since the size of the arithmetic unit array is 2*2, the input data and weight data are respectively taken in 2*2 units for calculation. Take the data and weight for the first time to perform matrix multiplication and addition operations, and then temporarily store the intermediate data. Each arithmetic unit PU realizes the multiplication and addition of the row of its corresponding input data and the column of weight data. For example, when the input data and weight data have been sent to the first input register and the second input register of the arithmetic unit, In the first clock cycle, PU11 calculates 1*0.1=0.1. Since 0.1 is only a part of the entire convolution, it is temporarily stored in the output register instead of being written into the memory. Similarly, PU12 calculates 1*0.5=0.5; PU21 calculates 5*0.1=0.5; PU22 calculates 5*0.5=2.5. After calculating the above data, in the second clock cycle, PU11 calculates 2*0.2=0.4, and accumulates the next calculation result of 0.1 to obtain: 0.4+0.1=0.5, which is temporarily stored in the output register; PU12 calculates 2* 0.6=1.2, and add the second calculation result 0.5, get: 1.2+0.5=1.7, temporarily store it in the output register; PU21 calculates 6*0.2=1.2, and add the second calculation result 0.5, get: 1.2+0.5 =1.7, temporarily store it in the output register; PU22 calculates 6*0.6=3.6, and accumulates the second calculation result 2.5 to obtain: 3.6+2.5=6.1, temporarily stores it in the output register. In the third clock cycle, PU11 calculates 3*0.3=0.9, and accumulates the next calculation result 0.5 to obtain: 0.9+0.5=1.4, which is temporarily stored in the output register; PU12 calculates 3*0.7=2.1, and accumulates The second calculation result of 1.7 is obtained: 2.1+1.7=3.8, which is temporarily stored in the output register; PU21 calculates 7*0.3=2.1, and the second calculation result of 1.7 is accumulated, and the following calculation result is obtained: 2.1+1.7=3.8, which is temporarily stored Output register; PU22 calculates 7*0.7=4.9, and accumulates the second calculation result 6.1 to obtain: 4.9+6.1=11, which is temporarily stored in the output register. In the fourth clock cycle, PU11 calculates 4*0.4=1.6, and accumulates the next calculation result 1.4 to get: 1.6+1.4=3, which is temporarily stored in the output register; PU12 calculates 4*0.8=3.2, and accumulates The second calculation result of 3.8 is obtained: 3.2+3.8=7, which is temporarily stored in the output register; PU21 calculates 8*0.4=3.2, and the second calculation result of 3.8 is accumulated, and the following calculation result is obtained: 3.2+3.8=7, which is temporarily stored Output register; PU22 calculates 8*0.8=6.4, and accumulates the second calculation result 7.4 to obtain: 6.4+11=17.4, temporarily store it in the output register to obtain output data, as shown in Figure 14f.

Finally, all output results will be stored in the storage area.

It is understandable that there can be multiple first input registers, second input registers, and output registers of each arithmetic unit PU, so that the convolution operation control circuit transmits data to the first input register and the second input register and from the output When reading data from a register, it can be performed alternately between different registers, so as to achieve a parallel operation of transferring data to the register and arithmetic unit for operation, thereby improving calculation efficiency.

The embodiments of the present disclosure also provide a convolution operation method, which is based on any of the foregoing convolution operation circuits, and is characterized in that it includes: fetching a convolution operation instruction from a memory; Instruction is decoded, and the decoded convolution operation instruction is sent to the convolution operation circuit; based on the decoded convolution operation instruction, the convolution operation circuit obtains input data and The data is weighted and the calculation is performed, and the calculation result is stored in the memory after the calculation is completed.

An embodiment of the present disclosure also provides an electronic device, including: a memory, configured to store computer-readable instructions; and one or more processors, configured to run the computer-readable instructions, so that the processor can realize The convolution operation method described in any one of the foregoing embodiments.

The embodiments of the present disclosure also provide a non-transitory computer-readable storage medium, which is characterized in that the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute any of the foregoing embodiments. The convolution operation method.

An embodiment of the present disclosure provides a computer program product, which is characterized by including computer instructions, and when the computer instructions are executed by a computing device, the computing device can perform any of the convolution operations in the foregoing embodiments method.

An embodiment of the present disclosure provides a chip, which is characterized by including the convolution operation circuit described in any of the foregoing embodiments.

An embodiment of the present disclosure provides a computing device, which is characterized by including the chip described in any of the foregoing embodiments.

The flowcharts and block diagrams in the drawings of the present disclosure illustrate the possible implementation architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations Or it can be realized by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure can be implemented in software or hardware. Among them, the name of the unit does not constitute a limitation on the unit itself under certain circumstances.

The functions described hereinabove may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logical device (CPLD) and so on.

In the context of the present disclosure, a machine-readable medium may be a tangible medium, which may contain or store a program for use by the instruction execution system, apparatus, or device or in combination with the instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

Claims (11)

A convolution operation circuit, including: Convolution operation control circuit; An arithmetic unit array, the arithmetic unit array includes a plurality of arithmetic units; The convolution operation control circuit is used to receive a convolution operation instruction, and transmit the input data and weight data to the operation units participating in the operation of the plurality of operation units one by one according to the operation sequence indicated by the convolution operation instruction, so The arithmetic unit participating in the operation performs a convolution operation on the input data and the weight data according to the instruction of the instruction, where the instruction is a single instruction. 3. The convolution operation circuit according to claim 1, wherein the convolution operation instruction includes an instruction name, a first address of input data, a first address of weight data, and a first address of output data. The convolution operation circuit according to claim 3, wherein: The arithmetic unit includes an arithmetic unit, and the arithmetic unit includes at least a multiplier and an adder; The operation unit is configured to combine the multiplier and the adder according to the convolution operation instruction to perform a convolution operation. The convolution operation circuit according to claim 3, wherein the operation unit participating in the operation performs a convolution operation operation on the input data and the weight data according to the instruction of the instruction, comprising: Calculating the product of the input data and the weight data by the multiplier; Calculating the accumulated value of the product by the adder; The accumulated value is output. The convolution operation circuit according to claim 4, wherein: The input data is row vector data of the first input matrix; The weight data is row vector data of the convolution kernel. A convolution operation device, including: Memory, used to store convolution operation instructions, input data, weight data and output data; An instruction fetching module, connected to the memory, and configured to acquire the convolution operation instruction from the memory; A decoding module, connected to the instruction fetching module, and configured to decode the convolution operation instruction acquired by the instruction fetching module; Register for storing the attribute data of the input data, the attribute data of the weight data, and the attribute data of the output data; The execution module is connected to the decoding module, the memory and the register, and includes the convolution operation circuit according to claims 1-6, which is used to execute the decoded convolution operation instruction. The convolution operation device according to claim 6, wherein: The execution module obtains the decoded convolution operation instruction from the decoding module; The execution module obtains the attribute data of the input data, the attribute data of the weight data, and the attribute data of the output data from the register; The execution module obtains the input data and the weight data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data; The execution module calculates the input data and weight data according to the decoded convolution operation instruction to obtain output data; The execution module stores the output data in the memory according to the attribute data of the output data. The convolution operation device according to claim 7, wherein: The input data is data of a first input matrix, the weight data is data of a convolution kernel, and the output data is data of an output matrix obtained by convolution; The attribute data of the input data includes the number of rows, the number of columns, the depth, and the depth storage interval of the first input matrix; The attribute data of the weight data includes the number of rows, the number of columns, the step size of the convolution kernel, and the row storage interval of the convolution kernel; The attribute data of the output data includes the number of rows, the number of columns, the depth, and the depth storage interval of the output matrix. The convolution operation device according to claim 7, wherein the execution module obtains the input data and all the input data for calculation from the memory according to the attribute data of the input data and the attribute data of the weight data. The weight data includes: The execution module reads the data of the first input matrix according to the preset first reading method and the attribute data of the input data; The execution module reads the data of the convolution kernel according to the preset second reading method and the attribute data of the weight data. The convolution operation device according to claim 9, wherein: The first reading method is reading by row or reading by column; The second reading method is reading in rows or reading in columns. A convolution operation method based on the convolution operation circuit of any one of claims 1 to 5, characterized in that it comprises: Fetch the convolution operation instruction from the memory; Decoding the convolution operation instruction, and sending the decoded convolution operation instruction to the convolution operation circuit; Based on the decoded convolution operation instruction, the convolution operation circuit obtains input data and weight data from the memory and performs operations, and stores the operation result in the memory after the operation is completed.

Images