TWI898206B - Operation conversion method for neural network, method of performing matrix multiplication operation based on convolution operation, and intelligence processing unit - Google Patents

Abstract

An operation conversion method for neural networks, a method of performing a matrix multiplication operation based on a convolution operation, and an intelligence processing unit (IPU). The method of performing a matrix multiplication operation based on a convolution operation includes the following steps: (A) reading a first data from a first storage device and storing the first data in a second storage device; (B) reading from the first storage device a second data and storing the second data in the second storage device; (C) performing the convolution operation on the first data and the second data to obtain a first result; and (D) storing the first result in the first storage device. The first result is equal to a second result obtained by performing the matrix multiplication operation on the first data and the second data.

Description

Neural network operation conversion method, matrix multiplication method based on convolution operation, and intelligent processing unit

The present invention relates to matrix multiplication operations, and more particularly to a method and apparatus for implementing matrix multiplication operations based on convolution operations.

Refer to Figure 1, which shows a schematic diagram of matrix multiplication. The matrix multiplication operator multiplies matrix ML by matrix MR to obtain the matrix product MO (i.e., ML.MR = MO). Matrix ML and matrix MR are the two operands of the matrix multiplication operator. Matrix ML has row_L (=5) columns, and ML has col_L (=1) columns. Matrix MR has row_R (=1) columns, and MR has col_R (=5) columns. Therefore, the matrix product MO is a matrix of row_L × col_R (=5 × 5). The value of each entry (MO(rr,cc)) of the matrix product MO is MO(1,1)=a1b1, MO(1,2)=a1b2, ..., MO(2,1)=a2b1, ..., and so on.

Transform neural networks (TNNs) contain a large number of matrix multiplication operators. However, for electronic devices without matrix multiplication acceleration circuitry, the large number of matrix multiplications is a burden, impacting device performance and user experience.

In view of the shortcomings of the prior art, one purpose of the present invention is to provide a method for neural network operation conversion, a method for performing matrix multiplication operations based on convolution operations, and an intelligent processing unit to improve the shortcomings of the prior art.

One embodiment of the present invention provides a neural network operation conversion method for converting a matrix multiplication operation into a convolution operation, the method comprising: (A) obtaining a first operator and a second operator of the matrix multiplication operation from a storage device; (B) determining third-dimensional information of a third operator based on first-dimensional information of the first operator; (C) determining fourth-dimensional information of a fourth operator based on second-dimensional information of the second operator; (D) setting an offset parameter and a scale parameter; (E) generating a convolution operator based on the third-dimensional information, the fourth-dimensional information, the offset parameter, and the scale parameter; and (F) storing the convolution operator in the storage device. The convolution operator performs the convolution operation on the third operand and the fourth operand, and the result of the convolution operation on the third operand and the fourth operand is substantially equal to the result of the matrix multiplication operation on the first operand and the second operand.

Another embodiment of the present invention provides a method for performing a matrix multiplication operation based on a convolution operation, comprising: (A) reading first data from a first storage device and storing the first data in a second storage device; (B) reading second data from the first storage device and storing the second data in the second storage device; (C) performing the convolution operation on the first data and the second data to obtain a first result; and (D) storing the first result in the first storage device. The first result is equal to a second result obtained by performing the matrix multiplication operation on the first data and the second data.

Another embodiment of the present invention provides an intelligent processing unit. The intelligent processing unit is coupled to a first storage device and includes: a second storage device, a direct memory access circuit, and a calculation circuit. The direct memory access circuit is coupled to the second storage device and is configured to perform the following steps: (A) read a first data from the first storage device and store the first data in the second storage device; and (B) read a second data from the first storage device and store the second data in the second storage device. The calculation circuit is coupled to the second storage device and is configured to perform the following steps: (C) perform a convolution operation on the first data and the second data to obtain a first result. The direct memory access circuit further stores the first result in the first storage device, where the first result is equal to a second result obtained by performing a matrix multiplication operation on the first data and the second data.

The technical means embodied in the embodiments of the present invention can improve at least one of the shortcomings of the prior art. Therefore, the present invention can improve the performance of electronic devices compared to the prior art.

The features, implementation, and effects of the present invention are described in detail below with reference to the accompanying drawings and examples.

cc: Number of columns

ML,MR,MR': Matrix

MO: Matrix Product

rr: Number of columns

IB,KB,KB': Input data

OB: Result of convolution operation

OBV: Table

n:Batch number

h: height

w: width

c: Channel

400: Operation conversion device

410,710:Processor

420,702:Memory

430: Storage device

700: Electronic devices

701: Chip

720: Intelligent Processing Unit

722: Direct Memory Access Circuit

724: Cache memory

726: Computing Circuits

727: Convolution Core

728: Vector Core

305, S310, S320, S330, S340, S350, S360, S370, S380, S390, S610, S810, S820, S830, S840, S850, S860: Steps

Figure 1 is a schematic diagram of matrix multiplication; Figure 2 is a schematic diagram of the present invention implementing matrix multiplication using convolution operations; Figure 3 is a flow chart of one embodiment of the present invention's neural network operation conversion method; Figure 4 is a functional block diagram of one embodiment of the present invention's operation conversion device; Figure 5 is a schematic diagram of another embodiment of the present invention's data rearrangement; Figure 6 is a flow chart of another embodiment of the present invention's neural network operation conversion method; Figure 7 is a functional block diagram of one embodiment of the present invention's electronic device; and Figure 8 is a flow chart of one embodiment of the present invention's method for performing matrix multiplication using convolution operations.

The technical terms used in the following descriptions are based on customary terminology in this technical field. If this manual provides explanations or definitions for certain terms, the explanations or definitions in this manual shall prevail for such terms.

The present invention discloses a method for transforming neural network operations, a method for performing matrix multiplication based on convolution operations, and an intelligent processing unit. Because some components of the intelligent processing unit of the present invention may be known components, details of these components will be omitted in the following description without affecting the full disclosure and feasibility of the present invention. Furthermore, part or all of the process of the method for performing matrix multiplication based on convolution operations of the present invention may be implemented in software and/or firmware and executed by the intelligent processing unit of the present invention or its equivalent. Therefore, the following description of the method invention will focus on the steps rather than the hardware without affecting the full disclosure and feasibility of the method invention.

Since convolution neural networks are one of the most common neural networks, the present invention proposes an electronic device and method for implementing matrix multiplication operations based on convolution operations.

Please refer to Figure 2, which shows a schematic diagram of the present invention implementing matrix multiplication using a convolution operation. Input data IB and input data KB are two operands of a 2D convolution operator. For example, input data IB can be a tensor or a portion of a tensor (e.g., a tile), and input data KB can be weight data (convolution kernel) corresponding to input data IB.

The dimension information of the input data IB is as follows: batch number (n) is 1, height (h) is 1, width (w) is row_L (=5), and channel (c) is col_L (=1).

The dimensionality of the input data KB is as follows: the batch number (n) is col_R (=5) (i.e., the input data KB contains 5 blocks, each corresponding to a convolution kernel), the height (h) is 1, the width (w) is 1, and the channel (c) is row_R.

The dimension information of the convolution result OB is as follows: batch number (n) is 1, depth (d) is 1, height (h) is 1, width (w) is row_L, and channel (c) is col_R.

The OBV table shows the value of each small block of the result of the convolution operation (OB(w,c)). More specifically, OB(1,1)=a1b1, OB(1,2)=a1b2, ..., OB(2,1)=a2b1, ..., and so on.

Refer to Figures 1 and 2 together. Because OB(x,y) = MO(x,y) (x and y are positive integers), the convolution operation can be used to perform matrix multiplication. In some embodiments, the convolution operation may set the bias parameter to 0 and the scale parameter to 1.

Please refer to Figure 3, which is a flow chart of an embodiment of an operation conversion method for a neural network of the present invention. The operation conversion method is used to convert a matrix multiplication operator into a convolution operator. Please refer to Figure 4, which is a functional block diagram of an embodiment of an operation conversion device of the present invention. The operation conversion device 400 includes a processor 410 (for example, a central processing unit, a microprocessor, a microprocessing unit), a memory 420 (which can be a volatile memory (volatile memory), such as a dynamic random access memory (DRAM)) and a storage device 430 (which can be a volatile memory or a non-volatile memory (non-volatile memory), such as a DRAM, a flash memory (flash memory) or a hard drive). Memory 420 stores multiple program codes. Storage device 430 stores multiple instructions for a neural network, some of which involve matrix multiplication. Processor 410 executes the program codes in memory 420 to perform the process shown in Figure 3, converting matrix multiplication (matrix multiplication operator) into convolution (convolution operator). Operation conversion device 400 can be a general-purpose computer or a computer specifically designed to implement the operation conversion method. Figure 3 includes the following steps.

Step S305: The processor 410 reads an instruction of the neural network from the storage device 430.

Step S310: Processor 410 determines whether the instruction is a matrix multiplication operation. If so, processor 410 executes step S320; otherwise, processor 410 executes step S305 to read the next instruction.

Step S320: The processor 410 obtains the first operand and the second operand of the matrix multiplication operation. Taking Figure 1 as an example, in this step, the processor 410 obtains the matrix MR (first operand) and the matrix ML (second operand) from the storage device 430.

Step S330: Processor 410 generates a data reordering instruction to reorder the data of the first operand. Due to the nature of the convolution operation, the data (i.e., elements) of the right matrix (i.e., matrix MR) in the original matrix multiplication (ML.MR) must be reordered. For example, referring to Figures 1 and 2, matrix ML and input data IB are not reordered, but input data KB is the result of reordering matrix MR. To be more specific, the first column of the matrix MR corresponds to the block of the input data KB where the batch number (n) is equal to 1, the second column of the matrix MR corresponds to the block of the input data KB where the batch number (n) is equal to 2, and so on.

For a matrix, rearranging the data is equivalent to performing a transpose operation on the matrix. The details of the matrix transpose operation are well known to those skilled in the art and will not be elaborated on here.

Please refer to Figure 5, which is a schematic diagram of another embodiment of the data rearrangement of the present invention. After data rearrangement, matrix MR' becomes input data KB'. The batch number (n) of input data KB' is col_R (=5), and the channel (c) of input data KB' is row_R (=3). For comparison, if data rearrangement is not performed, the corresponding input data of matrix MR' has a batch number (n) of 1, a width (w) of row_R (=3), and a channel (c) of col_R (=5).

Please continue to refer to Figure 3.

Step S340: The processor 410 determines the dimension information of the input data KB (the third operator, i.e., one of the operators of the convolution operation) based on the dimension information of the matrix MR (the first operator). This step is equal to or equivalent to the reshaping operation of the convolution operation. For example, the dimension information of the matrix MR in Figure 1 is row_R×col_R=1×5, and the dimension information of the input data KB in Figure 2 is (w,h,n,c)=(1,1,col_R,row_R)=(1,1,5,1). The convolution operation will process the input data KB according to the dimension information of the input data KB (the third operator). In other words, the dimension information of the input data KB determines how the convolution operation views or processes the input data KB.

Step S350: Processor 410 determines the dimension information of input data IB (the fourth operator, i.e., another operator of the convolution operation) based on the dimension information of matrix ML (the second operator). Similarly, this step is equivalent to or equivalent to the reshaping operation of the convolution operation. For example, the dimension information of matrix ML in Figure 1 is row_L × col_L = 5 × 1, while the dimension information of input data IB in Figure 2 is (w, h, n, c) = (row_L, 1, 1, col_L) = (5, 1, 1, 1). The convolution operation processes input data IB based on the dimension information of input data IB (the fourth operator). In other words, the dimension information of input data IB determines how the convolution operation views or processes input data IB.

The result of a convolution operation is related to the dimensionality of the input data. When the convolution operation instruction contains the arranged dimensionality information, the result of the convolution operation can be equal to or substantially equal to the result of matrix multiplication as expected.

Step S360: The processor 410 sets an offset parameter and a scale parameter. More specifically, the processor 410 sets the offset parameter to 0 and the scale parameter to 1.

Step S370: The processor 410 generates a convolution operator based on the dimensional information of the input data KB (the third operand), the dimensional information of the input data IB (the fourth operand), the offset parameter, and the scale parameter. This convolution operator is used to replace the aforementioned matrix multiplication operation. More specifically, the input data IB and the input data KB are the operands of the convolution operator, and the result of the convolution operation, OB, is equal to or substantially equal to the result of the matrix multiplication operation (matrix product MO).

Step S380: The processor 410 stores the convolution operator in the storage device 430. In practice, the convolution operator can be stored in the storage device 430 in the form of a convolution operation instruction.

Step S390: The processor 410 determines the dimension information of the convolution result OB based on the dimension information of the matrix multiplication result (i.e., the matrix product MO). This step is equivalent to a reshaping operation of the convolution operation, so that subsequent instructions or operations can treat the convolution result OB as a matrix (2-dimensional data).

In some embodiments, after step S390 is completed, processor 410 continues to execute step S305. In other words, processor 410 scans storage device 430 to check each instruction.

Note that because the convolution operation satisfies the commutative property (i.e., IB * KB = KB * IB), in different embodiments, the input data KB can correspond to the matrix ML, and the input data IB can correspond to the matrix MR (i.e., the input data IB is the data of the matrix MR after data rearrangement).

Please refer to Figure 6, which is a flow chart of another embodiment of the neural network operation conversion method of the present invention. Figure 6 is similar to Figure 3, except that in the embodiment of Figure 6, the processor 410 determines whether the matrix MR has undergone data rearrangement before step S330 (step S610). More specifically, in an overall operation including the matrix multiplication operation (i.e., ML.MR=MO), if the matrix MR has undergone data rearrangement before the matrix multiplication operation, step S330 can be omitted.

Please refer to Figure 7, which is a functional block diagram of an embodiment of the electronic device of the present invention. Electronic device 700 can perform the aforementioned convolution operation. Electronic device 700 includes a chip 701 and a memory 702. Memory 702 is a type of storage device (can be a volatile memory (e.g., DRAM)). Chip 701 can be a chip with specific functions (e.g., an image processing chip), including a processor 710 and an intelligence processing unit (IPU) 720. Processor 710 can be a circuit or electronic component with program execution capability, such as a central processing unit, a microprocessor, a microprocessing unit, a digital signal processor, an application specific integrated circuit (ASIC), or an equivalent circuit thereof. In some cases, processor 710 collaborates with intelligent processing unit 720 to implement the functions of chip 701. More specifically, processor 710 transmits instructions (e.g., instructions related to convolution operations or vector operations) to intelligent processing unit 720, and intelligent processing unit 720 executes these instructions.

The intelligent processing unit 720 includes a direct memory access (DMA) circuit 722, a cache memory 724 (a type of storage device), and a computation circuit 726. The computation circuit 726 includes a convolution core 727 and a vector core 728. The convolution core 727 is used to perform convolution operations, and the vector core 728 is used to perform vector operations.

Please refer to Figure 8, which is a flowchart of one embodiment of a method for performing matrix multiplication based on convolution operations according to the present invention. The process of Figure 8 can be executed by the electronic device 700 (more specifically, the chip 701) of Figure 7 and includes the following steps. It is assumed that the following convolution operation corresponds to the matrix multiplication operation of ML.MR (see Figure 1). Please note that, unlike convolution operations, matrix multiplication operations do not satisfy the commutative property (i.e., ML.MR ≠ MR.ML).

Step S810: Processor 710 determines whether a data reordering instruction is present. If so (i.e., step S330 in Figure 3 or Figure 6 has been executed by computation conversion device 400), processor 710 controls intelligent processing unit 720 to execute step S820. Otherwise, processor 710 controls intelligent processing unit 720 to execute steps S830 through S860. If no data reordering instruction is present, this indicates that the data for the matrix multiplication operation has been previously reordered. In one embodiment, when the data for the matrix multiplication operation, such as matrix MR, is a constant value, the data is pre-reordered, and the reordered data is stored in a storage device.

Step S820: The direct memory access circuit 722 reads first original data (e.g., matrix MR) from a first storage device (e.g., memory 702), performs a data rearrangement operation on the first original data to convert the first original data into first data (e.g., input data KB, i.e., weight data for a subsequent convolution operation), and then stores the first data in the first storage device. More specifically, the direct memory access circuit 722 performs the data rearrangement operation on the first original data by writing and reading the first original data to and from cache memory 724. This data rearrangement operation is equivalent to a transpose operation in matrix operations. Then, the direct memory access circuit 722 writes the first data back to the first storage device.

Step S830: Direct memory access circuit 722 reads the first data (e.g., input data KB) from the first storage device and stores the first data in the second storage device (e.g., cache 724). Note that when there is no data reordering instruction (i.e., the result of step S810 is no), the first original data is already arranged in the format of the input data for the convolution operation (e.g., the format of input data KB). Note that step S830 does not reorder the first data.

Step S840: The direct memory access circuit 722 reads the second data (e.g., input data IB, essentially equivalent to matrix ML) from the first storage device and stores the second data in the second storage device. Note that step S840 does not rearrange the second data.

Step S850: The calculation circuit 726 (more specifically, the convolution core 727) performs the convolution operation on the first data and the second data to obtain a result (e.g., the convolution result OB), and stores the result in a second storage device. In some embodiments, the offset parameter and scale parameter of the convolution operation are 0 and 1, respectively. In some embodiments, the offset parameter and scale parameter are pre-stored in the memory 702, and the intelligent processing unit 720 reads these parameters from the memory 702 and sets them accordingly.

Step S860: The direct memory access circuit 722 stores the result in the first storage device for the chip 701 to perform other subsequent operations.

As discussed above, because the instructions executed by chip 701 have been pre-processed, the result of the convolution operation executed by chip 701 is equivalent to the result of the matrix multiplication (for example, see Figures 1 and 2 , the result of the convolution operation OB is equal to or substantially equal to the matrix product MO).

In summary, even if an electronic device does not implement a matrix multiplication acceleration circuit, the present invention can still be used to accelerate matrix multiplication operations, thereby improving performance and enhancing the user experience.

Although the embodiments of the present invention have been described above, these embodiments are not intended to limit the present invention. Those skilled in the art may modify the technical features of the present invention based on the explicit or implicit content of the present invention. All such modifications may fall within the scope of the patent protection sought for the present invention. In other words, the scope of patent protection for the present invention shall be determined by the scope of the patent application set forth in this specification.

S305, S310, S320, S330, S340, S350, S360, S370, S380, S390: Step

Claims (5)

A neural network operation conversion method is performed by an operation conversion device to convert a matrix multiplication operation into a convolution operation. The operation conversion device includes a processor and a storage device coupled to each other. The method includes: (A) the processor obtains a first operand and a second operand of the matrix multiplication operation from the storage device; (B) the processor determines third-dimensional information of a third operand based on first-dimensional information of the first operand; (C) the processor determines fourth-dimensional information of a fourth operand based on second-dimensional information of the second operand; (D) the processor sets an offset parameter and a scale parameter; (E) the processor generates a convolution operator based on the third-dimensional information, the fourth-dimensional information, the offset parameter, and the scale parameter; and (F) the processor stores the convolution operator in the storage device; wherein the convolution operator performs the convolution operation on the third operand and the fourth operand, and the result of the convolution operation of the third operand and the fourth operand is substantially equal to the result of the matrix multiplication operation of the first operand and the second operand; wherein the matrix multiplication operation generates a first result, and the convolution operator generates a second result. The method further includes: (G) determining sixth dimensional information of the second result based on fifth dimensional information of the first result. The method of claim 1 further comprises: the processor generating a data arrangement instruction before the step (B), wherein the data arrangement instruction is used to rearrange the first operand. The method of claim 2, wherein the first operand is a matrix, and the data arrangement instruction is equivalent to performing a transpose operation on the matrix. The method of claim 2, wherein the third operand is a weight data of the convolution operation. The method of claim 1, wherein the first operand is a matrix A, the second operand is a matrix B, and the matrix multiplication operation is to calculate B·A, the method further comprising: the processor generating a data arrangement instruction before the step (B), the data arrangement instruction being used to rearrange the data of the matrix A.