HK1245462B - Prefetching weights for use in a neural network processor - Google Patents

Description

Prefetching weights for neural network processors

Technical Field

This specification is about computing neural network inference on hardware.

Background Art

A neural network is a machine learning model that uses one or more layers to generate an output, such as a classification, based on input. In addition to the output layer, some neural networks include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network, the next hidden layer or output layer. Each layer of the network generates an output based on the current values of its corresponding set of parameters.

Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has an associated set of kernels. Each kernel includes values established by a user-created neural network model. In some implementations, the kernel recognizes specific image contours, shapes, or colors. The kernel can be represented as a matrix structure of weight inputs. Each convolutional layer can also process a set of activation inputs. The set of activation inputs can also be represented as a matrix structure.

Some existing systems perform calculations for a given convolutional layer in software. For example, the software can apply each kernel for the layer to the set of activation inputs. That is, for each kernel, the software can overlay the kernel that can be represented multidimensionally on a first portion of the activation inputs that can be represented multidimensionally. The software can then calculate the dot product from the overlapping elements. The dot product can correspond to a single activation input, such as the activation input element with the upper left position in the overlapping multidimensional space. For example, using a sliding window, the software can then shift the kernel to cover the second portion of the activation input and calculate another dot product corresponding to another activation input. The software can repeatedly perform this process until each activation input has a corresponding dot product. In some implementations, the dot product is input to an activation function that generates an activation value. The activation values can be combined, for example, merged, before being sent to the next layer of the neural network.

One type of convolution calculation requires numerous matrix multiplications in a large space. Processors can perform these matrix multiplications through exhaustive methods. For example, although this is computationally and time-intensive, the processor can repeatedly compute the sums and products required for convolution. However, this architecture limits the extent of parallelism within the processor.

Summary of the Invention

In general, this specification describes a dedicated hardware circuit for computing neural network inference.

In general, one innovative aspect of the subject matter described herein can be embodied in a circuit for performing neural network computations on a neural network comprising a plurality of layers, the circuit comprising: a systolic array comprising a plurality of cells; a weight extraction unit configured, for each of a plurality of neural network layers, to: send, for the neural network layer, a plurality of weight inputs to cells along a first dimension of the systolic array; and a plurality of weight sequencer units, each weight sequencer unit coupled to a different cell along the first dimension of the systolic array, the plurality of weight sequencer units configured, for each of the plurality of neural network layers, to: transfer, for the neural network layer, the plurality of weight inputs to cells along a second dimension of the systolic array in a plurality of clock cycles, wherein respective weight inputs are stored within respective cells along the second dimension, and wherein respective cells are configured to compute a product of an activation input and a respective weight input using a multiplication circuit.

Implementations may include one or more of the following: a value sequencer unit configured to, for each of the plurality of neural network layers, send a plurality of activation inputs to cells along the second dimension of the systolic array of the neural network layer. The first dimension of the systolic array corresponds to rows of the systolic array, and wherein the second dimension of the systolic array corresponds to columns of the systolic array. Each cell is configured to pass a weight control signal to an adjacent cell, the weight control signal causing circuitry in the adjacent cell to transfer or load weight inputs for the adjacent cell. a weight path register configured to store the weight input transferred to the cell; a weight register coupled to the weight path register; a weight control register configured to determine whether the weight input is stored in the weight register; an activation register configured to store the activation input and configured to send the activation input to another activation register in a first adjacent cell along the first dimension; a multiplication circuit coupled to the weight register and the activation register, wherein the multiplication circuit is configured to output a product of the weight input and the activation input; an addition circuit coupled to the multiplication circuit and configured to receive the product and a first local sum from a second adjacent cell along the second dimension, wherein the addition circuit is configured to output a second local sum of the product and the first local sum; and a local sum register coupled to the addition circuit and configured to store the second local sum, wherein the local sum register is configured to send the second local sum to another addition circuit in a third adjacent cell along the second dimension. Each weight sequencer unit includes: a pause counter corresponding to the weight control register in a corresponding cell coupled to the weight sequencer unit; and a decrement circuit configured to decrement the input to the weight sequencer unit to generate a decremented output and send the decremented output to the pause counter. The value in each pause counter is the same, and each weight sequencer unit is configured to load the corresponding weight input to a corresponding different cell of the systolic array, wherein the loading includes sending the weight input to the multiplication circuit. The value in each pause counter reaches a predetermined value, causing the plurality of weight sequencer units to pause the transfer of the plurality of weight inputs along the second dimension. The systolic array is configured to generate, for each of the plurality of neural network layers, an accumulated output for the neural network layer from each product.

Certain embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. Prefetching weights enables a neural network processor to perform computations more efficiently. The processor can coordinate loading weight inputs into a systolic array using a weight fetch unit and a weight sequencer unit, thereby eliminating the need to couple an external memory unit to each cell in the systolic array. The processor can pause, i.e., "freeze," the transfer of weight inputs to synchronize the execution of multiple convolution computations.

The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a flow chart of an exemplary method for performing computations on a given layer of a neural network.

FIG2 illustrates an exemplary neural network processing system.

FIG3 shows an example architecture including a matrix computation unit.

FIG4 illustrates an exemplary architecture of a unit cell within a systolic array.

FIG5 shows an exemplary matrix structure having a spatial dimension and a feature dimension.

FIG6 shows an exemplary diagram of how a core matrix structure is sent to a systolic array.

FIG. 7 shows an exemplary diagram of the weight inputs inside a cell after three clock cycles.

FIG. 8 is an exemplary illustration of how a control signal causes an activation input to be transferred or loaded.

Like reference numerals and numbers refer to like elements.

DETAILED DESCRIPTION

Neural networks with multiple layers can be used to compute inferences. For example, given an input, a neural network can compute an inference about that input. The neural network computes the inference by processing the input through the various layers of the neural network. Specifically, the layers of a neural network are arranged in a sequence, each with a corresponding set of weights. Each layer receives an input and processes it according to the set of weights for that layer to generate an output.

Thus, to compute inferences from received inputs, a neural network receives inputs and processes the inputs sequentially through various neural network layers to produce inferences, with the outputs from one neural network layer being provided as inputs to the next neural network layer. Data input to a neural network layer, e.g., either inputs to the neural network or outputs to the neural network layer of a layer sequentially below it, can be referred to as activation inputs to the layer.

In some implementations, the layers of a neural network are arranged in a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be arranged so that the output of a layer can be sent back as input to the previous layer.

1 is a flow chart of an exemplary process 100 for performing computations on a given layer of a neural network using dedicated hardware circuitry. For convenience, method 100 will be described with respect to a system having one or more circuits for performing method 100. Method 100 can be performed on various layers of a neural network to compute inferences from received inputs.

The system receives a set of weight inputs for a given layer (step 102) and a set of activation inputs (step 104). The set of weight inputs and the set of activation inputs can be received from a dynamic memory of a dedicated hardware circuit and a unified buffer, respectively. In some implementations, the set of weight inputs and the set of activation inputs are both received from a unified buffer.

The system uses a matrix multiplication unit of a dedicated hardware circuit to generate a cumulative value from the weight inputs and the activation inputs (step 106). In some implementations, the cumulative value is the dot product of the group of weight inputs and the group of activation inputs. That is, for a set of weights, the system can multiply each weight input with each activation input and add the products together to form the cumulative value. The system can then calculate the dot product of other groups of weights with other groups of activation inputs.

The system can generate the layer output from the accumulated values using a vector computation unit of a dedicated hardware circuit (step 108). In some implementations, the vector computation unit applies an activation function to the accumulated values. The output of the layer can be stored in a unified buffer and used as input to subsequent layers in the neural network, or can be used to determine inferences. When the received input has been processed through the various layers of the neural network to generate inferences for the received input, the system has completed processing of the neural network.

FIG2 illustrates an exemplary application-specific integrated circuit 200 for performing neural network computations. System 200 includes a host interface 202. Host interface 202 is capable of receiving instructions including parameters for neural network computations. The parameters may include at least one or more of the following: how many layers should be processed; a corresponding set of weight inputs for each of the layers; an initial set of activation inputs, i.e., inputs to the neural network from which inference is to be computed; corresponding input and output sizes for each layer; a stride value for the neural network computation; and the type of layer to be processed, e.g., a convolutional layer or a fully connected layer.

The host interface 202 can send instructions to the sequencer 206, which converts the instructions into low-level control signals that control the circuit to perform neural network calculations. In some implementations, the control signals regulate the flow of data in the circuit, such as how the groups of weight inputs and activation inputs flow through the circuit. The sequencer 206 can send control signals to the unified buffer 208, the matrix calculation unit 212, and the vector calculation unit 214. In some implementations, the sequencer 206 also sends control signals to the direct memory access engine 204 and the dynamic memory 210. In some implementations, the sequencer 206 is a processor that generates a clock signal. The sequencer 206 can use the timing of the clock signal to send control signals to various components in the circuit 200 at the appropriate time. In some other implementations, the clock signal from the external processor passes through the host interface 202.

The host interface 202 can send the set of weight inputs and the initial set of activation inputs to the direct memory access engine 204. The direct memory access engine 204 can store the set of activation inputs in a unified buffer 208. In some implementations, the direct memory access stores the set of weights in a dynamic memory 210, which can be a storage unit. In some implementations, the dynamic memory is external to the circuit.

The unified buffer 208 is a storage buffer that can be used to store groups of active inputs from the direct memory access engine 204 and the outputs of the vector computation unit 214. The direct memory access engine 204 can also read the outputs of the vector computation unit 214 from the unified buffer 208.

The dynamic memory 210 and the unified buffer 208 can send the groups of weight inputs and the groups of activation inputs, respectively, to the matrix calculation unit 212. In some implementations, the matrix calculation unit 212 is a two-dimensional systolic array. The matrix calculation unit 212 can also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the matrix calculation unit 212 is a general-purpose matrix processor.

The matrix calculation unit 212 can process the weight inputs and activation inputs and provide a vector of outputs to the vector calculation unit 214. In some implementations, the matrix calculation unit sends the vector of outputs to the unified buffer 208, and the unified buffer 208 sends the vector of outputs to the vector calculation unit 214. The vector calculation unit can process the vector of outputs and store the processed vector of outputs in the unified buffer 208. For example, the vector calculation unit 214 can apply a nonlinear function to the output of the matrix calculation unit, such as a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 214 generates normalized values, merged values, or both. The processed vector of outputs can be used as activation inputs to the matrix calculation unit 212, for example, for use in subsequent layers in a neural network. The matrix calculation unit 212 will be described in detail below with reference to Figures 3 and 4.

FIG3 illustrates an exemplary architecture 300 including a matrix computation unit. The matrix computation unit is a two-dimensional systolic array 306. The array 306 includes a plurality of cells 304. In some implementations, a first dimension 320 of the systolic array 306 corresponds to columns of cells, and a second dimension 322 of the systolic array 306 corresponds to rows of cells. A systolic array can have more rows than columns, more columns than rows, or an equal number of rows and columns.

In the illustrated example, the value loader 302 sends activation inputs to the rows of the array 306, and the weight extraction interface 308 sends weight inputs to the columns of the array 306. However, in some other implementations, the activation inputs are passed to the columns of the array 306, and the weight inputs are passed to the rows of the array 306.

Value loader 302 can receive activation input from a unified buffer (e.g., unified buffer 208 of FIG. 2 ). Each value loader can send a corresponding activation input to a different leftmost cell of array 306. The leftmost cell can be a cell along the leftmost column of array 306. For example, value loader 312 can send activation input to cell 314. A value loader can also send activation input to an adjacent value loader, and the activation input can be used at another leftmost cell of array 306. This allows the activation input to be transferred for use in another specific cell of array 306.

The weight extraction interface 308 can receive weight inputs from a storage unit (e.g., dynamic memory 210 of FIG. 2 ). The weight extraction interface 308 can send corresponding weight inputs to different top-most cells of the array 306. The top-most cells can be cells along the top-most row of the array 306. For example, the weight extraction interface 308 can send weight inputs to cells 314 and 316.

In some implementations, a host interface, such as host interface 202 of FIG2 , transfers activation inputs along one dimension, such as rightward, across array 306, while simultaneously transferring weight inputs along another dimension, such as downward, across array 306. For example, during one clock cycle, an activation input at cell 314 can be transferred to an activation register in cell 316 to the right of cell 314. Similarly, a weight input at cell 316 can be transferred to a weight register at cell 318 below cell 314.

In each clock cycle, each cell can process a given weight input and a given activation input to generate an accumulated output. The accumulated output can also be passed to adjacent cells along the same dimension as the given weight input. The individual cells are further described below with reference to Figure 4.

In some implementations, weights and activations are transferred across more than one cell during a given clock cycle to transition from one convolutional computation to another.

The accumulated output can be passed along the same column as the weight input, for example, toward the bottom of the column of array 306. In some implementations, at the bottom of each column, array 306 can include an accumulator unit 310 that stores and accumulates the individual accumulated outputs from each column when performing calculations on a layer with more weight inputs than columns or a layer with more activation inputs than rows. In some implementations, each accumulator unit stores multiple parallel accumulations. This will be further described below with reference to Figure 6. The accumulator unit 310 can accumulate the individual accumulated outputs to generate a final accumulated value. The final accumulated value can be passed to the vector calculation unit. In some other implementations, when processing a layer with fewer weight inputs than columns or a layer with fewer activation inputs than rows, the accumulator unit 310 passes the accumulated value to the vector calculation unit without performing any accumulation.

As activation inputs and weight inputs flow through a circuit, the circuit can "freeze" or pause the flow of a set of weight inputs to accurately calculate the product value. That is, the circuit can pause a set of weight inputs so that a specific set of weight inputs can be applied to a specific set of activation inputs.

In some implementations, the weight sequencer 324 configures whether the weight input is moved to an adjacent cell. The weight sequencer 326 can receive a control value from a host (e.g., the host interface 202 of Figure 2) or from an external processor. Each weight sequencer can pass the control value to the corresponding cell in the array 306. In particular, the control value can be stored in a weight control register in the cell, such as the weight control register 414 of Figure 4. The control value can determine whether the weight input is transferred or loaded along the dimension of the array, which will be described below with reference to Figure 8. The weight sequencer can also send the control value to an adjacent weight sequencer, which can adjust the transfer or loading of the corresponding weight input for the corresponding cell.

In some implementations, the control value is represented as an integer. Each weight sequencer may include a pause count register that stores the integer. The weight sequencer may also decrement the integer before storing the control value in the pause count register. After storing the control value in the pause count register, the weight sequencer can send the integer to an adjacent weight sequencer and send the integer to the corresponding cell. For example, each weight sequencer may have a decrement circuit configured to generate a decrementing integer from the control value. The decrementing integer may be stored in the pause count register. The stored control value can be used to coordinate simultaneous transfer pauses across an entire column of the array, which will be further described below with reference to Figure 8.

In some implementations, pausing weights in a circuit enables developers to debug the circuit.

Other approaches to pausing weights are possible. For example, instead of passing the value in a pause count register to adjacent pause count registers, control values can be passed using a tree. That is, at a given cell, the signal can be passed to all adjacent cells, not just one, allowing the signal to propagate quickly through the systolic array.

FIG. 4 illustrates an exemplary architecture 400 of a cell within a systolic array, such as systolic array 306 in FIG. 3 .

The cell may include an activation register 406 that stores an activation input. The activation register can receive activation inputs from a left neighboring cell, i.e., a neighboring cell to the left of a given cell, or from a unified buffer, depending on the position of the cell within the systolic array. The cell may include a weight register 402 that stores weight inputs. Depending on the position of the cell within the systolic array, the weight inputs can be passed from a top neighboring cell or from a weight extraction interface. The cell may also include a sum register 404. The sum register 404 can store an accumulated value from a top neighboring cell. A multiplication circuit 408 can be used to multiply the weight inputs from the weight register 402 with the activation inputs from the activation register 406. The multiplication circuit 408 can output the product to an addition circuit 410.

The adding circuit can sum the product with the accumulated value from the summing register 404 to generate a new accumulated value. The adding circuit 410 can then send the new accumulated value to another summing register located in the bottom adjacent cell. The new accumulated value can be used as an operand for the summation in the bottom adjacent cell.

In some implementations, the cell also includes a general control register. The control register can store a control signal that determines whether the cell should transfer the weight input or the activation input or the activation input to an adjacent cell. In some implementations, transferring the weight input or the activation input requires more than one clock cycle. The control signal can also determine whether to pass the activation input or the weight input to the multiplication circuit 408, or whether to determine whether the multiplication circuit 408 operates on the activation input or the weight input. The control signal can also be passed to more than one adjacent cell, for example, using a wire.

In some implementations, the weights are pre-transferred to the weight path register 412. The weight path register 412 can receive weight inputs, for example, from the top neighboring cell, and pass the weight inputs to the weight register 402 based on a control signal. The weight register 402 can statically store the weight inputs so that when activation inputs are passed to the cell, for example, via the activation register 406, over the course of multiple clock cycles, the weight inputs remain in the cell and are not passed to neighboring cells. Thus, the weight inputs can be applied to multiple activation inputs using the multiplication circuit 408, and the corresponding accumulated values can be passed to neighboring cells.

In some implementations, the weight control register 414 controls whether the weight input is stored in the weight register 402. For example, if the weight control register 414 stores a control value of 0, the weight register 402 can store the weight input sent by the weight path register 412. In some implementations, storing the weight input in the weight register 402 is referred to as loading the weight input. Once the weight input is loaded, the weight input can be sent to the multiplication circuit 408 for processing. If the weight control register 414 stores a non-zero control value, the weight register 402 can ignore the weight input sent by the weight path register 412. The control value stored in the weight control register 414 can be passed to more than one adjacent cell, for example, for a given cell, the control value can be sent to the weight control register in the cell to the right of the given cell.

Cells can also transfer weight inputs and activation inputs to adjacent cells. For example, weight path register 412 can send a weight input to another weight path register in the adjacent cell at the bottom. Activation register 406 can send an activation input to another activation register in the adjacent cell on the right. Thus, both the weight input and the activation input can be reused by other cells in the array in subsequent clock cycles.

FIG5 shows an exemplary matrix structure 500 having a spatial dimension and a feature dimension. Matrix structure 500 can represent a set of activation inputs or a set of weight inputs. In this specification, a matrix structure for a set of activation inputs is referred to as an activation matrix structure, and in this specification, a matrix structure for a set of weight inputs is referred to as a core matrix structure. Matrix structure 500 has three dimensions: two spatial dimensions and one feature dimension.

In some implementations, the spatial dimensions correspond to the space or location of a set of activation inputs. For example, if the neural network is processing an image having two dimensions, the matrix structure can have two spatial dimensions corresponding to the spatial coordinates of the image, i.e., XY coordinates.

The feature dimensions correspond to features of the activation inputs. Each feature dimension can have a depth level; for example, the matrix structure 500 has depth levels 502, 504, and 506. As an example, if the matrix structure 500 represents a 3×3×3 image sent as a set of activation inputs to the first layer, the X and Y dimensions of the image (3×3) can be spatial dimensions, while the Z dimension (3) can be a feature dimension corresponding to the R, G, and B values. That is, depth level 502 can correspond to features of nine "1" activation inputs, such as red values, depth level 504 can correspond to features of nine "2" activation inputs, such as green values, and depth level 506 can correspond to features of nine "3" activation inputs, such as blue values.

Although only three depth levels of feature dimensions are illustrated in the example of Figure 5 , a given feature dimension can have a large number, such as hundreds of feature dimensions. Similarly, although only one feature dimension is shown, a given matrix structure can have multiple feature dimensions.

In order to perform calculations on the convolution layer using the matrix structure 500, the system needs to convert the convolution calculation into a two-dimensional matrix multiplication.

FIG6 shows an exemplary diagram of how the matrix structure 500 of FIG5 is processed by the systolic array 606 at a given convolutional layer. The matrix structure 600 can be a set of activation inputs. Generally, the neural network processor can send activation inputs (e.g., elements in the matrix structure 600) and weight inputs (e.g., kernels A-D 610) to the rows and columns of the array, respectively. The activation inputs and weight inputs can be transferred to the right and bottom of the systolic array, respectively, and must arrive at specific locations, such as specific registers at specific cells. Once these inputs are determined to be in place, for example by examining control signals, the processor can perform calculations using the inputs stored in the cells to generate the output of the given layer.

As described above, the neural network processor "flattens" the matrix structure 600 before sending a portion of the structure 600 to a row of the systolic array. That is, the neural network processor can separate the depth layers 602 of the matrix structure 600, such as depth layers 602, 604, and 606 of FIG. 6, and send each depth layer to a different cell. In some implementations, each depth layer is sent to a cell on a different row of the systolic array 606. For example, the processor can send activation inputs from a first depth layer, such as a matrix of nine "1" activation inputs, to the leftmost cell at the first row of the systolic array 606, send a second depth layer, such as a matrix of nine "2" activation inputs, to the leftmost cell at the second row, send a third depth layer, such as a matrix of nine "3" activation inputs, to the leftmost cell at the third row, and so on.

A given layer may have multiple cores, such as cores A-D 610. Cores A-D 610 may have a matrix structure of dimensions 3×3×10. The processor may send each core matrix structure to a cell at a different column of the systolic array 606. For example, core A may be sent to the top cell in the first column, core B may be sent to the top cell in the second column, and so on.

When the matrix structure is sent to a cell, the first element of the matrix can be stored in the cell during one clock cycle. In the next clock cycle, the next element can be stored in the cell. The stored first element can be transferred to an adjacent cell, as described above with reference to FIG4. The transfer of inputs can continue until all elements of the matrix structure are stored in the systolic array 606. Both the activation input and the weight input can be transferred after one or more clock cycles through each cell. The transfer of inputs in the systolic array will be further described below with reference to FIG7.

FIG7 shows an exemplary diagram 700 of weight inputs within a cell of an exemplary 3×3 systolic array after three clock cycles. Each cell can store weight inputs and activation inputs, as described above with reference to FIG5 . The weight inputs can be sent to different columns of the systolic array for convolution computations, as described above with reference to FIG7 . As an example, the system sends a first kernel matrix structure with weight inputs of 1, 2, and 4 to the first column of the systolic array. The system sends a second kernel structure with weight inputs of 3, 5, and 7 to the second column. The system sends a third kernel structure with weights of 6, 8, and 10 to the third column. After each clock cycle, the weight inputs can be shifted in one dimension, such as from top to bottom, while the activation inputs can be shifted in another dimension (not shown), such as from left to right.

The weight inputs can be stored in cells in an interleaved manner. That is, the state of the systolic array after the first clock cycle 702 shows a "1" in the top left cell. The "1" represents the weight input "1" stored in the cell. In the next clock cycle 704, the "1" is transferred to the cell below the top left cell, and another weight input "2" from the core is stored in the top left cell, and the weight input "3" is stored at the top cell of the second column.

In the third clock cycle 706, the weights are shifted again. In the first column, the bottom cell stores the weight input "1," the weight input "2" is stored in the location where the weight input "1" was stored in the previous cycle, and the weight input "4" is stored in the top left cell. Similarly, in the second column, "3" is shifted downward, and the weight input "5" is stored in the top middle cell. In the third column, the weight input "6" is stored in the top right cell.

In some implementations, a control signal for the weight input that determines whether the weight input should be transferred is also transferred along with the weight input.

Activation inputs can be shifted in other dimensions in a similar manner, for example, from left to right.

Once the activation inputs and weight inputs are in place, the processor can perform a convolution calculation, for example, by using the multiplication and addition circuits in the cell to generate a set of accumulated values to be used in the vector computation unit.

Although the system is described with weight inputs sent to the columns of the array and activation inputs sent to the rows of the array, in some implementations, weight inputs are sent to the rows of the array and activation inputs are sent to the columns of the array.

FIG8 is an exemplary illustration of how a control value can cause a weight input to be transferred or loaded. The control value 806 can be sent by the host and can be stored by the weight sequencer, as described above with reference to FIG3. The values in the figure represent the control values stored in the weight sequencers 808-814 corresponding to rows 1-4 804 of the systolic array, respectively, on a per clock cycle 802 basis.

In some implementations, if the control value in a given weight sequencer is non-zero, the weight input in the corresponding cell of the systolic array will be transferred to the adjacent cell. If the control value in a given weight sequencer is zero, the weight input can be loaded into the corresponding cell and used to calculate the product with the activation input in the cell.

As an example, the host can determine that four weight inputs should be transferred before loading. In clock cycle 0, the host can send a control value of 5 to the weight sequencer 808, i.e., the weight sequencer corresponding to row 1. The weight sequencer 808 includes a decrement circuit that takes one clock cycle to output a control value of 4 based on the control value 5. Therefore, the control value of 4 is stored in the weight sequencer 808 in the subsequent clock cycle, i.e., clock cycle 1.

In clock cycle 1, the host sends a control value of 4 to the weight sequencer 808. Thus, in clock cycle 2, the weight sequencer 808 stores a control value of 3 using, for example, a decrement circuit. In clock cycle 1, the weight sequencer 808 can send the control value of 4 to the weight sequencer 810. Thus, in clock cycle 2, after processing the control value of 4 using the decrement circuit of the weight sequencer 810, the weight sequencer 810 can store the control value of 3.

Similarly, the host can send control values of 3, 2, and 1 in clock cycles 2, 3, and 4, respectively. Because the decrement circuitry in each weight sequencer 808-814 introduces a delay when decrementing the control value, decrementing the control value 806 in each clock cycle can ultimately cause each weight sequencer to store the same control value, i.e., a control value of 1 in clock cycle 4 and a control value of 0 in clock cycle 5.

In some implementations, when each weight sequencer outputs a control value of 0, the systolic array pauses the transfer of the weight input and loads the weight input in each cell. That is, by loading the weight input, the systolic array enables the weight input to be used as an operand in a dot product calculation, thereby starting processing the layer in the neural network.

In some implementations, after the calculations are completed, to restart the transfer of weights, the host changes the control value to a non-zero value, such as sending a control value of 5 during clock cycle 7. The transfer process can be repeated as described above with reference to clock cycle 0.

In some implementations, the control value starts with another offset, such as 1.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital circuitry, in tangibly implemented computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof.

The term "data processing equipment" encompasses all types of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The equipment may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may also include code that creates an execution environment for the associated computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

A computer program (which may also be referred to as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages or purely functional or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, such as files that store one or more modules, subroutines, or code portions. A computer program may be deployed to execute on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

For example, the computer that is suitable for executing a computer program includes and can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or based on a central processing unit of any other type. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or from both. The basic element of a computer is a central processing unit for executing or running instructions, and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks or optical disks, or is operationally coupled to the large-capacity storage device to receive data from the large-capacity storage device or data are transferred to the large-capacity storage device, or both. However, a computer does not necessarily have such a device. Moreover, a computer can be embedded in another device, such as, to name a few, a mobile phone, a personal digital assistant (PDA), a mobile audio player or a mobile video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

To communicate with a user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) monitor or an LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, with which the user can communicate input to the computer. Other types of devices can also be used to communicate with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including voice input, speech input, or tactile input. In addition, the computer can interact with the user by sending documents to or receiving documents from a device used by the user; and by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server; or in a computing system that includes a middleware component, such as an application server; or in a computing system that includes a front-end component, such as a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification; or in a computing system that includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

Computing systems can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the content that may be claimed for protection, but should be interpreted as a description of the specific features of the specific embodiments of a particular invention. Some features described in the context of each embodiment in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in the form of any suitable sub-combination. In addition, although the features may be described above as working in some combinations, and even initially claimed to be protected in this way, one or more features in the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a change in the sub-combination.

Similarly, although the operations are shown in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order shown or in sequence, or that all of the operations shown be performed, in order to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments. Instead, it should be understood that the program components and systems described can generally be integrated together in a single software product, or packaged into multiple software products.

Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the specific order or sequence shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

Claims (20)

1. A circuit for performing neural network computation on a neural network comprising multiple layers, the circuit comprising: The hardware matrix calculation unit includes circuitry for a pulsating array, the pulsating array comprising a plurality of cells, each of the plurality of cells including a weight register disposed within the cell for storing weight inputs received from a source outside the cell. The hardware circuitry for the weight extraction unit is configured, for each of the plurality of neural network layers, as follows: For this neural network layer, multiple weight inputs are sent to cells along the first dimension of the systolic array; and Hardware circuitry for multiple weight sequencer units disposed outside each of the plurality of cells, each weight sequencer unit being coupled to a different cell along the first dimension of the systolic array, wherein for each of the plurality of neural network layers, each weight sequencer unit is configured as follows: Control values are provided for storage in control registers set within the different cells coupled to the weight sequencer unit, the control values being used to transfer the plurality of weight inputs to cells along a second dimension of the systolic array for the neural network layer over multiple clock cycles, wherein each weight input is stored in a corresponding cell using the weight registers and along the second dimension, and wherein each of the cells is configured to use multiplication circuitry to compute the product of the activation input and the corresponding weight input. 2. The circuit according to claim 1 further includes: The value sequencer unit is configured, for each of the plurality of neural network layers, to send a plurality of activation inputs to cells along the second dimension of the systolic array for that neural network layer. 3. The circuit of claim 1, wherein the first dimension of the pulsating array corresponds to a row of the pulsating array, and wherein the second dimension of the pulsating array corresponds to a column of the pulsating array. 4. The circuit of claim 1, wherein each cell is configured to pass a weight control signal to an adjacent cell, the weight control signal causing the circuit in the adjacent cell to transfer or load weight input for the adjacent cell. 5. The circuit of claim 1, wherein each cell includes hardware circuitry for: A weight path register is located within the cell and coupled to the weight register, the weight path register being configured to store weight inputs transferred to the cell; A weight control register is set within the cell and is used to store at least the control value provided by the weight sequencer or the weight control signal transmitted from the adjacent cell. The weight control register is configured to determine whether to store the weight input in the weight register. An activation register is set within the cell and configured to store activation input, and configured to send the activation input to another activation register in a first adjacent cell along the first dimension; A multiplication circuit, wherein the multiplication circuit is disposed within the cell and coupled to the weight register and the activation register, wherein the multiplication circuit is configured to output the product of the weight input and the activation input; An addition circuit, disposed within the cell and coupled to the multiplication circuit, and configured to receive the product and a first partial sum from a second adjacent cell along the second dimension, wherein the addition circuit is configured to output a second partial sum of the product and the first partial sum; and A local sum register, which is located within the cell and coupled to the adder circuit, and configured to store the second local sum, the local sum register being configured to send the second local sum to another adder circuit in a third adjacent cell along the second dimension. 6. The circuit of claim 5, wherein each weight register unit comprises: A pause counter, the pause counter corresponding to the weight control register in the corresponding cell coupled to the weight sequencer unit; and A decrementing circuit is configured to decrement the input to the weight sequencer unit to generate a decrementing output, and to send the decrementing output to the pause counter. 7. The circuit of claim 6, wherein the values in each pause counter are identical, and each weight sequencer unit is configured to load a corresponding weight input into a corresponding different cell of the pulsation array, wherein the loading includes sending the weight input to the multiplication circuit. 8. The circuit of claim 6, wherein the values in each pause counter are different, and each weight sequencer unit is configured to transfer the corresponding weight input to an adjacent weight sequencer unit along the second dimension. 9. The circuit of claim 6, wherein the values in each pause counter reach predetermined values to cause the plurality of weight sequencer units to pause the transfer of the plurality of weight inputs along the second dimension. 10. The circuit of claim 1, wherein the pulsation array is configured to: for each of the plurality of neural network layers, generate an accumulated output for the neural network layer from the respective product. 11. A method for performing neural network computation on a neural network comprising multiple layers, the method comprising, for each of the multiple neural network layers: At the weight extraction unit, multiple weight inputs of cells along the first dimension of the systolic array, which includes circuitry for the systolic array, are sent to a hardware matrix calculation unit. The plurality of weight inputs are stored in a corresponding weight register within each of the plurality of cells along the first dimension of the pulsating array; and Hardware circuitry for each of the plurality of weight sequencer units provides control values for storage in a control register located in a specific cell along the first dimension of the systolic array, wherein the control values cause the plurality of weight inputs to be transferred to cells along a second dimension of the systolic array over a plurality of clock cycles, wherein each weight sequencer unit is located outside each of the plurality of cells and coupled to a different cell along the first dimension of the systolic array, wherein each weight input is stored using the weight register and in a corresponding cell along the second dimension, and wherein each cell is configured to compute the product of an activation input and a corresponding weight input using multiplication circuitry. 12. The method of claim 11, further comprising: At the value sequencer unit, multiple activation inputs are sent to cells along the second dimension of the systolic array of the neural network layer. 13. The method of claim 11, wherein the first dimension of the pulsating array corresponds to a row of the pulsating array, and wherein the second dimension of the pulsating array corresponds to a column of the pulsating array. 14. The method of claim 11, further comprising: for each cell, transmitting a weight control signal to an adjacent cell, the weight control signal causing the circuit in the adjacent cell to transfer or load weight input for the adjacent cell. 15. The method of claim 11, wherein each cell includes hardware circuitry for: A weight path register is located within the cell and coupled to the weight register, the weight path register being configured to store the weight input that is transferred to the cell; A weight control register is set within the cell and is used to store at least the control value provided by the weight sequencer or the weight control signal transmitted from the adjacent cell. The weight control register is configured to determine whether to store the weight input in the weight register. An activation register is set within the cell and configured to store activation input, and configured to send the activation input to another activation register in a first adjacent cell along the first dimension; A multiplication circuit is disposed within the cell and coupled to the weight register and the activation register, wherein the multiplication circuit is configured to output the product of the weight input and the activation input; An addition circuit, disposed within the cell and coupled to the multiplication circuit, and configured to receive the product and a first partial sum from a second adjacent cell along the second dimension, wherein the addition circuit is configured to output a second partial sum of the product and the first partial sum; and A local sum register, which is located within the cell and coupled to the adder circuit, and configured to store the second local sum, the local sum register being configured to send the second local sum to another adder circuit in a third adjacent cell along the second dimension. 16. The method of claim 15, further comprising: At the decrement circuit in each weight sequencer unit, the corresponding input to the weight sequencer unit is decremented to generate the corresponding decremented output. For each weight sequencer unit, the corresponding decreasing output is sent to the corresponding pause counter, which corresponds to the weight control register in the corresponding cell coupled to the weight sequencer unit. 17. The method of claim 16, wherein the values in each pause counter are identical, and the method further comprises: at each weight sequencer unit, loading a corresponding weight input into a corresponding different cell of the pulsation array, wherein loading includes sending the weight input to the multiplication circuit. 18. The method of claim 16, wherein the values in each pause counter are different, and the method further comprises: at each weight sequencer unit, transferring the corresponding weight input to an adjacent weight sequencer unit along the second dimension. 19. The method of claim 16, wherein the values in each pause counter reach predetermined values to cause the plurality of weight sequencer units to pause the transfer of the plurality of weight inputs along the second dimension. 20. The method of claim 11, further comprising: generating a corresponding accumulated output for the neural network layer from the respective product at a systolic array for each of the plurality of neural network layers.