HK1261497B - Vector processing units, computing systems having the same, and methods performed thereon - Google Patents

Description

Vector processing unit, computing system containing it, and method performed therein

Technical Field

This specification relates to a localized vector processing unit that can be used to perform various calculations associated with two-dimensional data arrays that can generally be referred to as vectors.

Background Technology

Vector processing units can be used for computations associated with deep neural network (“DNN”) layers, such as numerical simulation, graphics processing, game console design, supercomputing, and machine learning computation.

Generally speaking, a neural network is a machine learning model that uses one or more model layers to generate outputs, such as classification, from received inputs. A neural network with multiple layers can be used to compute inference by processing the inputs passed through each layer of the neural network.

Summary of the Invention

Compared to the characteristics of conventional Vector Processing Units (VPUs), this specification describes a VPU configured to partition computation into: a) an example Single Instruction Multiple Data (SIMD) VPU, which offers increased flexibility, increased memory bandwidth requirements, and relatively low computational density; b) a Matrix Unit (MXU), which offers lower flexibility, lower memory bandwidth requirements, and higher computational density; and c) a low-memory-bandwidth Cross-Axis Unit (XU) for performing certain operations that may not be suitable for the SIMD paradigm and may not have the computational density of MXU computational operations. In general, the contrast between the computational characteristics of at least a) and b) provides an enhanced SIMD processor design architecture relative to current/conventional SIMD processors. In some embodiments, the described VPU is an example Von-Neumann SIMD VPU.

Generally, an innovative aspect of the subject matter described in this specification can be embodied in a vector processing unit comprising: one or more processor units, each configured to perform arithmetic operations associated with vectorized computation of a multidimensional data array; and a vector memory communicating with each of the one or more processor units. The vector memory includes a storage bank configured to store data used by each of the one or more processor units to perform the arithmetic operations. The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling high-bandwidth data communication based on the placement of the individual processor units relative to each other and based on the placement of the vector memory relative to each processor unit.

In some embodiments, the vector processing unit is coupled to a matrix operation unit configured to receive at least two operands from a specific processor unit, the operands being used by the matrix operation unit to perform operations associated with vectorized computation of the multidimensional data array. In some embodiments, the vector processing unit further includes a first data serializer coupled to the specific processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the specific processor unit and received by the matrix operation unit. In some embodiments, the vector processing unit further includes a second data serializer coupled to the specific processor unit, the second data serializer being configured to serialize output data provided by the specific processor unit and received by at least one of the matrix operation unit, a cross-lane unit, or a reduction and permutation unit.

In some embodiments, each of the one or more processor units includes a plurality of processing resources, and the plurality of processing resources includes at least one of a first arithmetic logic unit, a second arithmetic logic unit, a multidimensional register, or a functional processor unit. In some embodiments, the vector memory is configured to load data associated with a particular memory bank into a corresponding processor unit, and wherein the data is used by a specific resource of the corresponding processor unit. In some embodiments, the vector processing unit further includes a crossbar connector between the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with the vector memory bank to a specific resource among the plurality of processing resources of the particular processor unit.

In some embodiments, the vector processing unit further includes a random number generator that communicates with the resources of a specific processor unit. The random number generator is configured to periodically generate numbers that can be used as operands for at least one operation performed by the specific processor unit. In some embodiments, the vector processing unit provides a main processing channel and includes a plurality of processor units, each forming a processor subchannel within the vector processing unit. In some embodiments, each processor subchannel is dynamically configured on a per-access basis to access a specific memory bank of the vector memory to retrieve data for performing one or more arithmetic operations associated with vectorized computation of the multidimensional data array.

Another innovative aspect of the subject matter described in this specification can be embodied in a computing system having a vector processing unit, the computing system comprising: processor units, each including a first arithmetic logic unit configured to perform a plurality of arithmetic operations; a vector memory communicating with each of the one or more processor units, the vector memory including a storage bank configured to store data used by each of the one or more processor units to perform the arithmetic operations; and a matrix operation unit configured to receive at least two operands from a particular processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computation.

The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling data communication to be exchanged with a first bandwidth based on a first distance between at least one processor unit and the vector memory. The vector processing unit and the matrix operation unit are coupled, enabling data communication to be exchanged with a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit. The first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth.

In some embodiments, the computing system further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit. In some embodiments, the computing system further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of the matrix operation unit, cross-channel unit, or reduction and permutation unit. In some embodiments, each of the one or more processor units further includes a plurality of processing resources, the plurality of processing resources comprising at least one of a second arithmetic logic unit, a multidimensional register, or a functional processor unit.

In some embodiments, the vector memory is configured to load data associated with a particular memory bank into a corresponding processor unit, wherein the data is used by a specific resource of the corresponding processor unit. In some embodiments, the computing system further includes a crosslink between the one or more processor units and the vector memory, the crosslink being configured to provide data associated with the vector memory bank to a specific resource among the plurality of processing resources of the particular processor unit. In some embodiments, the computing system further includes a random number generator that communicates with the resources of the particular processor unit, the random number generator being configured to periodically generate numbers that can be used as operands for at least one operation performed by the particular processor unit. In some embodiments, the computing system further includes a data path extending between the vector memory and the matrix operation unit, the data path enabling data communication associated with direct memory access operations occurring between the vector memory and at least the matrix operation unit.

Another innovative aspect of the subject matter described in this specification can be embodied in a computer-implemented method in a computing system having a vector processing unit. The method includes: providing data from a vector memory for performing one or more arithmetic operations, the vector memory including a storage bank for storing a corresponding dataset; receiving data from a specific storage bank of the vector memory by one or more processor units, the data being used by the one or more processor units to perform one or more arithmetic operations associated with vectorized computation; and receiving at least two operands from a specific processor unit by a matrix operation unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computation. The one or more processor units and the vector memory are tightly coupled within a region of the vector processing unit, such that data communication occurs at a first bandwidth based on a first distance between at least one processor unit and the vector memory. The vector processing unit and the matrix operation unit are coupled such that data communication occurs at a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit. The first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth.

The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. Compared to conventional vector processors, vector processing units, which include highly localized data storage and computational resources, can provide increased data throughput. The described vector memory and vector processing unit architecture enables localized, high-bandwidth data processing and arithmetic operations associated with the vector elements of the example matrix-vector processor. Therefore, based on the use of vector processing resources arranged in a tightly coupled configuration within the circuit chip, computational efficiency associated with vector arithmetic operations can be improved and accelerated.

Other implementations in this and other aspects include corresponding systems, apparatuses, and computer programs configured to perform actions of methods encoded on a computer storage device. A system of one or more computers can be configured, by means of software, firmware, hardware, or combinations thereof installed on the system, to cause the system to perform the actions during operation. One or more computer programs can be configured, by means of instructions, to cause the apparatus to perform the actions when the instructions are executed by a data processing device.

Details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other potential features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

Attached Figure Description

Figure 1 shows a block diagram of an example computing system that includes one or more vector processing units and multiple computing resources.

Figure 2 shows a block diagram of the hardware structure of the example vector processing unit.

Figure 3 shows a block diagram of an example computing system including a multiply-accumulate array and multiple computing resources.

Figure 4 is an example flowchart of the process for performing vector calculations.

Similar reference numerals and names in the various figures indicate similar elements.

Detailed Implementation

The subjects described in this specification generally relate to vector processing units (VPUs) that include highly localized data processing and computing resources configured to provide increased data throughput relative to existing vector processors. The described VPUs include an architecture that supports localized, high-bandwidth data processing and arithmetic operations associated with vector elements of the example matrix-vector processor.

Specifically, this specification describes a computing system including computing resources of a Virtual Processing Unit (VPU), which can be arranged in a tightly coupled manner within a predetermined region of an integrated circuit chip. The predetermined region can be divided into multiple VPU channels, and each channel can include multiple localized and different computing resources. Within each VPU channel, the resource includes a vector memory structure, which can include multiple memory banks, each having multiple memory address locations. The resource can also include multiple vector processing units or VPU sub-channels, each VPU sub-channel including multiple different computing assets/resources.

Each VPU subchannel can include a multidimensional data/file register configured to store multiple vector elements, and at least one arithmetic logic unit (ALU) configured to perform arithmetic operations on the vector elements accessible from and stored therein. The computing system can also include at least one matrix processing unit that receives serialized data from the respective VPU subchannel. Generally, the matrix processing unit is capable of performing non-local, low-bandwidth, and high-latency computations associated with, for example, neural network inference workloads.

For the described computing system, the highly localized nature of the vector processing capabilities provides high-bandwidth and low-latency data exchange between the vector memory and multiple VPU sub-channels, between the respective VPU sub-channels, and between the data register and the ALU. The substantially adjacent proximity of these resources enables data processing operations to occur within the VPU channels with full flexibility and at performance and data throughput exceeding the expectations of existing vector processors.

For example, the computational system described in this specification can perform computations on neural network layers by distributing vectorized computations across multiple matrix-vector processors. The computational process performed within a neural network layer may include multiplying an input tensor, which includes input activations, with a parameter tensor, which includes weights. A tensor is a multidimensional geometric object, and example multidimensional geometric objects include matrices and data arrays.

Generally, the description of one or more functions of the described VPU may be illustrated with reference to computations associated with neural networks. However, the described VPU should not be limited to machine learning or neural network computations. Rather, the described VPU can be used for computations associated with a variety of technical fields related to implementing vector processors to achieve desired technical goals.

Furthermore, in some implementations, large computation sets can be processed separately, allowing a first subset of computations to be processed within a separate VPU channel, while a second subset of computations can be processed within an example matrix processing unit. Therefore, this specification describes a dataflow architecture capable of implementing two types of data connectivity (e.g., local VPU channel connectivity and non-local matrix unit connectivity) to achieve the advantages associated with both forms of data processing.

Figure 1 shows a block diagram of an example computing system 100 including one or more vector processing units and multiple computing resources. The computing system 100 (system 100) is an example data processing system for performing tensor or vectorized computations associated with inference workloads of multi-layer DNNs. System 100 generally includes a vector processing unit (VPU) channel 102, a core sequence generator 104, external memory (Ext.Mem.) 106, and chip-to-chip interconnect (ICI) 108.

As used herein, a channel typically corresponds to a region, segment, or portion of the example integrated circuit chip that can include the computing/data processing resources of the VPU. Similarly, as used herein, a sub-channel typically corresponds to a sub-region, sub-segment, or sub-portion of a channel in the example integrated circuit chip that can include the computing/data processing resources of the VPU.

System 100 may include multiple VPU channels 102 disposed on an integrated circuit (IC) chip 103. In some embodiments, IC chip 103 may correspond to a portion or segment of a larger IC chip, which includes other circuit components/computing resources depicted in FIG1 in adjacent chip segments. In other embodiments, IC chip 103 may correspond to a single IC chip, which typically does not include the other circuit components/computing resources depicted in FIG1.

As shown, the other components/computing resources can include reference features (i.e., external memory 106, ICI 108, MXU 110, XU 112, RPU 113) located outside the area enclosed by the dashed line of IC chip 103. In some embodiments, a plurality of VPU channels 102 form the described VPU, and the VPU can be enhanced by functionality provided by at least one of MXU 110, XU 112, or RPU 113. For example, 128 VPU channels 102 can form the VPU described in the example. In some cases, fewer than 128 VPU channels 102 or more than 128 VPU channels 102 can form the VPU described in the example.

As discussed in more detail below, each VPU channel 102 can include a vector memory (vmem 204 in FIG. 2) with multiple memory banks having address locations for storing data associated with elements of a vector. The vector memory provides an on-chip vector memory accessible to corresponding vector processing units of the plurality of VPU channels 102 that can be located within IC chip 103. Generally, external memory 106 and ICI 108 each exchange data communication with an individual vmem 204 (described below), each vmem 204 associated with a corresponding VPU channel 102. This data communication can typically include, for example, writing vector element data to or reading data from the vmem of a particular VPU channel 102.

As shown, in some embodiments, IC chip 103 can be a single VPU channel configuration providing vector processing capabilities within system 100. In some embodiments, in addition to the single VPU channel configuration, system 100 can also include a multi-VPU channel configuration having a total of 128 VPU channels 102, which provides even more vector processing capabilities within system 100. The 128 VPU channel configuration is discussed in more detail below with reference to FIG2.

External memory 106 is an example memory structure used by system 100 to provide and/or exchange high-bandwidth data with vector memory associated with the corresponding vector processing unit of VPU channel 102. Generally, external memory 106 can be a remote or non-local memory resource configured to perform various direct memory access (DMA) operations to access, read, write, or otherwise store and retrieve data associated with address locations of vector memory within system 100. External memory 106 can be described as an off-chip memory configured to exchange data with on-chip vector memory (e.g., vmem 204) of system 100. For example, referring to FIG1, external memory 106 can be located outside IC chip 103, and therefore can be remote or non-local relative to computing resources located within IC chip 103.

In some implementations, system 100 may include an embedded processing device (discussed below) that executes software-based programming instructions (e.g., accessible from instruction memory) to, for example, move blocks of data from external memory 106 to vmem 204. Furthermore, the execution of the programmed instructions by the embedded processor may cause external memory 106 to initiate data transfers to load and store data elements in a vector memory accessible by the corresponding vector processing unit of VPU channel 102. The stored data elements may be instantiated into vector elements corresponding to register data accessible by a specific vector processing unit, in preparation for performing one or more vector arithmetic operations.

In some implementations, the vmem 100, external memory 106, and other associated memory devices of system 100 may each include one or more non-transitory machine-readable storage media. These non-transitory machine-readable storage media may include solid-state memory, disks (internal hard disks or removable disks), optical disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., EPROM, EEPROM, or flash memory), or any other tangible medium capable of storing information. System 100 may also include one or more processors and memories that can be supplemented or incorporated into dedicated logic circuitry.

ICI 108 provides an example resource capable of managing and/or monitoring multiple interconnected data communication paths coupled to different computing/data processing resources within system 100. In some embodiments, ICI 108 may typically include data communication paths that enable data to flow between non-local/off-chip devices and on-chip/local computing resources. Furthermore, ICI 108 may also typically include communication paths that enable data to flow between various on-chip or local computing resources located within IC chip 103.

The multiple communication paths within system 100, which couple various resources, can each be configured to have different or overlapping bandwidth or throughput data rates. As used herein, in the context of computing systems, the terms bandwidth and throughput generally correspond to the rate of data transmission, such as bit rate or data volume. In some implementations, bit rate can be measured in, for example, bits per second or bits per clock cycle, while data volume can correspond to the general width of data moving through multiple channels of system 100 in bits per word (e.g., 2 channels × 16 bits).

System 100 may also include a matrix unit (MXU) 110, a cross-channel unit (XU) 112, a reduction and permutation unit (RPU) 113, a matrix return element (mrf) 114, a cross-channel return element (xrf) 116, and an input control 122. Generally, the input control 122 may be a conventional control line used by a non-local control device (e.g., core sequence generator 104) to provide one or more control signals causing at least one of the MXU 110, XU 112, RPU 113, mrf 114, xrf 116, or PRNG 118 to perform a desired function. In some embodiments, the core sequence generator 104 provides multiple control signals to components of the VPU channel 102 via the input control 122, thereby controlling the overall function of the VPU channel 102.

Although described in the example of Figure 1, the MRF 114, XRF 116, and PRNG 118, and their respective functions, are discussed in more detail below with reference to the embodiments of Figure 2. Similarly, the MXU 110, XU 112, and RPU 113 are discussed in more detail below with reference to the embodiments of Figures 2 and 3.

Figure 1 includes data list 124 (also shown as feature 224 in Figure 2), which indicates the relative magnitude of data throughput, in bits, associated with a specific data path of “N” channels, where N can vary from, for example, 1 to 16 channels/within that range. As shown in Figures 1 and 2, different dashed line features can be used to describe the data lines to indicate that specific channels/data paths can have different individual throughput (in bits/bytes) attributes. Note that data lists 124 and 224 are not included in system 100 but are shown in Figures 1 and 2 for clarity and to indicate the throughput of specific data paths coupled with different computing resources.

Figure 2 shows a block diagram of the hardware structure of an example vector processing unit of the system of Figure 1. The computing system 200 (system 200) generally includes multiple vector processing units 202, a vector memory (vmem) 204, a register file 206, a processing unit interconnect 207, a first arithmetic logic unit (ALU) 208a, a second ALU 208b, a dedicated unit 210, a first crossbar 212a, and a second crossbar 212b. In the embodiment of Figure 2, the vector processing unit 202 is depicted as a sub-channel of the VPU channel 202. In some embodiments, multiple (x8) vector processing units 202 can be disposed within a single VPU channel 202.

In some embodiments, one or more circuit portions of system 100 can be disposed within a predetermined region of IC chip 103. As described above, system 100 can include a plurality of VPU channels 102 disposed on IC chip 103. In some embodiments, IC chip 103 can be divided into portions or segments including sub-segments, within which certain computing resources are disposed. Thus, in the example of FIG2, a single VPU channel 102 can include a plurality of VPU sub-channels (i.e., vector processing units) 202 disposed on IC chip portion 203, IC chip portion 203 corresponding to a larger sub-portion/sub-segment of IC chip 103.

Generally, each processor unit 202 of VPU channel 102 can include multiple processing resources, and each processor unit 202 can be configured to perform arithmetic operations (via ALU) associated with vectorized computation of multidimensional data arrays. As shown, each vector processing unit or sub-channel 102 includes register file 206, ALU 208a and ALU 208b, and dedicated unit 210. The computing resources located within IC chip section 203 can be tightly coupled together and are therefore arranged substantially adjacent to each other within IC chip section 203. This substantially adjacent proximity of the processing resources enables data operations to occur in VPU channel 102 with sufficient flexibility and at high bandwidth or data throughput.

In some implementations, "tight coupling" can refer to wiring between components/computing resources and data transmission bandwidth, both of which connect the components/resources within, for example, 100 micrometers of each other. In other implementations, "coupling" rather than "tight coupling" can refer to wiring between components/resources and data transmission bandwidth, each of which connects the components within, for example, 200 micrometers to 10 millimeters of each other.

In alternative implementations, components or computing resources of systems 100 and 200 can be tightly coupled or coupled with reference to a specific ratio of the total chip size (e.g., the size of chip 103 or the size of chip portion 203). For example, "tight coupling" can correspond to components connected within a maximum of 5% of the total chip edge size, while "coupling" can correspond to components further away, for example, up to 50% of the total chip edge size.

In some embodiments, innovative features of the VPU of the described computing system 100 include components and/or computing resources in VPU channels 102, each within a specific or threshold distance from each other, such that data (e.g., one or more 32-bit words) can easily traverse that distance in a single clock cycle (i.e., line delay). In some embodiments, these innovative features of the described VPU directly correspond to the tight coupling arrangement of at least the components of the VPU channels 102 relative to each other.

In some implementations, the conductors (i.e., wires) providing data flow paths between different tightly coupled resources of subchannel 102 can be relatively short in length, however, the total number of conductors or the bus width can be large when the bus is a set of wires. The larger bus width (compared to the width of a conventional IC bus) enables high-bandwidth data transmission, corresponding to a large number of operations. The high-bandwidth attribute of these multiple operations allows data to traverse the localized resources of vector processing unit 102 with low latency. As used herein, high bandwidth and low latency correspond to hundreds (or thousands in some implementations) of operations associated with moving multiple 16-bit to 32-bit words (i.e., high bandwidth) from one computing resource to another within a single clock cycle (i.e., low latency). The high-bandwidth, low-latency attributes of system 200 are described in more detail later herein.

Generally, each individual vmems 204 associated with a corresponding VPU channel 102 is configured to exchange data communication with external memory 106. This data communication typically includes, for example, external memory 106 writing to/reading vector element data from the vmems 204 of the corresponding VPU channel 102. The vmems 204 communicate with each processor unit 202 and their respective multiple processing resources (e.g., ALU 208a/208b). The vmems 204 can include multiple memory banks that store data used by each processor unit 202 at corresponding address locations to instantiate vectors accessed by the ALU 208a/208b (via register 206) to perform one or more arithmetic operations.

In some implementations, VPU channel 102 can include a data path extending between loosely coupled memories located at one or more locations in system 200, such as vmem 204 and system 200. The loosely coupled memories can include off-chip memory, on-chip memory that does not require tight coupling or high bandwidth, memory from other processing units, such as other VPUs on the interconnect, or data transferred to or from a connected host computer. In some implementations, DMA transfers can be initiated by control signals locally (e.g., from CS unit 104) or remotely (e.g., via the host computer). In some implementations, data communication traverses the data path via ICI network 108, while in other implementations, the data communication traverses the data path via processor unit 202. In some implementations, the DMA path can also be serialized/deserialized using the same mechanism used for data paths extending to and from MXU 110.

System 200 typically provides a two-dimensional (2-D) array of tightly coupled data paths, enabling system 100 to perform thousands of data operations per clock cycle. The two-dimensional array corresponds to a total of 128 channels (e.g., 128 VPU channels 102) multiplied by 8 sub-channels per channel. A VPU channel 102 can be described as a processing unit comprising multiple (e.g., x8) processor units (i.e., sub-channels), each typically coupled to one of multiple (e.g., x8) memory banks. The 2D array of data paths in system 200 can have spatial characteristics, thereby enabling the coupling and implementation of specific data paths on separate hardware structures.

In some implementations, for the eight different vector processing units 202 (i.e., x8-dimensional) of VPU channel 102, when the eight vector processing units 202 exchange data with other resources of system 200 such as MXU 110, XU 112, and RPU 113 (discussed below), data operations on a single channel 102 can be serialized and deserialized via deserializers 222a/b. For example, a particular vector processing operation can include the VPU channel 102 sending multiple (x8) 32-bit words to MXU 110. Thus, each of the eight vector processing units 202 in a single channel 102 can send a 32-bit word accessible from its local register 206 to MXU 110.

In some implementations, the 32-bit word can be transmitted serially as a 16-bit rounded floating-point number at an example data rate of one word per clock cycle (16 bits/clock cycle). The vector processing operation can also include the MXU 110 providing each of the eight vector processing units 202 with the result of a multiplication operation performed by the MXU. The result can be received by the VPU channel 102 and simultaneously stored (i.e., deserialized) in the corresponding registers 206 of the eight sub-channels 202 within a single processor clock cycle (256 bits/clock cycle).

Crosslinker 212a provides a data path from vmem 204 to at least one processor unit 202 and includes a 32-bit word traversing the data path during certain data transfer operations. Similarly, crosslinker 212b provides a data path from at least one processor unit 202 to vmem 204 and includes a 32-bit word traversing the data path during certain data transfer operations. In some embodiments, the interface between vmem 204 and a particular VPU subchannel 202 is a load-type instruction. For example, a specific operation instruction (e.g., from instruction memory) can specify for each subchannel 202 a particular memory bank that the subchannel will access to retrieve vector-related data for loading into local register 206. In some embodiments, each processor unit 202 can be dynamically configured on a per-access basis to access a specific memory bank of vmem 204 to retrieve vector data.

In some embodiments, data transmission via cross connectors 212a/b occurs in the x8 dimension of the aforementioned 2-D data path array within system 200. Cross connectors 212a/b provide full connectivity between each individual subchannel 202 (x8) within each of the 128 channels and each individual memory bank (x8) of vmem 204. Generally, because vmem 204 is located within IC chip portion 203 in relatively close proximity to the corresponding processor unit 202, cross connectors 212a/b can be implemented via wires of relatively short length but with a relatively large bus width (or line count) to facilitate high data throughput between processor unit 202 and vmem 204. In some embodiments, vmem 204 can perform a broadcast function to provide a specific set of vector data to multiple vector processing units 202.

As described above, each vector processing unit 202 can include a multidimensional data/file register 206 configured to store multiple vector elements. Therefore, register 206 can be a fixed-length storage unit that stores data corresponding to a single vector. Specifically, register 206 can fill a specific vector register with multiple vector elements using data received by processor unit 202 (from vmem 204). In some embodiments, register 206 fills up to 32 vector registers, typically represented as _V0 - _V31 , using data received from a specific memory bank of vmem 204. More specifically, each vector register can include multiple 32-bit words. As used herein, a vector can typically correspond to an array (linear or nonlinear) of binary values corresponding to certain types of data, such as integers or floating-point numbers.

The 32-bit data can correspond to one or more ALU operands. In some implementations, each vector processing unit 202 accesses a specific memory bank of vmem 204 to load its own local register file 206 to perform its own local processing. In the example process, one or more vector processing units 202 of system 200 can be configured to execute instructions (e.g., code sequences) for example arithmetic operations. Subchannel interconnect 207 can be used to move data between at least two different vector processing units of system 200.

The arithmetic operations can include two register load operations, an addition operation, and a store operation. In some embodiments, in response to system 200 receiving certain control signals from a higher-level controller device, instructions for the operation can be fetched from instruction memory (not shown) and decoded locally. Regarding the operations, the first load sequence can include system 200 loading vector data from example memory address 0x00F100 of vmem 204 into at least one vector register ( _V0 ) of vector processing unit 202. Similarly, the second load sequence can include system 200 loading vector data from example memory address 0x00F200 of vmem 204 into at least one other vector register ( _V1 ) of vector processing unit 202.

In terms of hardware layout, in some implementations, vmem 204 can be divided into 128 channels, with 8 memory banks per channel, and each memory bank having multiple address locations. Therefore, during the load sequence of the operation, sample registers within systems 100, 200 will receive vector data from vmem 204. In some implementations, and as noted above, the sample vector processing unit can include multiple VPU channels 102. Therefore, the respective registers 206 on one or more VPU channels 102 can cooperate to form a vector register spanning 128 channels multiplied by 8 sub-channels 202 dimensions.

In the 128 dimensions, a single VPU channel 102 can load from its corresponding vmem 204. More specifically, in the sub-channel dimension (x8), each sub-channel 202 can load its vector register from a specific memory bank among the eight memory banks of vmem 204. In some embodiments, strided memory access operations can also be performed. Regarding this operation, the completion of the first loading sequence results in vector data being loaded into vector register _V0 , such that the register will contain 128 × 8 values. For clarity, in some embodiments, the full dimension of the vector registers of the VPU channel 102 can be 128 channels x 8 sub-channels x 32 registers x 32 bits. Thus, 128 × 8 corresponds to the total number of sub-channels, and 32 × 32 corresponds to the number of bits in the vector register for each sub-channel.

The completion of the second loading sequence results in vector data being loaded into vector register _V1 , so that the register will also contain 128 x 8 values. Next, an addition instruction via ALU 208a or 208b can be executed, which includes adding _V0 (128 x 8 values) to _V1 (128 x 8 values). In some embodiments, an example permutation operation (to sort, rearrange, or order the data) can be performed on the summed vector data after the storage operation to store the data in example vector register V3. Furthermore, as discussed below, a permutation operation can be performed to move data between at least two different VPU channels 102.

The high bandwidth and low latency attributes of the localized resources within IC chip section 203 can be characterized with reference to the following examples. Generally, the 128×8 dimensions of system 200 create 1024 potential data paths within the example VPU. These data paths correspond to the eight memory banks in a single channel (VPU channel 102) of vmem 204, which provide eight individual 32-bit words to each of the eight individual sub-channels within VPU channel 202 along eight individual channels (via cross connectors 212a/b). More specifically, these eight individual channels are replicated across 128 channels to create the 1024 potential data paths.

When each 32-bit word traverses the path between the first resource and the second resource in IC chip segment 203, the 1024 data paths can correspond to 1024 operations. Furthermore, ALUs 208a and 208b create an additional 1024 potential data paths, corresponding to at least 2048 operations that can occur on multiple resources within IC chip 203. Therefore, the tight coupling, high localization, and high bandwidth properties of the resources in IC chip segment 203 enable at least 2048 operations to occur within a single clock cycle. Moreover, depending on the type of operation performed on the word, each of these 2048 operations occurring simultaneously in a single clock cycle can include a 32-bit word (e.g., a vector or operand) traversing a specific data path.

In some implementations, and to extend the above examples, one or more of the following can occur within a single clock cycle executed by system 200: 1) eight vectors move from vmem 204 to eight sub-channels 202; 2) two vector operands move from register 206 to ALUs 208a and 208b; 3) two result vectors move from ALUs 208a and 208b to register 206; 4) eight vector operands move from the respective sub-channels 202 to serializers 214 or 216 (described below); 5) eight result vectors move from mrf 114 or xrf 116 (described below); and 6) eight result vectors move from the eight sub-channels to XU/RPU serializer 218 (described below). The foregoing example operations merely illustrate the high bandwidth properties of the tightly coupled localized resources of system 200.

Dedicated unit 210 provides additional local processing capabilities, and in some embodiments, it can be synonymous with the functionality provided by the ALUs 208a/208b of the corresponding sub-channels 202. In some embodiments, dedicated unit 210 can be described as a functional processor unit. For example, dedicated unit 210 can be designed to process and evaluate unary transcendental functions associated with arithmetic operations on vector data stored in local register 206. Thus, certain complex arithmetic operations corresponding to, for example, exponential or logarithmic functions can be performed by dedicated unit 210.

As described above, the technical feature of the described systems 100 and 200 is that each sub-channel 202 is physically close together (i.e., very tightly coupled), such that high-bandwidth arithmetic operations performed by ALUs 208a and 208b occur simultaneously during a single processor clock cycle. In some embodiments, certain complex arithmetic operations may require additional clock cycles to complete. Therefore, system 200 may employ a dedicated unit 210 to separate certain complex multi-cycle operations for special processing.

PRNG 118 can be a shared resource configured to generate pseudo-random numbers that can be used by registers 206 on multiple sub-channels 202 during vector arithmetic operations performed by the ALUs 208a/208b of each sub-channel 202. Generally, PRNG 118 can receive at least one control signal from vector processing unit 202 to initialize the example digital generator circuitry to an initial state. PRNG 118 can then evolve from this initial state to periodically generate random numbers that can be used by a specific vector processing unit 202 to perform an operation associated with vector operations.

Generally, each vector processing unit 202 typically performs read operations on the PRNG 118. Occasionally, a specific subchannel may provide control signals to the PRNG 118 to perform a write sequence, for example, to induce a numerical reproducibility operation. Certain reproducibility operations can be used to implement specific numerical techniques suitable for computations involving neural network inference workloads. Furthermore, during vectorization computation, it may be advantageous for the system 200 to slightly distort the numerical rounding operations associated with the computation by injecting random noise, thereby producing some narrower representations of one or more numerical values. Also, in some embodiments, the PRNG 118 can provide another source of operands for data processing occurring within the subchannel 202.

System 200 also includes a first data serializer 214, a second data serializer 216, an XU/RPU serializer 218, and data deserializers 222a/b, each coupled to a specific processor unit 202. Generally, data serializers 214 and 216 are configured to serialize vector output data, which can include at least two operands provided by the specific processor unit 202 and received by the MXU 110. As shown, the serialized vector data can be provided to the MXU 110 via data paths 220a/b, such that a first operand can be provided via the first data path 220a and a second operand can be provided via the second data path 220b. In some embodiments, data serializers 214 and 216 can be configured to function as shift registers, sequentially shifting operand data out over multiple clock cycles (high latency).

Generally, data serializers 214 and 216 enable the respective sub-channels 202 to transmit serialized vector output data via time-division multiplexing over expensive interconnects. These expensive interconnects provide data paths 220a/b/c to remote non-local coprocessing resources that perform certain multiplication operations on the received serialized vector data. As described above, for the embodiment of FIG2, the remote non-local coprocessing resources can correspond to resources outside of IC chip segment 203 (e.g., MXU 110, XU 112, and RPU 113). These resources typically receive low-bandwidth (e.g., a single 32-bit operand), high-latency (over multiple clock cycles) vector data via data paths 220a/b/c.

Regarding data movement and data volume, each of the 128 channels (i.e., VPU channels 102) can have eight data words or operands, each 32 bits wide. These eight data words can correspond to each of the eight sub-channels within VPU channel 102. System 200 can be configured to load these eight data words into, for example, data serializers 214, 216, or 218. These eight data words can then be shifted out to one of MXU 110, XU 112, or RPU 113 over a period of eight processor clock cycles. In contrast to the short, wide, high-bandwidth data paths between the tightly coupled localized resources of IC chip segment 203, each of MXU 110, XU 112, and RPU 113 is considerably farther and non-local relative to the cell proximity of the resources of its corresponding sub-channel 202.

Therefore, in the example VPUs incorporated into systems 100 and 200, each clock cycle, the VPU is capable of executing instructions to perform operations utilizing and/or shifting 1024 words, each 32 bits wide. As the vector data portion collectively forming the 1024 words arrives and/or passes through a single data serializer 214, 216, the data then proceeds via data paths 220a/b that operate or shift out only 128 words per clock cycle. Thus, the data serializers 214, 216 can be configured to serialize only data in the x8 dimension, ensuring parallelism on each VPU channel 102 in the x128 dimension.

For example, data serializers 214 and 216 can be functionally independent of each other, so that in the first clock cycle (e.g., cycle N), system 200 can cause all 1024 words (8 words per channel and 1 word per subchannel for all 128 channels) to be loaded into the memory location of, for example, the first data serializer 214 for a specific vector processing unit 202. System 200 can then execute one or more instructions to cause the contents of each of the first data serializers 214 on the 128 channels to be shifted out to MXU 110 via the corresponding data path 220a at a bandwidth of 16 bits per clock cycle. In some embodiments, the 32-bit words received by serializer 214 can be transmitted serially as 16-bit rounded floating-point numbers.

Furthermore, for clarity, although the 32-bit operands provided to the MXU 110, XU 112, and RPU 113 are described herein as “words,” the operands can generally correspond to numbers (e.g., floating-point numbers), and the descriptor “word” is used only to indicate a fixed-size segment of binary data that can be processed as a unit by the hardware devices of the example processor core.

Referring again to the example data stream sequence, in the second clock cycle (e.g., cycle N+1), system 200 can cause an additional 1024 words (8 words per channel and 1 word per subchannel for all 128 channels) to be loaded into the memory location of, for example, the second data serializer 216 for the same vector processing unit 202. System 200 can then execute one or more instructions to cause the contents of each of the second data serializers 216 on the 128 channels to be shifted out to, for example, MXU 110 via the corresponding data path 220b at a bandwidth of 16 bits per clock cycle. Thus, the data paths 220a/b extending from the data serializers 214, 216 can be used in parallel with each other.

In some implementations, this example data stream sequence can continue to load several sets of matrix multiplication operands onto the MXU 110 over multiple data cycles (e.g., cycle N+2, cycle N+3, etc.). When loaded, a large number of matrix multiplication operations associated with, for example, vectorized computations can be processed by the MXU 110 to compute example inference workloads. The results of these matrix multiplications can be received and stored in, for example, a memory unit of the mrf 114, for reception by a specific subchannel 202 within a specific VPU channel 102. The mrf 114 includes a first-in-first-out (FIFO) function and can be configured to save/store return data (multiplication results) associated with longer-latency operations. The return data stored in the memory of the mrf 114 can be written back to the vector register 206 using a separate, shorter-latency instruction.

The matrix multiplication result can be moved in the serialized data stream from MXU110 to mrf 114 at a throughput of 32 bits per clock cycle. In some embodiments, the result of the matrix multiplication is received in a first time period and, after being deserialized by deserializer 222a, stored in mrf 114 for reception by subchannel 202 in a second time period later than the first time period. In some embodiments, the second time period corresponds to a time point that can occur from 1 clock cycle to 128 clock cycles.

For example, during the first processor clock cycle, mrf 114 can receive the matrix multiplication result in a first time period and store the result in the memory address of mrf 114. After system 200 has executed another 100 processor clock cycles to perform other vector processing operations, system 200 can then execute an instruction to pop mrf 114 and receive the result data in a second time period 100 clock cycles later. As described above, mrf 114 executes a first-in-first-out data stream sequence, such that the first received matrix multiplication result is first written to a specific vector register of register 206.

Regarding reduction and permutation operations, the RPU 113 can include sigma units and permutation units. In some implementations, the result of computation processed by the sigma unit is provided to the permutation unit. The sigma unit or the permutation unit can be disabled so that data is not altered by passing through a specific unit. Generally, the sigma unit performs sequential reduction on a single data line. The reduction can include summation and various types of comparison operations.

In response to receiving input data, the permutation unit can perform fully universal crosslink operation, partially based on a command/control vector, which is a set of bits from the input data. For reduction operations, the data used by RPU 113 can be in 32-bit floating-point (FP) format; while for permutation operations, various data types/formats can be used, including FP, integers, and addresses. In some implementations, RPU 113 provides any received data to XU 112, receives result data from XU 112, and performs one or more muxing operations to generate different output streams with multiple result data.

In some implementations, the permutation operation can be performed by RPU 113 to move data between at least two different VPU channels 102. Generally, the permutation instruction causes 128×8 data values to be moved from the corresponding register 206 to the sub-channel XU/RPU serializer 218. Specifically, during the execution of the operation, 32-bit vector result data is serialized in x8 dimensions. Therefore, within each of the 128 channels (VPU channels 102), eight vector result words corresponding to eight sub-channels can be moved from the first VPU channel 102 to the second VPU channel 102 within a time period of eight processor clock cycles.

The vector data can move along data path 220c in the serialized data stream from XU/RPU serializer 218 to XU/RPU 112, 113 at a throughput of 32 bits per clock cycle across two channels. For a specific VPU channel 102 that receives cross-channel vector data, xrf 116 can include, for example, a memory configured to store the cross-channel vector result data received at that specific VPU channel 102. In some embodiments, the vector data can be received in a first time period and stored in xrf 116 after being deserialized by deserializer 222b for reception by subchannel 202 in a second time period later than the first time period.

In some implementations, the second time period corresponds to a time point ranging from one clock cycle to 128 clock cycles. For example, during the first processor clock cycle, xrf 116 can receive vector data from the first VPU channel 102 during the first time period and store the result in the memory address of xrf 116. After system 200 has executed another 100 processor clock cycles to perform other vector processing operations, system 200 can then execute instructions to pop xrf 116 and receive vector data during the second time period 100 clock cycles later. Generally, similar to mrf 114, xrf 116 also implements a first-in-first-out data stream sequence, such that the first received vector is written to a specific vector register in register 206.

Figure 3 shows a block diagram of the example computing system of Figure 1, including a multiply-accumulate array and multiple computing resources. As shown, system 300 can typically include one or more of the components discussed above with reference to Figures 1 and 2. System 300 can also include an embedded coprocessor 302. Generally, processor 302 can be configured to execute software-based programming instructions to move blocks of data from external memory 106 to multiple virtual memory modules (vmem 204). Furthermore, the execution of these instructions can cause external memory 106 to initiate data transfers to load and store data elements within vmem 204.

Figure 3 includes a data map 304 that indicates the relative magnitude, for example, in bits, associated with the data throughput of a particular flow path. As shown, the data map 304 includes various legends that correspond to the individual throughput (in bits) of a given path between certain computing resources. Note that the data map 304 is not included in system 300, but is shown in Figure 3 for clarity and to indicate the throughput of a particular data path coupled with different computing resources.

Generally, the example in Figure 3 provides another representation of the resources of system 200. For example, system 300 includes two VPU channels 102, corresponding to two of the 128 individual channels discussed above. Similarly, for each channel 102, system 300 also includes two sub-channels 202, corresponding to two of the eight individual sub-channels discussed above. System 300 also includes eight individual channels 306, which provide data flow paths (via cross connector 212) between the eight banks of vmem 204 and the corresponding eight sub-channels 202.

As described above, and as shown in data mapping 304, eight 32-bit vector words can be moved from vmem 204 to the eight individual sub-channels during a single processor clock cycle. As shown, in some embodiments, vmem 204 can be static random access memory (SRAM), and sub-channel 202 can be described as a single-input multiple-data unit. System 300 also includes another representation of the MXU 110 and cross-channel (XU) unit 112 discussed above with reference to FIG. 2.

Generally, the MXU 110 corresponds to a multiply-accumulate operator with 128×128 dimensions and is therefore configured to receive a large set of vector-matrix multiplication operands. As mentioned above, once a sufficient number of vector operands are loaded, the MXU 110 is able to handle a large number of matrix multiplication operations associated with vectorized computation to compute example inference workloads.

As shown, each subchannel 202 includes a data flow path toward (outward) XU 112 and a data flow path from (inward) XU 112 toward subchannel 202. These two distinct flow paths correspond to the function of XU that enables vector data to move between at least two distinct VPU channels 102. Therefore, each VPU channel 102 will typically include an outward vector data flow path toward XU 112 to correspond to the case where vector data from the first VPU channel 102 moves to the second VPU channel 102. Similarly, each VPU channel 102 will typically include an inward vector data flow path from XU 112 to correspond to the case where vector data from the first VPU channel 102 is received by the second VPU channel 102.

Figure 4 is an example flowchart of the process of performing vector calculations using the computing systems of Figures 1 and 2. Therefore, process 400 can be implemented using the aforementioned computing resources of systems 100 and 200.

Process 400 begins at block 402 and the vector memory (vmem 204) provides data for performing one or more arithmetic operations. As described above, vmem 204 can include multiple memory banks for storing the corresponding vector dataset. The vector data is provided to one or more processor units 202 of VPU channel 102. At block 404, at least one processor unit 202 receives the vector data provided by vmem 204. The received data can correspond to a specific memory bank, and further can be vector data accessed by the processor unit 202 from a specific address location of the memory bank. The data received by the processor unit 202 is used by register 206 to instantiate a specific vector register having multiple vector elements.

At block 406, based on the tight coupling between processor unit 202 and vmem 204, data communication between vmem 204 and at least one processor unit 202 is exchanged at a specific bandwidth (first bandwidth). In some embodiments, for certain operations, data can move at an example bandwidth or data rate of 256 bits per clock cycle (8 channels x 32 bits) on one or more of the eight channels interconnecting channel 204 and register 206 of processor unit 202. Generally, multiple operations can occur during a given clock cycle, and the number of operations can be in the range of 1024-2048 operations per clock cycle (e.g., high bandwidth operations).

In some implementations, the two computing resources are tightly coupled when a specific distance (e.g., a first distance) between processor unit 202 and vmem 204 is in the range of 0.001-100 micrometers. For example, processor unit 202 and vmem 202 can be tightly coupled when the first distance is between 0.001 micrometers and 0.1 micrometers, between 0.01 micrometers and 10 micrometers, or between 0.1 micrometers and 100 micrometers. Similarly, when a specific distance between multiple processing resources of processor unit 202 (e.g., register file 206, ALU 208a/b, and dedicated unit 210) is also in the range of 0.001-100 micrometers, the multiple resources can be tightly coupled relative to each other. The example distances provided above with reference to processor unit 202 and vmem 204 can also be applied to the distances between multiple resources of vector processing unit 202.

At block 408, the received data can be used, accessed, or modified by ALU 208a or ALU 208b of processor unit 202 to perform one or more arithmetic operations associated with vectorized computation for computing inference workloads. At block 410 of process 400, based on the fact that processor unit 202 and MXU 110 are coupled rather than tightly coupled, at least one data communication between processor unit 202 and MXU 110 is exchanged at a specific bandwidth (second bandwidth).

In some implementations, for certain operations, data can be moved at an example bandwidth or data rate of 32 bits per clock cycle (2 lines x 16 bits) on at least one of the two data lines interconnecting a single processor unit 202 with the MXU 110. Typically, multiple operations can occur during a given clock cycle between local and non-local resources (e.g., sub-channel 202 to MXU or XU), and the number of operations can be in the range of 10-12 operations per clock cycle (e.g., low-bandwidth operations).

In some implementations, the two computing resources are coupled when a specific distance (e.g., a second distance) between processor unit 202 and vmem 204 is within an example range of 200 micrometers to 10 millimeters (mm). For example, processor unit 202 and MXU 110 can be coupled when the second distance is between 200 micrometers and 1 mm, between 500 micrometers and 2 mm, or between 1 mm and 10 mm. Similarly, processor unit 202 and XU 112 (or RPU 113) can be coupled rather than tightly coupled when a specific distance between processor unit 202 and XU 112 (or RPU 113) is also within the range of 200 micrometers to 10 mm.

The example distances provided above for reference processor unit 202 and MXU 110 can also be applied to the distance between vector processing unit 202 and XU 112 (or RPU 113). In some embodiments, the second distance can exceed 10 mm, but can be less than the standard distance between discrete ICs on the printed circuit board of the example computer system.

At block 412, MXU 110 receives at least two operands (each 32 bits wide) from at least one processor unit 202. Generally, these at least two operands are used by MXU 110 to perform operations associated with vectorized computation of a multidimensional data array. As described above, MXU 110 can include an array of multiply-accumulate operators (MAC array 310) configured to perform thousands of multiplication and floating-point operations associated with vectorized computation to compute inference workloads for specific neural network layers.

The MAC array 310 can be further configured to provide computation results back to the vmem 204 for storage at an address location in a specific memory bank. In some embodiments, the sub-channel 202 provides local vector result data to the XU 112, allowing the results to be shared between one or more other VPU channels 102. For example, the computation result (output) from the first VPU channel 102 can be used as input to a computation occurring within another second VPU channel 102. In some embodiments, the second VPU channel 102 can be configured to perform vectorized computations associated with the inference workload of another neural network layer.

Embodiments of the subject matter and functional operation described in this specification can be implemented as digital electronic circuits, as tangibly embodied computer software or firmware, as computer hardware—including the structures disclosed in this specification and their structural equivalents, or as a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier and intended for execution by or control of the operation of a data processing device. Alternatively, or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of these.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be performed by dedicated logic circuitry such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units), and the apparatus can also be implemented as said dedicated logic circuitry.

Computers suitable for executing computer programs include, for example, those based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled thereto to receive data from or transfer data to, or both. However, a computer does not need to have such devices.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; and disks such as internal hard disks or removable disks. The processor and memory can be supplemented by or incorporated into dedicated logic circuitry.

While this specification contains numerous specific details of implementation, these should not be construed as limiting any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described in this specification within the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features from a claimed combination can be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific or sequential order shown, or to perform all the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Preferred embodiment:

Example 1a:

A vector processing unit comprising:

One or more processor units, each configured to perform arithmetic operations associated with vectorized computations of a multidimensional data array; and

A vector memory that communicates data with each of the one or more processor units, wherein the vector memory includes a storage body configured to store data used by each of the one or more processor units to perform the arithmetic operation;

The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling high-bandwidth data communication based on the placement of the individual processor units relative to each other and the placement of the vector memory relative to each processor unit.

Example 1b:

The vector processing unit described in Example 1a, wherein the tight coupling is characterized by at least one of the following embodiments:

One or more processor units and the vector memory are physically close to each other within the IC chip segment, preferably arranged adjacent to each other;

- Distance less than 100 micrometers, preferably less than 10 micrometers, very preferably less than 1 micrometer; or

- The distance is less than 5% of the total edge size of the IC chip;

And/or the high bandwidth feature is present in at least one of the following embodiments:

- Hundreds or thousands of operations associated with multiple 16-bit to 32-bit words; or

- At least 2048 operations, preferably occurring on multiple resources within the IC chip, and very preferably including 32-bit words.

Preferably with low latency, and very preferably with the above-mentioned number of operations per single clock cycle.

Example 2:

The vector processing unit described in Embodiment 1a or 1b, wherein the vector processing unit is configured to be coupled to a matrix operation unit, the matrix operation unit being configured to receive at least two operands from a specific processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorization computation of the multidimensional data array.

Example 3:

The vector processing unit described in Embodiment 2 further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit.

Example 4:

The vector processing unit described in Embodiment 2 or 3 further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of the following: the matrix operation unit, the cross-channel unit, or the reduction and permutation unit.

Example 5:

The vector processing unit according to any one of embodiments 1a to 4, wherein each of the one or more processor units comprises:

Multiple processing resources, the multiple processing resources including at least one of a first arithmetic logic unit, a second arithmetic logic unit, a multidimensional register, or a functional processor unit.

Example 6:

The vector processing unit described in any one of Embodiments 1a to 5, wherein the vector memory is configured to load data associated with a specific memory into a corresponding processor unit, and wherein the data is used by specific resources of the corresponding processor unit.

Example 7:

The vector processing unit described in any one of Embodiments 1a to 6 further includes a crossbar connector between the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with the vector memory to a specific resource among a plurality of processing resources of a particular processor unit.

Example 8:

The vector processing unit described in any one of Embodiments 1a to 7 further includes a random number generator that communicates with the resources of a particular processor unit, the random number generator being configured to periodically generate numbers that can be used as operands for at least one operation performed by the particular processor unit.

Example 9:

The vector processing unit according to any one of Embodiments 1a to 8, wherein the vector processing unit provides a processing channel and includes a plurality of processor units, each of the plurality of processor units forming a processor sub-channel within the vector processing unit.

Example 10:

The vector processing unit described in Example 9, wherein each processor subchannel is dynamically configured on a per-access basis to access a specific memory bank of the vector memory to retrieve data for performing one or more arithmetic operations associated with the vectorized computation of the multidimensional data array.

Example 11:

A computing system with a vector processing unit, the system comprising:

One or more processor units, each including a first arithmetic logic unit configured to perform a plurality of arithmetic operations;

A vector memory that communicates data with each of the one or more processor units, the vector memory including a storage bank configured to store data used by each of the one or more processor units to perform the arithmetic operation; and

A matrix operation unit is configured to receive at least two operands from a specific processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computation;

The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling data communication to be exchanged with a first bandwidth based on a first distance between at least one processor unit and the vector memory;

The vector processing unit and the matrix operation unit are coupled such that data communication can be exchanged with a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit; and

Wherein the first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth.

Example 12:

The computing system of Embodiment 11 further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit.

Example 13:

The computing system of Embodiment 12 further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of the following: the matrix operation unit, the cross-channel unit, or the reduction and permutation unit.

Example 14:

The computing system described in any one of Embodiments 11 to 13, wherein each of the one or more processor units further includes a plurality of processing resources, the plurality of processing resources including at least one of a second arithmetic logic unit, a multidimensional register, or a functional processor unit.

Example 15:

The computing system of Embodiment 14, wherein the vector memory is configured to load data associated with a specific memory bank into a corresponding processor unit, and wherein the data is used by specific resources of the corresponding processor unit.

Example 16:

The computing system described in Embodiment 14 or 15 further includes a crossbar connector between the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with the vector memory to a specific resource among the plurality of processing resources of a particular processor unit.

Example 17:

The computing system described in any one of Embodiments 14 to 16 further includes a random number generator that communicates with the resources of a particular processor unit, the random number generator being configured to periodically generate numbers that can be used as operands for at least one operation performed by the particular processor unit.

Example 18:

The computing system described in any one of Embodiments 11 to 17 further includes a data path extending between the vector memory and the matrix operation unit, the data path enabling data communication associated with direct memory access operations occurring between the vector memory and at least the matrix operation unit.

Example 19:

A computer-implemented method in a computing system having a vector processing unit, the method comprising:

A vector memory provides data for performing one or more arithmetic operations, the vector memory including a storage body for storing the corresponding dataset;

One or more processor units receive data from a specific storage bank of the vector memory, the data being used by the one or more processor units to perform one or more arithmetic operations associated with vectorized computation; and

The matrix operation unit receives at least two operands from a specific processor unit, and the at least two operands are used by the matrix operation unit to perform operations associated with vectorized computation;

The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, such that data communication occurs with a first bandwidth based on a first distance between at least one processor unit and the vector memory;

The vector processing unit and the matrix operation unit are coupled such that data communication occurs with a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit; and

Wherein the first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth.

Example 20:

The computer-implemented method described in Example 19 further includes:

The serialized input data is provided by one of the first or second data serializers to at least one of the following: the matrix operation unit, the cross-channel unit, or the reduction and permutation unit, wherein the serialized input data includes a plurality of operands; and

The first data serializer and the second data serializer are disposed between the one or more processor units and at least one of the following: the matrix operation unit, the cross channel unit, or the reduction and permutation unit.

Claims (20)

1. A vector processing unit, comprising: Multiple processor units, each configured to perform arithmetic operations associated with vectorized computations of multidimensional data arrays; and A vector memory that communicates data with each of the plurality of processor units, wherein the vector memory includes a storage body configured to store data used by each of the plurality of processor units to perform the arithmetic operation; The plurality of processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling high-bandwidth data communication based on the tight coupling of the individual processor units relative to each other and the tight coupling of the vector memory relative to each processor unit. 2. The vector processing unit of claim 1, wherein the vector processing unit is configured to be coupled to a matrix operation unit, the matrix operation unit being configured to receive at least two operands from a particular processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorization computation of the multidimensional data array. 3. The vector processing unit of claim 2 further includes a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit. 4. The vector processing unit of claim 2 further includes a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of the following: the matrix operation unit, the cross-channel unit, and the reduction and permutation unit. 5. The vector processing unit of claim 1, wherein each of the plurality of processor units comprises: Multiple processing resources, the multiple processing resources including at least one of a first arithmetic logic unit, a second arithmetic logic unit, a multidimensional register, and a functional processor unit. 6. The vector processing unit of claim 1, wherein the vector memory is configured to load data associated with a particular memory bank into a corresponding processor unit, and wherein the data is used by a particular resource of the corresponding processor unit. 7. The vector processing unit of claim 1 further includes a crossbar connector between the plurality of processor units and the vector memory, the crossbar connector being configured to provide data associated with the vector memory to a specific resource among a plurality of processing resources of a particular processor unit. 8. The vector processing unit of claim 1 further includes a random number generator that communicates with the resources of a particular processor unit, the random number generator being configured to periodically generate numbers that can be used as operands for at least one operation performed by the particular processor unit. 9. The vector processing unit according to claim 1, wherein the vector processing unit provides a processing channel and includes a plurality of processor units, each of the plurality of processor units forming a processor sub-channel within the vector processing unit. 10. The vector processing unit of claim 9, wherein each processor subchannel is dynamically configured on a per-access basis to access a specific memory bank of the vector memory to retrieve data for performing one or more arithmetic operations associated with the vectorized computation of the multidimensional data array. 11. A computing system having a vector processing unit, the computing system comprising: One or more processor units, each processor unit including a first arithmetic logic unit configured to perform a plurality of arithmetic operations; A vector memory that communicates data with each of the one or more processor units, the vector memory including a storage bank configured to store data used by each of the one or more processor units to perform the arithmetic operation; and A matrix operation unit is configured to receive at least two operands from a specific processor unit, the at least two operands being used by the matrix operation unit to perform operations associated with vectorized computation; The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, enabling data communication to be exchanged with a first bandwidth based on a first distance between at least one processor unit and the vector memory; The vector processing unit and the matrix operation unit are coupled such that data communication can be exchanged with a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit; and Wherein the first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth. 12. The computing system of claim 11, further comprising a first data serializer coupled to the particular processor unit, the first data serializer being configured to serialize output data corresponding to one or more operands provided by the particular processor unit and received by the matrix operation unit. 13. The computing system of claim 12, further comprising a second data serializer coupled to the particular processor unit, the second data serializer being configured to serialize output data provided by the particular processor unit and received by at least one of the following: the matrix operation unit, the cross-channel unit, and the reduction and permutation unit. 14. The computing system of claim 11, wherein each of the one or more processor units further comprises a plurality of processing resources, the plurality of processing resources including at least one of a second arithmetic logic unit, a multidimensional register, and a functional processor unit. 15. The computing system of claim 14, wherein the vector memory is configured to load data associated with a particular memory bank into a corresponding processor unit, and wherein the data is used by a particular resource of the corresponding processor unit. 16. The computing system of claim 14, further comprising a crossbar connector between the one or more processor units and the vector memory, the crossbar connector being configured to provide data associated with the vector memory to a specific resource among the plurality of processing resources of a particular processor unit. 17. The computing system of claim 14, further comprising a random number generator for data communication with the resources of a particular processor unit, the random number generator being configured to periodically generate numbers that can be used as operands for at least one operation performed by the particular processor unit. 18. The computing system of claim 11 further includes a data path extending between the vector memory and the matrix operation unit, the data path enabling data communication associated with direct memory access operations occurring between the vector memory and at least the matrix operation unit. 19. A computer-implemented method executed in a computing system having a vector processing unit, the method comprising: A vector memory provides data for performing one or more arithmetic operations, the vector memory including a storage body for storing the corresponding dataset; One or more processor units receive data from a specific storage bank of the vector memory, the data being used by the one or more processor units to perform one or more arithmetic operations associated with vectorized computation; and The matrix operation unit receives at least two operands from a specific processor unit, and the at least two operands are used by the matrix operation unit to perform operations associated with vectorized computation; The one or more processor units and the vector memory are tightly coupled within the region of the vector processing unit, such that data communication occurs with a first bandwidth based on a first distance between at least one processor unit and the vector memory; The vector processing unit and the matrix operation unit are coupled such that data communication occurs with a second bandwidth based on a second distance between at least one processor unit and the matrix operation unit; and Wherein the first distance is less than the second distance, and the first bandwidth is greater than the second bandwidth. 20. The computer-implemented method according to claim 19, further comprising: The serialized input data is provided by one of the first and second data serializers to at least one of the following: the matrix operation unit, the cross-channel unit, and the reduction and permutation unit, wherein the serialized input data includes a plurality of operands; and The first data serializer and the second data serializer are disposed between the one or more processor units and at least one of the following: the matrix operation unit, the cross channel unit, and the reduction and permutation unit.