WO2023128009A1 - Neural processing device and synchronization method thereof - Google Patents

Abstract

A neural processing device and a synchronization method thereof are disclosed. The neural processing device comprises: first and second neural processors; a shared memory shared by the first and second neural processors; first and second semaphore memories that correspond to the first and second neural processors, respectively, and receive and store an L3 sync target, wherein synchronization between the first and second neural processors is performed according to the L3 sync target; and a global interconnection that connects the first and second neural processors to the shared memory, and includes an L3 sync channel through which a synchronization signal corresponding to the L3 sync target is transmitted.

Description

Neural Processing Apparatus and Synchronization Method Thereof

The present invention relates to a neural processing apparatus and a synchronization method thereof. Specifically, the present invention relates to a neural processing apparatus and a synchronization method thereof in which each processor performs synchronization instead of a central control processor.

In the past few years, artificial intelligence (AI) technology has been discussed as the most promising technology worldwide as a core technology of the 4th industrial revolution. The biggest problem with these artificial intelligence technologies is computing power. The most important thing in artificial intelligence technology that realizes human learning, reasoning, perception, and natural language implementation is to process a lot of data quickly.

Although the central processing unit (CPU) or graphics processing unit (GPU) of an off-the-shelf computer was used for deep learning learning and reasoning of early artificial intelligence, deep learning learning and reasoning tasks with high workloads were used. has limitations, so a neural processing unit (NPU) structurally specialized for deep learning tasks is in the limelight.

Such a neural network processing device includes a large number of processing units and cores therein, and synchronization of these modules is a part that must be clearly handled according to task dependencies. In existing processing devices, a control processor (or centralized controller) centrally controls these synchronization signals and manages operations in accordance with the order.

However, in this method, as more and more processing units and cores are included in the neural network processing device, a lot of latency may occur in synchronization processing and overhead of a control processor may increase.

Prior art literature: Patent Registration No. 10-2258566

An object of the present invention is to provide a neural processing apparatus capable of fast and efficient synchronization processing.

Another object of the present invention is to provide a method for synchronizing a neural processing device capable of fast and efficient synchronization processing.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

In order to solve the above problem, a neural processing apparatus according to some embodiments of the present invention provides first and second neural processors, a shared memory shared by the first and second neural processors, and each of the first and second neural processors. first and second semaphore memories corresponding to each other and receiving and storing an L3 sync target, wherein synchronization of the first and second neural processors is performed according to the L3 sync target; and a global interconnection including an L3 sync channel connecting the second neural processor and the shared memory and transmitting a synchronization signal according to the L3 sync target.

In addition, the global interconnection includes a data channel for transmitting data between the L3 sync channel, the shared memory, and the first and second neural processors, and a control signal for transmitting a control signal to the first and second neural processors. Can contain control channels.

Also, the first semaphore memory may include first and second fields respectively corresponding to the first and second neural processors.

The method may further include a first FIFO buffer that sequentially transfers values of the first field to the first neural processor.

The processor may further include a second FIFO buffer that sequentially transfers values of the second field to the first neural processor.

In addition, the L3 sync target includes first and second L3 sync targets, the first neural processor generates the first L3 sync target, and the second neural processor generates the second L3 sync target. can

In addition, the L3 sync target includes first and second sync target fields corresponding to the first and second neural processors, respectively, and the first and second sync target fields are configured by the first and second neural processors. It may include information on whether a synchronization signal according to the L3 sync target is received.

Also, the first and second sync target fields may be arranged in the order of virtual IDs of the first and second neural processors, respectively.

In addition, the first neural processor identifies a physical ID of a neural processor receiving a synchronization signal according to the L3 sync target by using the L3 sync target and the VPID table, and the VPID table identifies the virtual ID and the physical ID. It may contain information for conversion.

Also, when the first and second neural processors execute the same program, the physical ID of the first and second neural processors and the virtual ID of the first and second neural processors may correspond 1:1 to each other. can

In addition, the L3 sync target may be included in an instruction set architecture (ISA).

Also, the neural processor may include at least one neural core and a local interconnection for transmitting data between the at least one neural core.

In addition, the neural processor may further include an L2 sync pass through which a synchronization signal according to an L2 sync target for performing synchronization between the at least one neural core is transmitted.

Also, the L2 sync target may be included in the instruction set structure.

In addition, each of the at least one neural core includes a processing unit that receives input activations and weights, performs deep learning operations, and outputs output activations, and a local area that temporarily stores the input activations, the weights, and the output activations. may contain memory.

In addition, each of the at least one neural core may further include an activation buffer temporarily storing the input activation and the output activation between the processing unit and the local memory.

In addition, each of the at least one neural core may further include an activation load/store unit (LSU) for moving the input activation and the output activation between the activation buffer and the local memory.

In addition, each of the at least one neural core may further include a weight buffer temporarily storing the weight between the local memory and the processing unit.

In addition, each of the at least one neural core may further include an LSU for moving data between the local memory and the local interconnection.

In addition, the LSU may include a local memory store unit for storing the local memory and a local memory load unit for loading the local memory.

In addition, the LSU may include a neural core store unit that performs external storage in the neural core and a neural core load unit that externally loads the neural core.

In addition, an L1 sync pass through which a synchronization signal according to an L1 sync target for synchronization between the local memory and other elements is transmitted may be further included.

Also, the L1 sync target may be included in the instruction set structure.

Also, the processing unit may include a processing element array that performs a two-dimensional operation and a vector unit that performs a one-dimensional operation.

In addition, the processing unit includes a row register for supplying a first input to each row of the array of processing elements and a column register for receiving and supplying a second input to each column of the array of processing elements. Further, the first and second inputs may each include any one of weight and input activation.

In order to solve the above problem, a neural processing apparatus according to some embodiments of the present invention connects at least one neural processor, a shared memory, and the at least one neural processor and a shared memory, and a global used for L3 synchronization of the neural processor. an interconnection, wherein the neural processor includes at least one neural core, a local interconnection connecting the at least one neural core, and an L2 sync pass used for L2 synchronization of the at least one neural core; The neural core includes a processing unit that performs arithmetic operations, a local memory that temporarily stores data, and an L1 sync pass used for L1 synchronization of the local memory and the processing unit.

In addition, the global interconnection includes a data channel for transmitting data between the at least one neural processor and the shared memory, a control channel for transmitting a control signal between the at least one neural processor, and used for L3 synchronization. A sync channel may be included.

In addition, the neural processor may further include a local interconnection for transmitting data between the at least one neural core.

In addition, a data path used for data exchange between the local memory and elements including the processing unit may be included.

Also, the processing unit may include a processing element array that performs a two-dimensional operation and a vector unit that performs a one-dimensional operation.

Also, the processing element array includes a plurality of processing elements arranged in rows and columns, and each of the plurality of processing elements may perform multiplication.

In addition, the at least one neural processor includes first and second neural processors, and first and second neural processors receive and store a synchronization signal corresponding to the first and second neural processors and corresponding to an L3 sync target. The second semaphore memory may further include first and second semaphore memories in which synchronization of the first and second neural processors is performed according to the L3 sync target.

In addition, the first semaphore memory includes first and second fields respectively corresponding to the first and second neural processors, and sequentially transfers values of the first fields to the first neural processor. A first FIFO It may contain more buffers.

In addition, the first neural processor transmits an instruction set structure, and the instruction set structure includes an operation code, an L3 sync target for the L3 synchronization, an L2 sync target for the L2 synchronization, and an L1 sync target for the L1 synchronization. can include

In order to solve the other problem, a method for synchronizing a neural processing device according to some embodiments of the present invention is a method for synchronizing a neural processing device including first and second neural processors, wherein the first neural processor performs L3 synchronization. generating an L3 sync target for the first and second neural processors, the L3 sync target being arranged in order of virtual IDs of the first and second neural processors, and identifying a physical ID of the second neural processor using the L3 sync target and a VPID table; , The VPID table is a conversion table of the virtual ID and the physical ID of the neural processor, and a synchronization signal according to the L3 sync target is transmitted to the first semaphore memory of the second neural processor through the L3 sync channel of the global interconnection. and performing the L3 synchronization according to the value of the first semaphore memory by the second neural processor.

Also, the first semaphore memory may include first and second fields respectively corresponding to the first and second neural processors.

The performing of the L3 synchronization may include providing a value of the first field to the second neural processor in a FIFO manner and providing a value of the second field to the second neural processor in a FIFO manner. can

Also, the virtual ID may include first and second virtual IDs respectively corresponding to the first and second neural processors.

The first neural processor may also configure first and second neural cores, a local interconnection for transmitting data between the first and second neural cores, and a synchronization signal between the first and second neural cores. It may include an L2 sync pass for transmitting a synchronization signal according to an L2 sync target for

In addition, the first neural core includes a first processing unit that receives a first input activation and a first weight, performs a deep learning operation, and outputs a first output activation; the first input activation and the first weight; and a first local memory temporarily storing the first output activation and a first L1 sync pass transmitting a synchronization signal according to an L1 sync target for synchronization between the first local memory and the first processing unit. and the second neural core comprises a second processing unit that receives second input activation and a second weight, performs a deep learning operation, and outputs a second output activation; the second input activation and the second weight; and a second L1 sync pass for transmitting a second local memory temporarily storing the second output activation and a synchronization signal according to the L1 sync target for synchronization between the second local memory and the second processing unit. can include

In addition, data is stored in the first local memory, a synchronization signal according to the L1 sync target is transmitted through the first L1 sync pass in the first neural core, and the first neural core transmits a synchronization signal according to the second L1 sync pass. The method may further include transmitting a synchronization signal according to the L2 sync target to the second neural core through an L2 sync pass, and the second neural core receiving data through the local interconnection.

The first neural core may further include a first LSU for moving data between the first local memory and the first local interconnection, and the second neural core may include the second local memory and the second local interconnection. It may further include a second LSU for moving data between interconnections.

In addition, the first LSU includes a first local memory store unit for storing the first local memory, a first local memory load unit for loading the first local memory, and the first neural core. It may include a first neural core store unit that performs external storage and a first neural core load unit that externally loads the first neural core.

In addition, in the first neural core, transmitting the synchronization signal according to the L1 sync target through the first L1 sync pass causes the first local memory store unit to store the L1 sync target in the first neural core store. Can include sending to units.

In addition, the second LSU includes a second local memory store unit for storing the second local memory, a second local memory load unit for loading the second local memory, and the second neural core. A second neural core store unit that performs external storage and a second neural core load unit that externally loads the second neural core.

Transmitting the synchronization signal according to the L2 sync target may include the first neural core store unit transmitting the synchronization signal according to the L2 sync target to the second neural core load unit.

In addition, when the second neural core receives data, the second neural core load unit requests the data from the first local memory through the local interconnection, and the second neural core load unit receives the data from the local interconnection. and receiving the data from the first local memory through an interconnection.

A synchronization method of a neural processing apparatus according to some embodiments of the present invention for solving the above other problems includes first and second neural cores, a local interconnection connecting the first and second neural cores, and the first and second neural cores. and an L2 sync pass used for L2 synchronization of a second neural core, wherein the first neural core includes a first processing unit that performs a calculation task and data input/output to the first processing unit. A first local memory for temporarily storing and a first L1 sync pass used for L1 synchronization of the first local memory and the first processing unit, wherein the second neural core includes a second neural core that performs calculation tasks. A neural circuit comprising a processing unit, a second local memory temporarily storing data input/output to the second processing unit, and a second L1 sync pass used for L1 synchronization of the second local memory and the second processing unit. A synchronization method of a processing device, storing data in the first local memory, transmitting a synchronization signal according to the L1 sync target through the first L1 sync pass in the first neural core, and The neural core may transmit a synchronization signal according to the L2 sync target to the second neural core through the second L2 sync pass, and the second neural core may receive data through the local interconnection. .

The first neural core may further include a first LSU for moving data between the first local memory and the first local interconnection, wherein the first LSU performs storage of the first local memory. A first local memory store unit, and a first neural core store unit that performs storage externally from the first neural core, wherein the L1 sync target is included in the first neural core through the first L1 sync pass. Transmitting the synchronization signal according to may include transmitting, by the first local memory store unit, a synchronization signal according to the L1 sync target to the first neural core store unit.

In addition, the second neural core further includes a second LSU for moving data between the second local memory and the second local interconnection, and the second LSU includes an external load in the second neural core. and transmitting the synchronization signal according to the L2 sync target means that the first neural core store unit transmits the synchronization signal according to the L2 sync target to the second neural core load unit. It may include sending to

The neural processing apparatus may further include: a first neural processor including the first and second neural cores, the local interconnection, and the L2 sync path; a second neural processor different from the first neural processor; a global interconnection for transmitting data between the first and second neural processors, and first and second semaphore memories respectively corresponding to the first and second neural processors, wherein the global interconnection comprises the first and second semaphore memories; and a data channel, a control channel, and an L3 sync channel through which data, a control signal, and a synchronization signal according to an L3 sync target are respectively transmitted between a second neural processor, wherein the first neural processor generates the L3 sync target; The L3 sync target may be stored in the second semaphore memory, and the second neural processor may perform synchronization through a synchronization signal according to the L3 sync target.

The neural processing apparatus and synchronization method thereof of the present invention can minimize latency according to the synchronization request transmitted to the control processor because each processor, core, and memory elements transmit synchronization requests to each other to perform synchronization instead of a centralized control processor. there is.

In addition, since there is no need to perform scheduling work performed by the control processor, scheduling overhead of the neural processing apparatus can be greatly reduced.

In addition to the above description, specific effects of the present invention will be described together while explaining specific details for carrying out the present invention.

1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.

FIG. 2 is a block diagram for explaining the neural processing apparatus of FIG. 1 in detail.

FIG. 3 is a block diagram illustrating the neural core SoC of FIG. 2 in detail.

FIG. 4 is a structural diagram for explaining the global interconnection of FIG. 3 in detail.

FIG. 5 is a block diagram for explaining the neural processor of FIG. 3 in detail.

FIG. 6 is a block diagram for explaining the neural core of FIG. 5 in detail.

FIG. 7 is a block diagram for explaining the LSU of FIG. 6 in detail.

FIG. 8 is a block diagram for explaining the processing unit of FIG. 6 in detail.

FIG. 9 is a block diagram for explaining the local memory of FIG. 6 in detail.

FIG. 10 is a block diagram for explaining the local memory bank of FIG. 9 in detail.

11 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention.

12 is a block diagram illustrating an example of memory reorganization of a neural processing system in accordance with some embodiments of the present invention.

FIG. 13 is an enlarged block diagram of part A of FIG. 11 .

FIG. 14 is a diagram for explaining the first bank of FIG. 13 in detail.

15 is a conceptual diagram for explaining virtual ID assignment of a neural processing apparatus according to some embodiments of the present invention.

16 is a diagram for explaining a virtual ID allocation and a VPID table of a neural processing apparatus according to some embodiments of the present invention.

17 is a diagram for explaining a process of identifying a physical ID through a sync target and a VPID table.

18 is a directed acyclic graph for explaining the sequence of deep learning tasks.

19 is a conceptual diagram illustrating a synchronization signal transmission operation according to a sync target for L3 synchronization of a neural processing apparatus according to some embodiments of the present invention.

20 is a conceptual diagram for explaining a synchronization signal reception operation according to a sync target for L3 synchronization of a neural processing apparatus according to some embodiments of the present invention.

21 is a block diagram for explaining L1 and L2 synchronization of a neural processing apparatus according to some embodiments of the present invention.

22 is a ladder diagram for explaining L1 and L2 synchronization of a neural processing apparatus according to some embodiments of the present invention.

23 is a diagram for explaining an instruction set structure of a neural processing apparatus according to some embodiments of the present invention.

24 is a block diagram for explaining a software layer structure of a neural processing apparatus according to some embodiments of the present invention.

25 is a conceptual diagram for explaining a deep learning operation performed by a neural processing apparatus according to some embodiments of the present invention.

26 is a conceptual diagram for explaining learning and reasoning operations of a neural network of a neural processing apparatus according to some embodiments of the present invention.

27 is a flowchart illustrating a synchronization method of a neural processing apparatus according to some embodiments of the present invention.

FIG. 28 is a flowchart for explaining in detail the step of storing the L3 sync target of FIG. 27 and the step of providing it in a FIFO method.

29 is a flowchart illustrating a method for synchronizing L1 and L2 levels of a neural processing apparatus according to some embodiments of the present invention.

30 is a flowchart for explaining the data request step of FIG. 29 in detail.

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. According to the principle that an inventor may define a term or a concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical spirit of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all of the technical spirit of the present invention, so they can be replaced at the time of the present application. It should be understood that there may be many equivalents and variations and applicable examples.

Terms such as first, second, A, and B used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

Terms used in this specification and claims are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "include" or "having" in this application do not exclude in advance the possibility of existence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, it should not be interpreted in an ideal or excessively formal meaning. don't

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not contradict each other technically.

Hereinafter, a neural processing apparatus according to some embodiments of the present invention will be described with reference to FIGS. 1 to 26 .

1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.

Referring to FIG. 1 , a neural processing system (NPS) according to some embodiments of the present invention may include a first neural processing device 1, a second neural processing device 2, and an external interface 3. .

The first neural processing device 1 may be a device that performs calculations using an artificial neural network. The first neural processing device 1 may be, for example, a device specialized for performing a deep learning calculation task. However, this embodiment is not limited thereto.

The second neural processing device 2 may have a configuration identical to or similar to that of the first neural processing device 1 . The first neural processing device 1 and the second neural processing device 2 may be connected to each other through an external interface 3 to share data and control signals.

Although FIG. 1 shows two neural processing devices, a neural processing system (NPS) according to some embodiments of the present invention is not limited thereto. That is, in the neural processing system (NPS) according to some embodiments of the present invention, three or more neural processing devices may be connected to each other through the external interface 3. Also, conversely, a neural processing system (NPS) according to some embodiments of the present invention may include only one neural processing device.

FIG. 2 is a block diagram for explaining the neural processing apparatus of FIG. 1 in detail.

Referring to FIG. 2 , the first neural processing device 1 includes a neural core SoC 10, a CPU 20, an off-chip memory 30, a first non-volatile memory interface 40, and a first volatile memory interface ( 50), a second non-volatile memory interface 60 and a second volatile memory interface 70.

The neural core SoC 10 may be a system on chip device. The neural core SoC 10 is an artificial intelligence computing device and may be an accelerator. The neural core SoC 10 may be, for example, any one of a graphics processing unit (GPU), a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC). However, this embodiment is not limited thereto.

The neural core SoC 10 may exchange data with other external computing devices through the external interface 3 . In addition, the neural core SoC 10 may be connected to the non-volatile memory 31 and the volatile memory 32 through the first non-volatile memory interface 40 and the first volatile memory interface 50 , respectively.

The CPU 20 may be a controller that controls the system of the first neural processing device 1 and executes program operations. The CPU 20 is a general purpose arithmetic unit and may have low efficiency to perform parallel simple arithmetic operations frequently used in deep learning. Accordingly, the neural core SoC 10 may perform calculations for deep learning reasoning and learning tasks, thereby achieving high efficiency.

The CPU 20 may exchange data with other external computing devices through the external interface 3 . In addition, the CPU 20 may be connected to the non-volatile memory 31 and the volatile memory 32 through the second non-volatile memory interface 60 and the second volatile memory interface 70, respectively.

The off-chip memory 30 may be a memory disposed outside a chip of the neural core SoC 10 . The off-chip memory 30 may include a non-volatile memory 31 and a volatile memory 32 .

The non-volatile memory 31 may be a memory that continuously retains stored information even when power is not supplied. The non-volatile memory 31 includes, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable alterable ROM (EAROM), an erasable programmable read-only memory (EPROM), and an electrically erasable programmable memory (EEPROM). Read-Only Memory) (e.g., NAND Flash memory, NOR Flash memory), Ultra-Violet Erasable Programmable Read-Only Memory (UVEPROM), Ferroelectric Random Access Memory (FeRAM), MRAM (Magnetoresistive Random Access Memory), PRAM (Phase-change Random Access Memory), SONOS (silicon-oxide-nitride-oxide-silicon), RRAM (Resistive Random Access Memory), NRAM (Nanotube Random Access Memory), magnetic computer memory It may include at least one of a device (eg, hard disk, diskette drive, magnetic tape), an optical disk drive, and a 3D XPoint memory. However, this embodiment is not limited thereto.

Unlike the non-volatile memory 31, the volatile memory 32 may be a memory that continuously requires power to maintain stored information. The volatile memory 32 may include, for example, at least one of Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Synchronous Dynamic Random Access Memory (SDRAM), and Double Data Rate SDRAM (DDR SDRAM). there is. However, this embodiment is not limited thereto.

The first non-volatile memory interface 40 and the second non-volatile memory interface 60 may be, for example, PATA (Parallel Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), SATA ( Serial Advanced Technology Attachment) and PCI Express (PCIe). However, this embodiment is not limited thereto.

The first volatile memory interface 50 and the second volatile memory interface 70 are respectively, for example, SDR (Single Data Rate), DDR (Double Data Rate), QDR (Quad Data Rate), and XDR (eXtreme Data Rate) , Octal Data Rate). However, this embodiment is not limited thereto.

FIG. 3 is a block diagram illustrating the neural core SoC of FIG. 2 in detail.

2 and 3, the neural core SoC 10 includes at least one neural processor 1000, a shared memory 2000, a direct memory access (DMA) 3000, a non-volatile memory controller 4000, a volatile A memory controller 5000 and a global interconnection 6000 may be included.

The neural processor 1000 may be an arithmetic device that directly performs an arithmetic task. When there are a plurality of neural processors 1000 , calculation tasks may be allocated to each of the neural processors 1000 . Each of the neural processors 1000 may be connected to each other through the global interconnection 6000 .

The shared memory 2000 may be a memory shared by several neural processors 1000 . The shared memory 2000 may store data of each neural processor 1000 . In addition, the shared memory 2000 may receive data from the off-chip memory 30, temporarily store the data, and transfer the data to each neural processor 1000. Conversely, the shared memory 2000 may receive data from the neural processor 1000, temporarily store the data, and transfer the data to the off-chip memory 30 of FIG. 2 .

The shared memory 2000 may require a relatively fast memory. Accordingly, the shared memory 2000 may include SRAM, for example. However, this embodiment is not limited thereto. That is, the shared memory 2000 may include DRAM.

The shared memory 2000 may be a memory corresponding to an SoC level, that is, level 3 (L3). Accordingly, the shared memory 2000 may be defined as an L3 shared memory.

The DMA 3000 can directly control the movement of data without the need for the neural processor 1000 to control input/output of data. Accordingly, the number of interrupts of the neural processor 1000 can be minimized by the DMA 3000 controlling data movement between memories.

The DMA 3000 may control movement of data between the shared memory 2000 and the off-chip memory 30 . The non-volatile memory controller 4000 and the volatile memory controller 5000 may transfer data through the authority of the DMA 3000 .

The nonvolatile memory controller 4000 may control a read or write operation of the nonvolatile memory 31 . The nonvolatile memory controller 4000 may control the nonvolatile memory 31 through the first nonvolatile memory interface 40 .

The volatile memory controller 5000 may control a read or write operation of the volatile memory 32 . Also, the volatile memory controller 5000 may perform a refresh operation of the volatile memory 32 . The volatile memory controller 5000 may control the non-volatile memory 31 through the first volatile memory interface 50 .

The global interconnection 6000 may connect at least one of the neural processor 1000 , the shared memory 2000 , the DMA 3000 , the nonvolatile memory controller 4000 and the volatile memory controller 5000 to each other. In addition, the external interface 3 may also be connected to the global interconnection 6000 . The global interconnection 6000 is data communication between at least one neural processor 1000, a shared memory 2000, a DMA 3000, a non-volatile memory controller 4000, a volatile memory controller 5000, and an external interface 3. may be a moving path.

The global interconnection 6000 may transmit data as well as control signals and signals for synchronization. That is, in the neural processing apparatus according to some embodiments of the present invention, each neural processor 1000 may directly transmit and receive a synchronization signal, rather than a separate control processor managing synchronization signals. Accordingly, it is possible to block the latency of the synchronization signal generated by the control processor.

That is, when there are a plurality of neural processors 1000, there may be dependencies of individual tasks in which the task of one neural processor 1000 must be completed before the next neural processor 1000 can start a new task. The end and start of these individual tasks can be confirmed through a synchronization signal, and in the conventional technology, the control processor performs the reception of the synchronization signal and the instruction to start a new task.

However, as the number of neural processors 1000 increases and the dependencies of tasks are designed more complexly, the number of requests and instructions for synchronization tasks increases exponentially. Therefore, the latency according to each request and instruction can greatly reduce work efficiency.

Accordingly, in the neural processing apparatus according to some embodiments of the present invention, each neural processor 1000 may directly transmit a synchronization signal to other neural processors 1000 according to task dependencies instead of a control processor. In this case, compared to the method managed by the control processor, several neural processors 1000 can perform synchronization tasks in parallel, so that latency due to synchronization can be minimized.

In addition, the control processor needs to perform task scheduling of the neural processors 1000 according to task dependencies, and the overhead of such scheduling can greatly increase as the number of the neural processors 1000 increases. Therefore, in the neural processing device according to some embodiments of the present invention, the scheduling task is also performed by the individual neural processor 1000, and thus the performance of the device can be improved without the scheduling burden.

FIG. 4 is a structural diagram for explaining the global interconnection of FIG. 3 in detail.

Referring to FIG. 4 , a global interconnection 6000 may include a data channel 6100 , a control channel 6200 and an L3 sync channel 6300 .

The data channel 6100 may be a dedicated channel for transmitting data. At least one of the neural processor 1000, the shared memory 2000, the DMA 3000, the non-volatile memory controller 4000, the volatile memory controller 5000, and the external interface 3 communicate data to each other through the data channel 6100. can be exchanged.

The control channel 6200 may be a dedicated channel for transmitting a control signal. At least one of the neural processor 1000, the shared memory 2000, the DMA 3000, the non-volatile memory controller 4000, the volatile memory controller 5000, and the external interface 3 control each other through the control channel 6200 signals can be exchanged.

The L3 sync channel 6300 may be a dedicated channel for transmitting a synchronization signal. At least one neural processor 1000, a shared memory 2000, a DMA 3000, a non-volatile memory controller 4000, a volatile memory controller 5000, and an external interface 3 communicate with each other through the L3 sync channel 6300. Synchronization signals can be exchanged.

The L3 sync channel 6300 is set as a dedicated channel within the global interconnection 6000 so that a synchronization signal can be quickly transmitted without overlapping with other channels. Accordingly, the neural processing apparatus according to some embodiments of the present invention does not require a new wiring work and can smoothly perform synchronization work using the existing global interconnection 6000 .

FIG. 5 is a block diagram for explaining the neural processor of FIG. 3 in detail.

3 to 5 , the neural processor 1000 may include at least one neural core 100, an L2 shared memory 400, a local interconnection 200, and an L2 sync path 300.

At least one neural core 100 may divide and perform tasks of the neural processor 1000 . The number of neural cores 100 may be, for example, 8. However, this embodiment is not limited thereto. 4 and 5 show that several neural cores 100 are included in the neural processor 1000, but the present embodiment is not limited thereto. That is, the neural processor 1000 may be configured with only one neural core 100 .

The L2 shared memory 400 may be a memory shared by each of the neural cores 100 in the neural processor 1000 . The L2 shared memory 400 may store data of each neural core 100 . In addition, the L2 shared memory 400 may receive data from the shared memory 2000 of FIG. 3 , temporarily store the data, and transfer the data to each neural core 100 . Conversely, the L2 shared memory 400 may receive data from the neural core 100, temporarily store the data, and transfer the data to the shared memory 2000 of FIG. 3 .

The L2 shared memory 400 may be memory corresponding to a neural processor level, that is, level 2 (L2). The L3 shared memory, that is, the shared memory 2000 may be shared by the neural processor 1000 and the L2 shared memory 400 may be shared by the neural core 100 .

The local interconnection 200 may connect at least one neural core 100 and the L2 shared memory 400 to each other. The local interconnection 200 may be a path through which data moves between at least one neural core 100 and the L2 shared memory 400 . The local interconnection 200 may be connected to the global interconnection 6000 of FIG. 3 to transmit data.

The L2 sync pass 300 may connect at least one neural core 100 and the L2 shared memory 400 to each other. The L2 sync path 300 may be a path along which synchronization signals of at least one neural core 100 and the L2 shared memory 400 move.

The L2 sync path 300 may be formed physically separately from the local interconnection 200 . In the case of the local interconnection 200, unlike the global interconnection 6000, sufficient internal channels may not be formed. In this case, the L2 sync path 300 is formed separately so that synchronization signal transmission can be performed quickly and without delay. The L2 sync pass 300 may be used for synchronization performed at a level lower than that of the L3 sync channel 6300 of the global interconnection 6000.

FIG. 6 is a block diagram for explaining the neural core of FIG. 5 in detail.

Referring to FIG. 6 , the neural core 100 includes a load/store unit (LSU) 110, a local memory 120, a weight buffer 130, an activation LSU 140, an activation buffer 150, and a processing unit ( 160) may be included.

The LSU 110 may receive at least one of data, a control signal, and a synchronization signal from the outside through the local interconnection 200 and the L2 sync path 300 . The LSU 110 may transmit at least one of received data, a control signal, and a synchronization signal to the local memory 120 . Similarly, the LSU 110 may transfer at least one of data, control signals, and synchronization signals to the outside through the local interconnection 200 and the L2 sync path 300 .

FIG. 7 is a block diagram for explaining the LSU of FIG. 6 in detail.

Referring to FIG. 7 , the LSU 110 includes a local memory load unit 111a, a local memory store unit 111b, a neural core load unit 112a, a neural core store unit 112b, a load buffer LB, and a store. It may include a buffer SB, a load engine 113a, a store engine 113b and a conversion lookaside buffer 114.

The local memory load unit 111a may fetch a load instruction for the local memory 120 and issue the load instruction. When the local memory load unit 111a provides an issued load instruction to the load buffer LB, the load buffer LB may sequentially transmit memory access requests to the load engine 113a according to the input order.

Also, the local memory store unit 111b may fetch a store instruction for the local memory 120 and issue the store instruction. When the local memory store unit 111b provides the store instruction at issue to the store buffer SB, the store buffer SB may sequentially transmit memory access requests to the store engine 113b according to the input order.

The neural core load unit 112a may fetch load instructions for the neural core 100 and issue load instructions. When the neural core load unit 112a provides the issued load instruction to the load buffer LB, the load buffer LB may sequentially transmit memory access requests to the load engine 113a according to the input order.

Also, the neural core store unit 112b may fetch a store instruction for the neural core 100 and issue the store instruction. When the neural core store unit 112b provides the stored instruction to the store buffer SB, the store buffer SB may sequentially transmit memory access requests to the store engine 113b according to the input order.

The load engine 113a may receive a memory access request and load data through the local interconnection 200 . At this time, the load engine 113a can quickly find data by using a translation table of recently used virtual addresses and physical addresses in the translation lookaside buffer 114 . When the virtual address of the load engine 113a does not exist in the translation lookaside buffer 114, address translation information may be found in another memory.

The store engine 113b may receive a memory access request and load data through the local interconnection 200 . At this time, the store engine 113b can quickly find data by using a translation table of recently used virtual addresses and physical addresses in the translation lookaside buffer 114 . When the virtual address of the store engine 113b does not exist in the translation lookaside buffer 114, address translation information may be found in another memory.

The load engine 113a and the store engine 113b may send synchronization signals to the L2 sync path 300 . At this time, the synchronization signal may have a meaning that the work is finished.

Again, referring to FIG. 6 , the local memory 120 is a memory located inside the neural core 100 and can receive all input data necessary for the neural core 100 to work from the outside and temporarily store them. In addition, the local memory 120 may temporarily store output data calculated by the neural core 100 in order to transmit them to the outside. The local memory 120 may serve as a cache memory of the neural core 100 .

The local memory 120 may transmit the input activation (Act_In) to the activation buffer 150 and receive the output activation (Act_Out) by the activation LSU 140 . In addition to the activation LSU 140 , the local memory 120 may directly transmit/receive data with the processing unit 160 . That is, the local memory 120 may exchange data with each of the PE array 163 and the vector unit 164 .

The local memory 120 may be a memory corresponding to a neural core level, that is, level 1 (L1). Accordingly, the local memory 120 may also be defined as an L1 memory. However, unlike the L2 shared memory 400 and the L3 shared memory, that is, the shared memory 2000, the L1 memory is not shared and may be a private memory of the neural core.

The local memory 120 may transmit data such as activation or weight through a data path. The local memory 120 may transmit and receive synchronization signals through an L1 sync path, which is a separate dedicated path. The local memory 120 may exchange synchronization signals with, for example, the LSU 110, the weight buffer 130, the activation LSU 140, and the processing unit 160 through an L1 sync path. .

The weight buffer 130 may receive a weight from the local memory 120 . The weight buffer 130 may transfer the weight to the processing unit 160 . The weight buffer 130 may temporarily store weights before transferring them.

The input activation (Act_In) and the output activation (Act_Out) may refer to an input value and an output value of a layer of a neural network. In this case, when the neural network has a plurality of layers, the output activation value of the previous layer becomes the input value of the next layer, so the output activation (Act_Out) of the previous layer may be used as the input activation (Act_In) of the next layer.

The weight may mean a parameter that is multiplied with the input activation (Act_In) input in each layer. The weight is adjusted and determined in the deep learning step, and may be used to derive the output activation (Act_Out) through a fixed value in the inference step.

The activation LSU 140 may transfer an input activation (Act_In) from the local memory 120 to the activation buffer 150 and transfer an output activation (Act_Out) from the activation buffer 150 to the on-chip buffer. That is, the activation LSU 140 may perform both a load operation and a store operation of activation.

The activation buffer 150 may provide an input activation (Act_In) to the processing unit 160 and receive an output activation (Act_Out) from the processing unit 160. The activation buffer 150 may temporarily store input activation (Act_In) and output activation (Act_Out).

The activation buffer 150 can quickly provide activation to the processing unit 160 , particularly the PE array 163 , which requires a large amount of computation, and quickly receive the activation, thereby increasing the computational speed of the neural core 100 .

The processing unit 160 may be a module that performs calculations. The processing unit 160 may perform not only a 1-dimensional operation but also a 2-dimensional matrix operation, that is, a convolution operation. The processing unit 160 may generate an output activation (Act_Out) by receiving the input activation (Act_In), multiplying the received input activation (Act_In), and then adding the result.

FIG. 8 is a block diagram for explaining the processing unit of FIG. 6 in detail.

Referring to FIGS. 6 and 8 , the processing unit 160 may include a PE array 163 , a vector unit 164 , a column register 161 and a row register 162 .

The PE array 163 may perform multiplication by receiving the input activation (Act_In) and the weight (Weight). In this case, the input activation (Act_In) and the weight (Weight) may be calculated through convolution in the form of a matrix. Through this, the PE array 163 may generate an output activation (Act_Out). However, this embodiment is not limited thereto. The PE array 163 can also generate other types of outputs other than the output activation (Act_Out).

The PE array 163 may include at least one processing element (PE). The processing elements PE may be aligned with each other to perform multiplication of one input activation (Act_In) and one weight (Weight).

The PE array 163 may produce a subtotal sum of values for each multiplication. This subtotal can be utilized as an output activation (Act_Out). Since the PE array 163 performs 2D matrix multiplication, it may also be referred to as a 2D matrix compute unit.

The vector unit 164 may perform primarily one-dimensional operations. The vector unit 164 may perform deep learning operations together with the PE array 163 . Through this, the processing unit 160 may be specialized for necessary operations. That is, the neural core 100 can efficiently perform deep learning tasks because each of the calculation modules that performs a large amount of 2D matrix multiplication and 1D calculation is performed.

The column register 161 may receive the first input I1. The column register 161 may receive the first input I1, divide it, and provide it to each column of the processing element PE.

The low register 162 may receive the second input I2. The row register 162 may receive the second input I2, divide it, and provide it to each row of the processing element PE.

The first input (I1) may be an input activation (Act_In) or a weight (Weight). The second input I2 may be a value other than the first input I1 among input activation Act_In or weight. Alternatively, the first input I1 and the second input I2 may be values other than the input activation Act_In and the weight.

FIG. 9 is a block diagram for explaining the local memory of FIG. 6 in detail.

Referring to FIG. 9 , the local memory 120 may include a scheduler 121 and at least one local memory bank 122 .

When data is stored in the local memory 120, the scheduler 121 may receive the data from the load engine 113a. At this time, data may be allocated to the local memory bank 122 in a round robin manner. Accordingly, data may be stored in any one of the at least one local memory bank 122 .

Conversely, when data is loaded from the local memory 120, the scheduler 121 may receive data from the local memory bank 122 and transfer the data to the store engine 113b. The store engine 113b may store data externally through the local interconnection 200 .

FIG. 10 is a block diagram for explaining the local memory bank of FIG. 9 in detail.

Referring to FIG. 10 , the local memory bank 122 may include a local memory bank controller 122_1 and a local memory bank cell array 122_2.

The local memory bank controller 122_1 may manage read and write operations through addresses of data stored in the local memory bank 122 . That is, the local memory bank controller 122_1 may manage the input/output of data as a whole.

The local memory bank cell array 122_2 may have a structure in which cells directly storing data are aligned in rows and columns. The local memory bank cell array 122_2 may be controlled by the local memory bank controller 122_1.

11 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention.

Referring to FIG. 11 , the neural core SoC 10 may include first to eighth neural cores 100a to 100h and an on-chip memory (OCM). 11 shows 8 neural cores as an example, but this is just an example and the number of neural cores may vary.

The on-chip memory OCM may include first to eighth local memories 120a to 120h and a shared memory 2000 .

The first to eighth local memories 120a to 120h may be used as dedicated memories of the first to eighth neural cores 100a to 100h, respectively. That is, the first to eighth neural cores 100a to 100h and the first to eighth local memories 120a to 120h may correspond 1:1 to each other.

The shared memory 2000 may include first to eighth memory units 2100a to 2100h. The first to eighth memory units 2100a to 2100h may respectively correspond to the first to eighth neural cores 100a to 100h and the first to eighth local memories 120a to 120h. That is, the number of memory units may be eight, the same as the number of neural cores and local memories.

The shared memory 2000 may operate in one of two types of on-chip memory formats. That is, the shared memory 2000 may operate in either a local memory format or a global memory format. That is, the shared memory 2000 can implement two types of logical memories with one hardware.

When the shared memory 2000 is implemented in a local memory format, the shared memory 2000 is a dedicated memory for each of the first to eighth neural cores 100a to 100h, such as the first to eighth local memories 120a to 120h. (private memory) can operate. The local memory can operate at a relatively high-speed clock compared to the global memory, and when the shared memory 2000 operates in the form of a local memory, it can use a relatively faster clock.

When the shared memory 2000 is implemented in a global memory format, the shared memory 2000 may operate as a common memory used by the first neural core 100a and the second neural core 100b. there is. In this case, the shared memory 2000 may be shared not only by the first to eighth neural cores 100a to 100h but also by the first to eighth local memories 120a to 120h.

The global memory may generally use a lower clock than the local memory, but is not limited thereto. When the shared memory 2000 operates in a global memory format, the first to eighth neural cores 100a to 100h may share the shared memory 2000 . At this time, the shared memory 2000 is connected to the volatile memory 32 of FIG. 2 through the global interconnection 6000 and may operate as a buffer of the volatile memory 32 .

At least a part of the shared memory 2000 may operate in a local memory format and the rest may operate in a global memory format. That is, the entire shared memory 2000 may operate in a local memory format or the entire shared memory 2000 may operate in a global memory format. Alternatively, a part of the shared memory 2000 may operate in a local memory format and the remaining part may operate in a global memory format.

12 is a block diagram illustrating an example of memory reorganization of a neural processing system in accordance with some embodiments of the present invention.

Referring to FIGS. 11 and 12 , the first, third, fifth, and seventh dedicated regions AE1 and AE3 of the first, third, fifth, and seventh neural cores 100a, 100c, 100e, and 100g, respectively. , AE5, and AE7 may include only the first, third, fifth, and seventh local memories 120a, 120c, 120e, and 120g, respectively. In addition, the second, fourth, sixth, and eighth dedicated regions AE2, AE4, AE6, and AE8 of the second, fourth, sixth, and eighth neural cores 100b, 100d, 100f, and 100h, respectively, Second, fourth, sixth, and eighth local memories 120b, 120d, 120f, and 120h may be included. Also, the second, fourth, sixth, and eighth dedicated areas AE2 , AE4 , AE6 , and AE8 may include second, fourth, sixth, and eighth memory units 2100b, 2100d, 2100f, and 2100h. can The first, third, fifth, and seventh memory units 2100a, 2100c, 2100e, and 2100g of the shared memory 2000 may be used as a common area AC.

The common area AC may be a memory shared by the first to eighth neural cores 100a to 100h. The second dedicated area AE2 may include a second local memory 120b and a second memory unit 2100b. The second dedicated area AE2 may be an area in which the hardware-separated second local memory 120b and the second memory unit 210b operate in the same way and logically operate as one local memory. The fourth, sixth, and eighth dedicated areas AE4, AE6, and AE8 may also operate in the same manner as the second dedicated area AE2.

The shared memory 2000 according to the present embodiment can be used by converting an area corresponding to each neural core into a logical local memory and a logical global memory at an optimized ratio. The shared memory 2000 can adjust this ratio at run time.

That is, each neural core may perform the same task in some cases, but may also perform different tasks in other cases. In this case, the capacity of local memory and the capacity of global memory required for the work performed by each neural core are inevitably different each time. Accordingly, in the case where the composition ratio of the local memory and the shared memory is fixedly set as in the case of the conventional on-chip memory, inefficiency may occur due to calculation tasks allocated to each neural core.

Therefore, the shared memory 2000 of the neural processing apparatus according to the present embodiment can set an optimal ratio of local memory and global memory according to a computational task during run time, and can improve efficiency and speed of computation.

FIG. 13 is an enlarged block diagram of part A of FIG. 11 .

11 and 13, the shared memory 2000 includes a first local memory controller 122_1a, a second local memory controller 122_1b, a fifth local memory controller 122_1e, and a sixth local memory controller 122_1f. , first to eighth memory units 2100a to 2100h and a global controller 2200 . Other local memory controllers not shown may also be included in the present embodiment, but descriptions thereof are omitted for convenience.

The first local memory controller 122_1a may control the first local memory 120a. Also, the first local memory controller 122_1a may control the first memory unit 2100a. Specifically, when the first memory unit 2100a is implemented in a logical local memory format, control by the first local memory controller 122_1a may be performed on the first memory unit 2100a.

The second local memory controller 122_1b may control the second local memory 120b. Also, the second local memory controller 122_1b may control the second memory unit 2100b. That is, when the second memory unit 2100b is implemented in a logical local memory format, control by the first local memory controller 122_1a may be performed on the second memory unit 2100b.

The fifth local memory controller 122_1e may control the fifth local memory 120e. Also, the fifth local memory controller 122_1e may control the fifth memory unit 2100e. That is, when the fifth memory unit 2100e is implemented in a logical local memory format, control by the fifth local memory controller 122_1e may be performed on the fifth memory unit 2100e.

The sixth local memory controller 122_1f may control the sixth local memory 120f. Also, the sixth local memory controller 122_1f may control the sixth memory unit 2100f. That is, when the sixth memory unit 2100f is implemented in a logical local memory format, control by the sixth local memory controller 122_1f may be performed on the sixth memory unit 2100f.

The global controller 2200 may control all of the first to eighth memory units 2100a to 2100h. Specifically, when the first to eighth memory units 2100a to 2100h logically operate in a global memory format (that is, when they do not logically operate in a local memory format), the global controller 2200 performs a first memory operation. Units 2100a to 8th memory units 2100h may be controlled.

That is, the first to eighth memory units 2100a to 2100h are each controlled by the first to eighth local memory controllers 122_1a to 122_1h or controlled by the global controller 2200 depending on what type of memory is logically implemented. can be controlled by

When the local memory controllers including the first, second, fifth, and sixth local memory controllers 122_1a, 122_1b, 122_1e, and 122_1f respectively control the first to eighth memory units 2100a to 2100h, the first Since the to eighth local memory controllers 122_1a to 141h control the first to eighth memory units 2100a to 2100h in the same way as the first to eighth local memories 120a to 120h, the first to eighth neural cores (100a~100h) can be controlled by dedicated memory. Accordingly, the first to eighth memory units 2100a to 2100h may operate at a clock frequency corresponding to that of the first to eighth neural cores 100a to 100h.

The local memory controllers including the first local memory controller 122_1a, the second local memory controller 122_1b, the fifth local memory controller 122_1e, and the sixth local memory controller 122_1f are the LSU 110 of FIG. 6, respectively. can include

When the global controller 2200 controls at least one of the first to eighth memory units 2100a to 2100h, the global controller 2200 controls the first to eighth memory units 2100a to 2100h, respectively. It can be controlled by the global memory of the eighth neural cores 100a to 100h. Accordingly, at least one of the first to eighth memory units 2100a to 2100h may operate at a clock frequency independent of the clock frequency of the first to eighth neural cores 100a to 100h. However, this embodiment is not limited thereto.

The global controller 2200 may connect the first to eighth memory units 2100a to 2100h with the global interconnection 6000 of FIG. 3 . The first to eighth memory units 2100a to 2100h exchange data with the off-chip memory 30 of FIG. 1 by the global controller 2200 or exchange data with the first to eighth local memories 120a to 120h, respectively. can be exchanged.

Each of the first to eighth memory units 2100a to 2100h may include at least one memory bank. The first memory unit 2100a may include at least one first memory bank 2110a. The first memory bank 2110a may be an area obtained by dividing the first memory unit 2100a into a specific size. Each of the first memory banks 2110a may be memory devices having the same size. However, this embodiment is not limited thereto. 13 shows that four memory banks are included in one memory unit.

Similarly, the second, fifth, and sixth memory units 2100b, 2100e, and 2100f may each include at least one second, fifth, and sixth memory bank 2110b, 2110e, and 2110f.

Hereinafter, the first memory bank 2110a and the fifth memory bank 2110e will be described, which may be the same as other memory banks including the second and sixth memory banks 2110b and 2110f.

Each of the first memory banks 2110a may logically operate in a local memory format or logically operate in a global memory format. In this case, the first memory bank 2110a may operate independently of other memory banks in the first memory unit 2100a. However, this embodiment is not limited thereto.

When each memory bank operates independently, the first memory unit 2100a includes a first area that operates in the same way as the first local memory 120a and a second area that operates in a different way from the first local memory 120a. It can contain 2 areas. In this case, the first area and the second area do not necessarily coexist, and either area may occupy the entire first memory unit 2100a.

Similarly, the second memory unit 2100b may include a third area operating in the same way as the second local memory 120b and a fourth area operating in a different way from the second local memory 120b. In this case, the third area and the fourth area do not necessarily coexist, and either area may occupy the entire first memory unit 2100a.

In this case, the ratio of the first area to the second area may be different from the ratio of the third area to the fourth area. However, this embodiment is not limited thereto. Accordingly, the ratio of the first area to the second area may be the same as the ratio of the third area to the fourth area. That is, the memory configuration ratio in each memory unit can be varied as desired.

In general, in the case of a conventional system-on-chip, on-chip memories other than high-speed local memories are often composed of high-density, low-power SRAM. This is because SRAM has high efficiency in terms of chip area and power consumption compared to required capacity. However, the existing on-chip memory inevitably slows down the processing speed for tasks that require more data than the predetermined capacity of the local memory. There was no inefficiency at all.

In contrast, the shared memory 2000 according to some embodiments of the present invention may be selectively controlled by any one of the two controllers in some cases. In this case, the shared memory 2000 is not entirely controlled by a predetermined one of the two controllers, and may be independently controlled in units of memory units or units of memory banks.

Through this, the shared memory 2000 according to the present embodiment can obtain an optimal memory configuration ratio according to a computational task during run time, and thus perform a faster and more efficient calculation task. In the case of a processing unit specialized for artificial intelligence, the required size of local memory and global memory may vary for each specific application. Furthermore, when a deep learning network is used even for the same application, the required size of the local memory and the global memory may be different for each layer. In the shared memory 2000 according to the present embodiment, the configuration ratio of the memory can be changed during run time even when the operation step according to each layer is changed, so that fast and efficient deep learning work can be performed.

FIG. 14 is a diagram for explaining the first bank of FIG. 13 in detail. 14 illustrates the first memory bank 2110a, other memory banks may also have the same structure as the first memory bank 2110a.

Referring to FIG. 14 , the first memory bank 2110a may include a cell array Ca, a bank controller Bc, a first path unit P1 and a second path unit P2.

The cell array Ca may include a plurality of memory elements Cell therein. In the cell array Ca, a plurality of memory elements may be arranged in a lattice structure. The cell array Ca may be, for example, a static random access memory (SRAM) cell array.

The bank controller Bc may control the cell array Ca. The bank controller Bc may determine whether the cell array Ca operates in a local memory format or a global memory format and controls the cell array Ca accordingly.

Specifically, the bank controller Bc may determine whether to transmit/receive data in the direction of the first path unit P1 or the direction of the second path unit P2 during run time. The bank controller Bc may determine the transmission/reception direction of data according to the path control signal Spc.

The path control signal Spc may be generated by a pre-designed device driver or compiler. The path control signal Spc may be generated according to the characteristics of an arithmetic task. Alternatively, the path control signal Spc may be generated by an input received from a user. That is, the user may directly apply an input to the path control signal Spc in order to select the most optimal memory configuration ratio.

The bank controller Bc may determine a transmission/reception path of data stored in the cell array Ca through the path control signal Spc. The data exchange interface may vary according to the bank controller Bc determining the transmission/reception path of the data. That is, when the bank controller Bc exchanges data with the first path unit P1, the first interface may be used, and when data is exchanged with the second path unit P2, the second interface may be used. In this case, the first interface and the second interface may be different from each other.

Also, an address system in which data is stored may be different. That is, when a specific interface is selected, read and write operations can be performed with an address system corresponding thereto.

The bank controller Bc may operate at a specific clock frequency. For example, when the cell array Ca is an SRAM cell array, the bank controller Bc may operate at an operating clock frequency of a general SRAM.

The first path unit P1 may be connected to the bank controller Bc. The first path unit P1 may directly exchange data of the cell array Ca with the first neural core 100a. In this case, "directly" may mean mutual exchange without going through the global interconnection 6000 . That is, the first neural core 100a can directly exchange data with the first local memory 120a, and the first neural core 100a operates when the shared memory 2000 is logically implemented in the form of a local memory. Data can be exchanged through 1 path unit P1. The first path unit P1 may include a local memory controller including the first local memory controller 122_1a and the second local memory controller 122_1b of FIG. 13 .

The first path unit P1 may configure a multi-cycle sync-path. That is, the operating clock frequency of the first path unit P1 may be the same as the operating clock frequency of the first neural core 100a. The first local memory 120a can rapidly exchange data at the same clock frequency as the operating clock frequency of the first neural core 100a in order to rapidly exchange data at the same speed as the operation of the first neural core 100a. . Similarly, the first path unit P1 may operate at the same clock frequency as the operating clock frequency of the first neural core 100a.

In this case, the operating clock frequency of the first path unit P1 may be a multiple of the operating clock frequency of the bank controller Bc. In this case, a separate CDC (Clock Domain Crossing) operation for clock synchronization between the bank controller Bc and the first path unit P1 is not required, and thus data transmission delay may not occur. there is. Accordingly, faster and more efficient data exchange may be possible.

14 exemplarily, the operating clock frequency of the first path unit P1 may be 1.5 GHz. This may be twice the frequency of 750 MHz of the bank controller (Bc). However, the present embodiment is not limited thereto, and any number may be possible if the first path unit P1 operates at an integer multiple of the clock frequency of the bank controller Bc.

The second path unit P2 may be connected to the bank controller Bc. The second path unit P2 may exchange data of the cell array Ca through the global interconnection 6000 without directly exchanging the data with the first neural core 100a. That is, the first neural core 100a may exchange data with the cell array Ca through the global interconnection 6000 and the second path unit P2. At this time, the cell array Ca may exchange data not only with the first neural core 100a but also with other neural cores.

That is, the second path unit P2 may be a data exchange path between the cell array Ca and all neural cores when the first memory bank 2110a is logically implemented in a global memory format. The second path unit P2 may include the global controller 2200 of FIG. 13 .

The second path unit P2 may constitute an async-path. The operating clock frequency of the second path unit P2 may be the same as the operating clock frequency of the global interconnection 6000 . The second path unit P2 may also operate at the same clock frequency as the operating clock frequency of the global interconnection 6000 .

At this time, the operating clock frequency of the second path unit P2 may not be synchronized with the operating clock frequency of the bank controller Bc. In this case, a clock domain crossing (CDC) operation may be required for clock synchronization between the bank controller Bc and the second path unit P2. When the operating clock frequency of the bank controller Bc and the operating clock frequency of the second path unit P2 are not synchronized with each other, the degree of freedom in clock domain design can be increased. Accordingly, the difficulty of hardware design is lowered, and the hardware operation can be derived more easily.

The bank controller Bc may use different address systems when exchanging data through the first path unit P1 and when exchanging data through the second path unit P2. That is, the bank controller Bc may use the first address system through the first path unit P1 and use the second address system through the second path unit P2. In this case, the first address system and the second address system may be different from each other.

The bank controller Bc does not necessarily exist for each memory bank. That is, since the bank controller (Bc) is not a part for scheduling but serves to transmit signals, it is not an essential part for each memory bank having two ports. Thus, one bank controller Bc can control several memory banks. Several memory banks can operate independently even though they are controlled by the bank controller Bc. However, this embodiment is not limited thereto.

Of course, the bank controller Bc may exist for each memory bank. In this case, the bank controller Bc can individually control each memory bank.

13 and 14, when the first memory unit 210a exchanges data through the first path unit P1, the first address system is used and data is exchanged through the second path unit P2. In case of exchange, a second address scheme may be used. Similarly, when the second memory unit 210b exchanges data through the first path unit P1, the third address system is used, and when data is exchanged through the second path unit P2, the second address system is used. system can be used. In this case, the first address system and the third address system may be identical to each other. However, this embodiment is not limited thereto.

The first address system and the third address system may be used exclusively for the first neural core 100a and the second neural core 100b, respectively. The second address system may be commonly applied to the first neural core 100a and the second neural core 100b.

14 exemplarily, the operating clock frequency of the second path unit P2 may operate at 1 GHz. This may be a frequency that is not synchronized with the 750 MHz operating clock frequency of the bank controller (Bc). That is, the operating clock frequency of the second path unit P2 is not dependent on the operating clock frequency of the bank controller Bc and can be freely set.

A general global memory uses a slow SRAM (eg, 750 MHz) and a faster global interconnection (eg, 1 GHz), so that delays due to CDC work inevitably occur. In contrast, in the shared memory 2000 according to some embodiments of the present invention, there is room to use the first path unit P1 in addition to the second path unit P2, so delay due to CDC work can be avoided.

In addition, since a plurality of neural cores use one global interconnection 6000 in a general global memory, the overall processing speed may be easily reduced when the amount of data transfer occurs simultaneously. In contrast, in the shared memory 2000 according to some embodiments of the present invention, there is room to use the first path unit P1 in addition to the second path unit P2, so that the amount of data processed in the global controller 2200 is adequately reduced. A dispersing effect can also be obtained.

15 is a conceptual diagram for explaining virtual ID assignment of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 15 , the neural core SoC 10 may include a plurality of neural processors. 15 shows a case where there are, for example, 8 neural processors. The neural core SoC 10 may include first to eighth neural processors PP0 to PP7.

At this time, the first to fourth neural processors PP0 to PP3 may divide and perform one task by the same program. The fifth neural processor 1000 may perform one task alone, and the sixth to eighth neural processors PP5 to PP7 may divide and perform another task.

That is, 8 neural processors can be divided into 3 sets. In this case, the first set Set1 may include the first to fourth neural processors PP0 to PP3. The second set Set2 may include the fifth neural processor 1000 . The third set (Set3) may include the sixth to eighth neural processors PP5 to PP7.

In each set, a new virtual ID may be assigned. That is, the first to fourth neural processors PP0 to PP3 of the first set Set1 may be assigned first to fourth virtual IDs VP0 to VP3, respectively. The fifth neural processor 1000 of the second set (Set2) may be assigned a first virtual ID (VP0). The sixth to eighth neural processors PP5 to PP7 of the third set (Set3) may be assigned first to third virtual IDs VP0 to VP2.

Therefore, when different programs are executed, the same virtual ID can be assigned to different neural processors, but when the same program is executed together, the physical ID (i.e., the unique ID of each neural processor) and the virtual ID are different. It can correspond 1:1.

16 is a diagram for explaining a virtual ID allocation and a VPID table of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 16 , a case in which first to fourth neural processors PP0 to PP3 of the first set Set1 are assigned first to fourth virtual IDs VP0 to VP3 will be described. The order of the physical ID and the virtual ID may not be the same. That is, the first neural processor 1000 may be assigned a third virtual ID VP2 instead of the first virtual ID VP0. The second neural processor 1000 may be assigned a second virtual ID VP2 , and the third neural processor 1000 may be assigned a first virtual ID VP0 . The fourth neural processor 1000 may be assigned a fourth virtual ID VP4.

Accordingly, the VPID table (TB_VTP) may record a physical ID corresponding to a virtual ID. For example, when values of 3, 0, 1, and 2 are sequentially recorded in the VPID table (TB_VTP), it is possible to determine which physical IDs the first to fourth virtual IDs (VP0 to VP3) correspond to in reverse order. there is.

Specifically, the neural processor assigned the first virtual ID VP0 is the third neural processor 1000 by the number 2, and the neural processor assigned the second virtual ID VP2 is the second neural processor by the number 1. (1000). The neural processor assigned the third virtual ID (VP2) is the first neural processor 1000 by the number 0, and the neural processor assigned the fourth virtual ID (VP4) is the fourth neural processor 1000 by the number 3. am.

17 is a diagram for explaining a process of identifying a physical ID through a sync target and a VPID table.

Referring to FIG. 17 , the L3 sync target Sm_V may be a signal generated by each neural processor sending a synchronization signal. That is, the L3 sync target (Sm_V) may include, for example, 4 fields. This can be attributed to the fact that there are 4 neural processors in the same set. Each field of the L3 sync target Sm_V may correspond to first to fourth virtual IDs VP0 to VP3. That is, if 1, 0, 1, 1 are described in the L3 sync target Sm_V, 1, 1, 0, 1 may correspond to the first to fourth virtual IDs VP0 to VP3 in reverse order, respectively.

A meaning of '1' of the L3 sync target Sm_V may indicate a virtual ID of the neural processor 1000 to which a synchronization signal according to the L3 sync target Sm_V should be delivered. That is, the last 1 among 1, 0, 1, and 1 may mean that a synchronization signal according to the L3 sync target Sm_V should be transmitted to the neural processor of the first virtual ID VP0. That is, it can be represented by 1, 0, 1, 1 that the synchronization signal according to the L3 sync target Sm_V should be delivered to the remaining three neural processors except for the neural processor of the third virtual ID VP2.

The virtual IDs of the neural processors to which the synchronization signal according to the L3 sync target Sm_V should be transmitted are the first and second neural processors to which the synchronization signal according to the L3 sync target Sm_V should be transmitted. And after being identified as the fourth virtual IDs (VP0, VP1, VP3), the physical ID of the corresponding neural processor can be checked through the VPID table (TB_VTP). The neural processor can check the real address only after checking the physical ID.

Since the VPID table (TB_VTP) has values of 3, 0, 1, and 2, it can be seen that the physical IDs of the first, second, and fourth virtual IDs (VP0, VP1, and VP3) are 2, 1, and 3, respectively. . That is, the second to fourth neural processors PP1 to PP3 may be neural processors that receive synchronization signals according to the L3 sync target Sm_V.

18 is a directed acyclic graph for explaining the sequence of deep learning tasks.

Referring to FIG. 18 , a calculation task of a neural processing apparatus according to some embodiments of the present invention may be expressed through a directed acyclic graph. At this time, if the current task is expressed as TaskN, the previous task may be Task(N-1) and the next task may be Task(N+1).

That is, in order for the current task, TaskN, to be performed, Task(N-1) must be completed, and similarly, in order for the next task, Task(N+1), to be performed, the current task, TaskN, must be completed.

Therefore, a synchronization signal indicating that each task has been completed must be transmitted from the neural processor that performed the task, and the synchronization signal can be determined by a dependency chain in which the next task must be performed. Accordingly, the L3 sync target Sm_V may be an instruction in which information about a neural processor to perform the next task is written. When a value is written to the L3 sync target (Sm_V), a synchronization signal may be transmitted accordingly.

19 is a conceptual diagram illustrating a synchronization signal transmission operation according to a sync target for L3 synchronization of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 19 , the first neural processor 1000 may transmit synchronization signals according to the sync target Sm_V to the second to fourth neural processors PP1 to PP3. Through this, synchronization of the SoC level, that is, level 3 (L3) can be performed.

A neural processing apparatus according to some embodiments of the present invention may include first to third semaphore memories smp1 to smp3 respectively corresponding to the second to fourth neural processors PP1 to PP3. The first to third semaphore memories smp1 to smp3 may be included in each of the second to fourth neural processors PP1 to PP3. The first to third semaphore memories smp1 to smp3 may have the same shape as each other. Therefore, only the first semaphore memory smp1 will be described in detail below.

The first semaphore memory smp1 may correspond to the second neural processor 1000 . The first semaphore memory smp1 may include four fields respectively corresponding to the four neural processors included in the first set Set1.

For example, the first semaphore memory smp1 may include first to fourth fields, and the first to fourth fields may respectively correspond to the first to fourth neural processors PP0 to PP3. That is, the first to fourth fields may be arranged in the same order as the physical IDs of the first to fourth neural processors PP0 to PP3.

That is, the first field of the first semaphore memory smp1 is a part for the first neural processor 1000 and is expressed as 1 when a synchronization signal according to the L3 sync target Sm_V is received from the first neural processor 1000. , or can be expressed as 0. Of course, the opposite expression may also be possible.

Similarly, the first field value of the second semaphore memory smp2 and the third semaphore memory smp3 may also be displayed as 1 by receiving a synchronization signal according to the L3 sync target Sm_V by the first neural processor 1000. there is. In this way, the indication of 1, 0, 1, 1 of the first semaphore memory smp1 is a synchronization signal according to the L3 sync target Sm_V by the first, third, and fourth neural processors PP0, PP2, and PP3. may mean that is received.

When the current task, TaskN, is finished, the first neural processor 1000 sends a synchronization signal according to the L3 sync target (Sm_V) through the L3 sync channel 6300 of FIG. 4 to start the next task, Task (N+1). can be transmitted through Such synchronization may also be performed by other neural processors 1000, respectively.

Synchronization of the neural processing device according to the present embodiment can be performed in parallel since there is no centrally controlled control processor, thereby minimizing the occurrence of latency. In addition, the efficiency of the entire device can be maximized because the overhead of scheduling in consideration of the dependencies of tasks due to such synchronization is not required.

20 is a conceptual diagram for explaining a synchronization signal reception operation according to a sync target for L3 synchronization of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 20 , the first neural processor 1000 may receive synchronization signals from the first, third, and fourth neural processors PP0 , PP2 , and PP3 . Accordingly, the first to fourth fields of the first semaphore memory smp1 corresponding to the first neural processor 1000 may be filled with 1, 0, 1, 1, respectively.

A neural processing apparatus according to some embodiments of the present invention may include first to fourth FIFO buffers B1 to B4 respectively corresponding to the first to fourth fields. The first to fourth FIFO buffers may provide values of the first to fourth fields of the first semaphore memory smp1 to the first neural processor 1000 in a first in first out (FIFO) manner.

In general, the work of the neural processing device is not simply expressed as a straight line as shown in FIG. 18 . That is, one task can have a dependency chain according to several previous tasks. Accordingly, in the case of a task having more than one dependency chain, several semaphore memories may be required.

However, as the number of semaphore memories increases, the required memory space also increases accordingly, so resources required for a small space may be excessive. Accordingly, the neural processing apparatus according to some embodiments of the present invention may promote efficient utilization of memory space by adding a FIFO buffer to one semaphore memory per neural processor.

That is, if synchronization signals according to several dependencies are sequentially entered into the FIFO buffer, even one semaphore memory can sequentially process the synchronization signals without missing them. Therefore, this embodiment can perform tasks of several dependency chains without difficulty while increasing memory efficiency.

21 is a block diagram for explaining L1 and L2 synchronization of a neural processing apparatus according to some embodiments of the present invention, and FIG. 22 describes L1 and L2 synchronization of a neural processing apparatus according to some embodiments of the present invention. It is a ladder diagram for

Referring to FIG. 21 , the first neural core 100a includes a first neural core store unit 112b, a first neural core load unit 112a, a first local memory 120a, and a first local memory store unit 111b. ) and a first local memory load unit 111a.

Similarly, the second neural core 100b includes a second neural core store unit 112d, a second neural core load unit 112c, a second local memory 120b, a second local memory store unit 111d, and a second neural core load unit 112c. 2 local memory load units 111c may be included.

The second neural core store unit 112d of the second neural core 100b generates an L1 sync request signal (S10).

The L1 sync request signal is synchronized when an L1 sync generate signal is received later, and can be maintained in a stall state until then. That is, the L1 sync request signal may be generated in a ready state for synchronization.

The fourth neural core load unit 112f of the fourth neural core 100d may generate an L2 sync request signal (S11).

When there are a plurality of neural cores, each synchronization preparation time may be different. Naturally, the L2 sync request signal may be generated early like the fourth neural core 100d.

Subsequently, the second local memory store unit 111d stores data in the second local memory 120b (S12, ①). Subsequently, the second local memory store unit 111d transmits the L1 sync generation signal to the second neural core store unit 112d (S13, ②). In this case, the L1 sync generation signal may be transmitted using an L1 sync path. Accordingly, the L1 sync request signal of the second neural core store unit 112d may be synchronized.

Subsequently, the second neural core store unit 112d includes the first neural core load unit 112a of the first neural core 100a, the third neural core load unit 112e of the third neural core 100c, and the fourth neural core load unit 112e of the third neural core 100c. The L2 sync generation signal may be broadcast to the fourth neural core load unit 112f of the neural core 100d (S14, S15, S16, ③). At this time, the L2 sync generation signal may be transmitted through the L2 sync pass 300 .

At this time, the fourth neural core 100d for which the L2 sync request signal has already been generated is immediately synchronized and a load operation is performed (S17).

In contrast, when the L2 sync request signal is generated (S18), the first neural core 100a and the third neural core 100c may perform a load task (S19, ④, ⑤)

In the load operation, the first neural core load unit 112a may request data from the second local memory 120b through the local interconnection 200 (④) and receive a data reply to the request (⑤). ).

Similarly, when the L2 sync request signal is generated in the third neural core 100c (S20), a load operation may be performed (S21).

Both synchronization of L2 (level 2) and synchronization of L1 (level 1) of this embodiment are not managed by the control processor, but each element performs them in parallel, which can bring great advantages in terms of latency and efficiency.

23 is a diagram for explaining an instruction set structure of a neural processing apparatus according to some embodiments of the present invention.

23, an instruction set architecture (ISA) of a neural processing device according to some embodiments of the present invention includes an operation code (opcode), an L1 sync target (Target for L1 SYNC), and an L2 sync target ( Target for L2 SYNC) and L3 sync target (Target for L3 SYNC). That is, sync targets of levels 1 to 3 may all be included in the structure of the instruction set.

24 is a block diagram for explaining a software layer structure of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 24 , a software layer structure of a neural processing apparatus according to some embodiments of the present invention may include a DL framework 10000, a compiler stack 20000, and a backend module 30000.

The DL framework 10000 may refer to a framework for a deep learning model network used by a user. For example, a trained neural network may be created using a program such as TensorFlow or PyTorch.

The compiler stack 20000 may include an adaptation layer 21000, a compute library 22000, a front-end compiler 23000, a back-end compiler 24000, and a runtime driver 25000.

The adaptation layer 21000 may be a layer in contact with the DL framework 10000. The adaptation layer 21000 may quantize the user's neural network model generated in the DL framework 10000 and perform graph correction. Also, the adaptation layer 21000 may convert a model type into a required type.

The front-end compiler 23000 may convert various neural network models and graphs transmitted from the adaptation layer 21000 into a constant intermediate representation (IR). The converted IR may be a preset expression that is easy to handle in the backend compiler 24000 later.

The IR of the front-end compiler 23000 may be preliminarily optimized at the graph level. In addition, the front-end compiler 23000 may finally generate an IR through an operation of converting the layout into a hardware-optimized layout.

The backend compiler 24000 optimizes the IR converted by the frontend compiler 23000 and converts it into a binary file so that the runtime driver can use it. The backend compiler 24000 may generate optimized code by dividing a job into a scale suitable for hardware details.

The compute library 22000 may store template operations designed in a form suitable for hardware among various operations. The compute library 22000 provides the backend compiler 24000 with several template operations that require hardware to generate optimized codes.

The runtime driver 25000 may perform continuous monitoring during operation to drive the neural network device according to some embodiments of the present invention. Specifically, it may be responsible for executing interfaces of neural network devices.

The backend module 30000 may include an application specific integrated circuit (ASIC) 31000, a field programmable gate array (FPGA) 32000, and a C-model 33000. The ASIC 31000 may refer to a hardware chip determined according to a predetermined design method. The FPGA 32000 may be a programmable hardware chip. The C-model (33000) may refer to a model implemented by imitating hardware on software.

The backend module 30000 may perform various tasks and derive results using binary codes generated through the compiler stack 20000 .

25 is a conceptual diagram for explaining a deep learning operation performed by a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 25, the artificial neural network model 40000, as an example of a machine learning model, is a statistical learning algorithm implemented based on the structure of a biological neural network or an algorithm thereof in machine learning technology and cognitive science. is the structure that runs

In the artificial neural network model 40000, as in a biological neural network, nodes, which are artificial neurons that form a network by combining synapses, repeatedly adjust synaptic weights, and between correct outputs corresponding to specific inputs and inferred outputs. By learning to reduce the error of , it is possible to represent a machine learning model having problem solving ability. For example, the artificial neural network model 40000 may include an arbitrary probability model, a neural network model, and the like used in artificial intelligence learning methods such as machine learning and deep learning.

A neural processing apparatus according to some embodiments of the present invention may implement the form of the artificial neural network model 40000 to perform calculations. For example, the artificial neural network model 40000 may receive an input image and output information about at least a part of an object included in the input image.

The artificial neural network model 40000 is implemented as a multilayer perceptron (MLP) composed of multi-layer nodes and connections between them. The artificial neural network model 40000 according to this embodiment may be implemented using one of various artificial neural network model structures including MLP. As shown in FIG. 25, the artificial neural network model 40000 includes an input layer 41000 that receives input signals or data 40100 from the outside, and an output layer that outputs output signals or data 40200 corresponding to the input data. (44000), which is located between the input layer 41000 and the output layer 44000, receives signals from the input layer 41000, extracts characteristics, and delivers n (where n is a positive integer) to the output layer 44000. It is composed of hidden layers (42000 to 43000). Here, the output layer 44000 receives signals from the hidden layers 42000 to 43000 and outputs them to the outside.

The learning method of the artificial neural network model (40000) includes a supervised learning method that learns to be optimized for problem solving by inputting a teacher signal (correct answer), and an unsupervised learning method that does not require a teacher signal. ) way.

The neural processing apparatus may directly generate learning data for learning the artificial neural network model 40000 through simulation. In this way, a plurality of output variables corresponding to a plurality of input variables are matched in the input layer 41000 and the output layer 44000 of the artificial neural network model 40000, respectively, and the input layer 41000, hidden layers 42000 to 43000 and By adjusting synapse values between nodes included in the output layer 44000, learning can be performed so that a correct output corresponding to a specific input can be extracted. Through this learning process, the characteristics hidden in the input variables of the artificial neural network model 40000 can be identified, and the nodes of the artificial neural network model 40000 can reduce the error between the output variable calculated based on the input variable and the target output. You can adjust the synaptic value (or weight) between them.

26 is a conceptual diagram for explaining learning and reasoning operations of a neural network of a neural processing apparatus according to some embodiments of the present invention.

Referring to FIG. 26 , in a training phase, a plurality of training materials (TD) may be forwarded to an artificial neural network model (NN) and then forwarded again. Through this, weights and biases of each node of the artificial neural network model (NN) are tuned, and through this, learning can be performed so that more and more accurate results can be derived. In this way, through the training phase, the artificial neural network model (NN) may be converted into a learned neural network model (NN_T).

In the inference phase, new data ND may be input to the learned neural network model NN_T again. The learned neural network model NN_T may derive result data RD through already learned weights and biases by taking new data ND as an input. For the result data RD, it may be important which learning material TD was used in the training phase and how many of the learning materials TD were used.

Hereinafter, a synchronization method of a neural processing apparatus according to some embodiments of the present invention will be described with reference to FIGS. 17, 19, 20, 27, and 28 . Parts overlapping with the above-described embodiment are simplified or omitted.

27 is a flowchart illustrating a synchronization method of a neural processing apparatus according to some embodiments of the present invention, and FIG. 28 is a flowchart illustrating in detail the L3 sync target storage step and the FIFO method providing step of FIG. 27 .

Referring to FIG. 27 , the first neural processor generates an L3 sync target (S100).

Specifically, referring to FIG. 17 , the L3 sync target Sm_V may be a signal generated by each neural processor sending a synchronization signal. That is, the L3 sync target (Sm_V) may include, for example, 4 fields. This can be attributed to the fact that there are 4 neural processors in the same set. Each field of the L3 sync target Sm_V may correspond to first to fourth virtual IDs VP0 to VP3. That is, if 1, 0, 1, 1 are described in the L3 sync target Sm_V, 1, 1, 0, 1 may correspond to the first to fourth virtual IDs VP0 to VP3 in reverse order, respectively.

Referring again to FIG. 27 , a second neural processor to be received is identified using the L3 sync target and the VPID table (S200).

Specifically, referring to FIG. 17 , a neural processor that needs to send a synchronization signal according to the L3 sync target Sm_V is a neural processor to which a synchronization signal according to the L3 sync target Sm_V should be transmitted by the L3 sync target Sm_V. After the virtual IDs are identified as the first, second, and fourth virtual IDs (VP0, VP1, and VP3), the physical ID of the corresponding neural processor can be checked through the VPID table (TB_VTP). The neural processor can check the real address only after checking the physical ID.

Since the VPID table (TB_VTP) has values of 3, 0, 1, and 2, it can be seen that the physical IDs of the first, second, and fourth virtual IDs (VP0, VP1, and VP3) are 2, 1, and 3, respectively. . That is, the second to fourth neural processors PP1 to PP3 may be neural processors that receive synchronization signals according to the L3 sync target Sm_V.

Referring again to FIG. 27, the synchronization signal according to the L3 sync target is stored in the semaphore memory of the second neural processor through the L3 sync channel (S300).

Specifically, referring to FIG. 19 , the first semaphore memory smp1 may include first to fourth fields, and the first to fourth fields may respectively correspond to the first to fourth neural processors PP0 to PP3. there is. That is, the first to fourth fields may be arranged in the same order as the physical IDs of the first to fourth neural processors PP0 to PP3.

That is, the first field of the first semaphore memory smp1 is a part for the first neural processor 1000 and is expressed as 1 when a synchronization signal according to the L3 sync target Sm_V is received from the first neural processor 1000. , or can be expressed as 0. Of course, the opposite expression may also be possible.

Again, referring to FIG. 27, the value of the semaphore memory is provided to the second neural processor in a FIFO method (S400).

Specifically, referring to FIG. 20 , the neural processing apparatus according to some embodiments of the present invention may include first to fourth FIFO buffers B1 to B4 respectively corresponding to the first to fourth fields. The first to fourth FIFO buffers may provide values of the first to fourth fields of the first semaphore memory smp1 to the first neural processor 1000 in a first in first out (FIFO) manner.

Referring to FIG. 28, steps S300 and S400 will be described in detail.

The synchronization signal according to the L1 sync target of the first neural processor is stored in the first field of the semaphore memory of the second neural processor (S310), and the first field value of the semaphore memory is provided to the second neural processor in a FIFO manner ( S410).

Similarly, the synchronization signal according to the L1 sync target of the second neural processor is stored in the second field of the semaphore memory of the second neural processor (S320), and the value of the second field of the semaphore memory is stored in the second neural processor in a FIFO manner. It is provided (S420).

The synchronization signal according to the L1 sync target of the third neural processor is stored in the third field of the semaphore memory of the second neural processor (S330), and the value of the third field of the semaphore memory is provided to the second neural processor in a FIFO method ( S430).

The synchronization signal according to the L1 sync target of the fourth neural processor is stored in the fourth field of the semaphore memory of the second neural processor (S340), and the value of the fourth field of the semaphore memory is provided to the second neural processor in a FIFO manner ( S440).

That is, each field corresponds to each neural processor, and synchronization may be performed in parallel in a FIFO format.

Again, referring to FIG. 27 , the second neural processor performs synchronization through the L3 sync target (S500).

Hereinafter, a synchronization method of a neural processing apparatus according to some embodiments of the present invention will be described with reference to FIGS. 21, 22, 29, and 30 . Parts overlapping with the above-described embodiment are simplified or omitted.

FIG. 29 is a flowchart illustrating a synchronization method of L1 and L2 levels of a neural processing apparatus according to some embodiments of the present invention, and FIG. 30 is a flowchart illustrating a data request step of FIG. 29 in detail.

Referring to FIG. 29 , data is stored in the local memory of the first neural core (S1100). Subsequently, within the first neural core, the local memory store unit transmits a synchronization signal according to the L1 sync target to the neural core store unit (S1200).

Specifically, referring to FIGS. 21 and 22 , the second local memory store unit 111d stores data in the second local memory 120b (S12, ①). Subsequently, the second local memory store unit 111d transmits the L1 sync generation signal to the second neural core store unit 112d (S13, ②). In this case, the L1 sync generation signal may be transmitted using an L1 sync path. Accordingly, the L1 sync request signal of the second neural core store unit 112d may be synchronized.

Referring again to FIG. 29 , the neural core store unit of the first neural core transmits a synchronization signal according to the L2 sync target to the neural core load units of the second to fourth neural cores (S1300).

Specifically, referring to FIGS. 21 and 22 , the second neural core store unit 112d is configured to load the first neural core load unit 112a of the first neural core 100a and the third neural core 100c. The L2 sync generation signal may be broadcast to the third neural core load unit 112e and the fourth neural core load unit 112f of the fourth neural core 100d (S14, S15, S16, ③). At this time, the L2 sync generation signal may be transmitted through the L2 sync pass 300 .

Referring again to FIG. 29 , the second to fourth neural core load units request data from the local memory of the first neural core through the local interconnection (S1400).

Referring to FIG. 30 in detail, the second neural core receives a synchronization signal according to the L2 sync target (S1410) and determines whether an L2 sync request signal has already been generated (S1420). If not, it waits for the L2 sync request signal to be generated (S1430). If so, the second neural core requests data from the local memory of the first neural core (S1440).

Referring again to FIG. 29 , the second to fourth neural core load units receive data (S1500).

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (30)

first and second neural processors; a shared memory shared by the first and second neural processors; and First and second semaphore memories corresponding to the first and second neural processors and receiving and storing an L3 sync target, wherein synchronization of the first and second neural processors is performed according to the L3 sync target. first and second semaphore memories; A global interconnection connecting the first and second neural processors and the shared memory and including an L3 sync channel through which a synchronization signal according to the L3 sync target is transmitted. Neural Processing Unit. According to claim 1, The global interconnection, The L3 sync channel; a data channel for transmitting data between the shared memory and the first and second neural processors; A control channel for transmitting a control signal to the first and second neural processors, Neural Processing Unit. According to claim 1, The first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively. Neural Processing Unit. According to claim 3, Further comprising a first FIFO buffer for sequentially transferring values of the first field to the first neural processor. Neural Processing Unit. According to claim 1, The L3 sync target includes first and second L3 sync targets, The first neural processor generates the first L3 sync target; The second neural processor generates the second L3 sync target. Neural Processing Unit. According to claim 1, The L3 sync target includes first and second sync target fields corresponding to the first and second neural processors, respectively; The first and second sync target fields include information on whether the first and second neural processors receive synchronization signals according to the L3 sync target. Neural Processing Unit. According to claim 6, The first and second sync target fields are arranged in the order of virtual IDs of the first and second neural processors, respectively. Neural Processing Unit. According to claim 7, The first neural processor identifies a physical ID of a neural processor receiving a synchronization signal according to the L3 sync target by using the L3 sync target and a VPID table; The VPID table includes information for converting the virtual ID and the physical ID. Neural Processing Unit. According to claim 1, The L3 sync target is included in the instruction set architecture (ISA), Neural Processing Unit. According to claim 1, The neural processor, at least one neural core; Including a local interconnection for transmitting data between the at least one neural core, Neural Processing Unit. According to claim 10, The neural processor, Further comprising an L2 sync pass in which a synchronization signal according to an L2 sync target for performing synchronization between the at least one neural core is transmitted. Neural Processing Unit. According to claim 10, Each of the at least one neural core, A processing unit that receives input activations and weights, performs deep learning calculations, and outputs output activations; Including a local memory for temporarily storing the input activation, the weight, and the output activation, Neural Processing Unit. at least one neural processor; shared memory; and a global interconnection connecting the at least one neural processor and a shared memory and used for L3 synchronization of the neural processor; The neural processor includes at least one neural core; a local interconnection connecting the at least one neural core; An L2 sync pass used for L2 synchronization of the at least one neural core, The neural core, a processing unit that performs computational work; a local memory for temporarily storing data; Including an L1 sync pass used for L1 synchronization of the local memory and the processing unit, Neural Processing Unit. According to claim 13, The global interconnection, a data channel for transmitting data between the at least one neural processor and the shared memory; a control channel for transmitting a control signal between the at least one neural processor; Including a sync channel used for the L3 synchronization, Neural Processing Unit. According to claim 13, The neural processor, Further comprising a local interconnection for transmitting data between the at least one neural core, Neural Processing Unit. According to claim 13, Including a data path used for data exchange between the local memory and elements including the processing unit, Neural Processing Unit. According to claim 13, The at least one neural processor includes first and second neural processors; First and second semaphore memories corresponding to the first and second neural processors and receiving and storing a synchronization signal corresponding to an L3 sync target, wherein the first and second semaphore memories correspond to values of the first and second semaphore memories. and first and second semaphore memories in which synchronization of the second neural processor is performed. Neural Processing Unit. According to claim 17, The first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively; Further comprising a first FIFO buffer for sequentially transferring values of the first field to the first neural processor. Neural Processing Unit. According to claim 17, The first neural processor transmits an instruction set structure; The instruction set structure includes an operation code, an L3 sync target for the L3 synchronization, an L2 sync target for the L2 synchronization, and an L1 sync target for the L1 synchronization. Neural Processing Unit. A method for synchronizing a neural processing device including first and second neural processors, The first neural processor generates an L3 sync target for L3 synchronization, and the L3 sync target is arranged in order of virtual IDs of the first and second neural processors; Identifying a physical ID of the second neural processor using the L3 sync target and a VPID table, wherein the VPID table is a conversion table between the virtual ID and the physical ID of the neural processor; storing a synchronization signal according to the L3 sync target in a first semaphore memory of the second neural processor through an L3 sync channel of a global interconnection; The second neural processor performing L3 synchronization according to the value of the first semaphore memory. Synchronization method of neural processing device. According to claim 20, The first semaphore memory includes first and second fields corresponding to the first and second neural processors, respectively. Synchronization method of neural processing device. According to claim 21, Performing the L3 synchronization, Providing the value of the first field to the second neural processor in a FIFO manner; Including providing the value of the second field to the second neural processor in a FIFO manner. Synchronization method of neural processing device. According to claim 20, The virtual ID includes first and second virtual IDs corresponding to the first and second neural processors, respectively. Synchronization method of neural processing device. According to claim 20, The first neural processor, first and second neural cores; a local interconnection for transmitting data between the first and second neural cores; Including an L2 sync pass for transmitting a synchronization signal according to an L2 sync target between the first and second neural cores, Synchronization method of neural processing device. According to claim 24, The first neural core, A first processing unit receiving a first input activation and a first weight, performing a deep learning operation, and outputting a first output activation; a first local memory temporarily storing the first input activation, the first weight, and the first output activation; A first L1 sync pass for transmitting a synchronization signal according to an L1 sync target between the first local memory and the first processing unit; The second neural core, a second processing unit receiving a second input activation and a second weight, performing a deep learning operation, and outputting a second output activation; a second local memory temporarily storing the second input activation, the second weight, and the second output activation; And a second L1 sync pass for transmitting a synchronization signal according to the L1 sync target between the second local memory and the second processing unit. Synchronization method of neural processing device. According to claim 25, Storing data in the first local memory; Inside the first neural core, a synchronization signal according to the L1 sync target is transmitted through the first L1 sync pass; The first neural core transmits a synchronization signal according to the L2 sync target to the second neural core through the second L2 sync pass; Further comprising the second neural core receiving data through the local interconnection. Synchronization method of neural processing device. A neural processing apparatus including first and second neural cores, a local interconnection connecting the first and second neural cores, and an L2 sync pass used for L2 synchronization of the first and second neural cores, The first neural core includes a first processing unit that performs calculation tasks, a first local memory that temporarily stores data input/output to the first processing unit, and a combination of the first local memory and the first processing unit. Including a first L1 sync pass used for L1 synchronization, The second neural core includes a second processing unit that performs calculation tasks, a second local memory that temporarily stores data input and output to the second processing unit, the second local memory and the second processing unit. A synchronization method of a neural processing device including a second L1 sync pass used for L1 synchronization, Storing data in the first local memory; Inside the first neural core, a synchronization signal according to the L1 sync target is transmitted through the first L1 sync pass; The first neural core transmits a synchronization signal according to the L2 sync target to the second neural core through the second L2 sync pass; Further comprising the second neural core receiving data through the local interconnection. Synchronization method of neural processing device. According to claim 27, the first neural core further comprises a first LSU for moving data between the first local memory and the first local interconnection; The first LSU includes a first local memory store unit that stores the first local memory and a first neural core store unit that stores data from the first neural core to the outside; Transmitting the synchronization signal according to the L1 sync target through the first L1 sync pass in the first neural core, Transmitting, by the first local memory store unit, a synchronization signal according to the L1 sync target to the first neural core store unit. Synchronization method of neural processing device. 29. The method of claim 28, The second neural core further comprises a second LSU for moving data between the second local memory and the second local interconnection; The second LSU includes a second neural core load unit that performs an external load on the second neural core; Transmitting the synchronization signal according to the L2 sync target, The first neural core store unit transmits a synchronization signal according to the L2 sync target to the second neural core load unit. Synchronization method of neural processing device. According to claim 27, The neural processing apparatus may include: a first neural processor including the first and second neural cores, the local interconnection, and the L2 sync path; a second neural processor different from the first neural processor; a global interconnection for transmitting data between first and second neural processors, and first and second semaphore memories respectively corresponding to the first and second neural processors; The global interconnection includes a data channel, a control channel, and an L3 sync channel through which data, a control signal, and a synchronization signal according to an L3 sync target are respectively transmitted between the first and second neural processors; The first neural processor generates the L3 sync target; storing a synchronization signal according to the L3 sync target in the second semaphore memory; The second neural processor performing synchronization through the value of the second semaphore memory. Synchronization method of neural processing device.

Images