KR102796247B1 - Task Manager, Neural processing device and Method for Managing task thereof - Google Patents

Abstract

The present invention discloses a task manager, a neural processing device, and a task management method thereof. A neural core according to an embodiment includes a core global which generates task information corresponding to a task descriptor, a task execution unit which receives the task information from the core global, performs a task according to the task information, and generates a completion signal for completion of the task, and a task manager which monitors an execution time of the task by the task execution unit to generate a timeout detection signal, and generates a timeout report according to the timeout detection signal.

Description

Task manager, neural processing device and method for managing task thereof {Task Manager, Neural processing device and Method for Managing task thereof}

The present invention relates to a task manager, a neural processing device, and a task management method thereof. Specifically, the present invention can perform control to temporarily put a task process on hold and then proceed with the task process again according to a waiting field included in a task descriptor. Accordingly, the present invention relates to a task manager, a neural processing device, and a task management method thereof, which support more efficient task processing.

In the past few years, artificial intelligence (AI) technology has been considered the most promising technology worldwide as a core technology of the 4th industrial revolution. The biggest problem with this AI technology is computing performance. The most important thing for AI technology that realizes human learning ability, reasoning ability, perception ability, and natural language execution ability is to process a lot of data quickly.

In the early days of artificial intelligence, the central processing unit (CPU) or graphics processing unit (GPU) of a conventional computer was used for deep learning learning and inference, but there are limitations to the tasks of deep learning learning and inference with high workloads, so the neural processing unit (NPU), which is structurally specialized for deep learning tasks, is in the spotlight. These neural processing units have multiple calculation units inside, and each calculation unit can operate in parallel to increase calculation efficiency.

Here, even if the tasks distributed to each computing device are processed in parallel, if the types of tasks are different or if a distinction is required between tasks, appropriate control of the execution timing of the tasks becomes necessary. In other words, controlling the execution timing of tasks by managing tasks so that there is a temporary waiting time between distributed tasks can be important not only in terms of the sequential processing of the overall tasks, but also in terms of processing efficiency and management.

Patent Registration No. 10-2258566

The object of the present invention is to provide a task manager that efficiently performs task management.

Another object of the present invention is to provide a neural processing device that efficiently performs task management.

Another object of the present invention is to provide a task management method of a neural processing device that efficiently performs task management.

The purposes of the present invention are not limited to the purposes mentioned above, and other purposes and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. In addition, it will be easily understood that the purposes and advantages of the present invention can be realized by the means and combinations thereof indicated in the claims.

According to some embodiments of the present invention for solving the above problem, a task manager includes a task buffer which receives a task from a command processor and creates a task descriptor for the task, a task queue in which the task descriptor received from the task buffer waits, and a runtime handle which creates task information corresponding to the task descriptor transferred from the task queue and transfers the task information to a core global, wherein the runtime handle queues the task descriptor in the task queue when the task descriptor includes a wait field.

Additionally, the runtime handle can release the waiting state of the task descriptor in response to a progress signal provided from the command processor.

Additionally, the runtime handle may include a progress signal counter from which the progress signal is received.

Additionally, the runtime handle can release the waiting state of the task descriptor through the progress signal previously received in the progress signal counter.

Additionally, the progress signal counter may be configured to receive at least two progress signals.

In addition, the runtime handle can generate task information corresponding to a task descriptor whose waiting state has been released or a task descriptor that does not include the waiting field and pass it to the core global, and provide the task descriptor corresponding to the task information passed to the core global as a check-in data in the dump passage.

Additionally, the above-described Dunn passage can receive a completion signal for the task information provided through the core global, and generate a completion report by checking out the checked-in task descriptor in response to the completion signal.

Additionally, the runtime handle can stop the transmission of task information to the core global based on a danger signal provided from the dungeon passage.

Additionally, the task queue may include a first queue that receives the task descriptor from the task buffer, a dependency checker that receives the task descriptor from the first queue and performs a dependency check of the received task descriptor, and a second queue that receives the task descriptor for which the dependency check has been completed from the dependency checker.

Additionally, the second queue includes a first task descriptor and a second task descriptor that are sequentially stored, and as the first task descriptor is waited in the second queue by the runtime handle, the second task descriptor may also be waited in the second queue.

According to some embodiments of the present invention for solving the above-described other problems, a neural processing device includes a task manager which generates task information corresponding to a task descriptor depending on whether the task descriptor includes a wait field, a neural core which performs a task according to the task information and generates a completion signal of the task, and a core global which receives task information on the task, transmits the task information to the neural core, and receives the completion signal of the task from the neural core.

In addition, the task manager may include a task passage that generates the task descriptor, selectively generates the task information according to the task descriptor and transfers it to the core global, and a dunn passage that checks in the task descriptor from the task passage, receives the completion signal, checks out the task descriptor, and generates the completion report.

In addition, the task passage includes a task buffer which receives a task from a command processor and creates a task descriptor for the task, a task queue in which the task descriptor received from the task buffer waits, and a runtime handle which creates task information corresponding to the task descriptor transferred from the task queue and transfers it to a core global, wherein the runtime handle queues the task descriptor in the task queue when the task descriptor includes a wait field, and the runtime handle can release the wait state of the task descriptor in response to a progress signal provided from the command processor.

In addition, the runtime handle can generate task information corresponding to a task descriptor whose waiting state has been released or a task descriptor that does not include the waiting field and pass it to the core global, and provide the task descriptor corresponding to the task information passed to the core global as check-in data to the dunned passage.

A task management method of a neural processing device according to some embodiments of the present invention for solving the above another problem includes the steps of receiving a task descriptor in a task queue, checking whether the task descriptor includes a wait field, and, if the task descriptor includes the wait field, waiting the task descriptor in the task queue.

Additionally, the method may further include a step of releasing the waiting state of the task descriptor in response to a progress signal provided from the command processor.

Additionally, the progress signal may be a signal provided after the waiting state of the task descriptor.

Additionally, the progress signal may be a signal provided before the task descriptor waiting state.

In addition, the method may further include a step of generating task information corresponding to a task descriptor whose waiting state has been released or a task descriptor that does not include the waiting field and transferring the same to a core global; and a step of providing a task descriptor corresponding to the task information transferred to the core global as a dump passage as check-in data.

Additionally, the step of receiving the task descriptor in the task queue may include the steps of receiving a task from a command processor, generating a task descriptor for the task, storing the task descriptor in a first queue, checking a dependency of the task descriptor, and storing the task descriptor for which the dependency check is completed in a second queue.

The task manager of the present invention, the neural processing device and the task management method thereof can control the execution timing of tasks through a runtime handle configured in the task manager. That is, it can be managed so as to have a temporary waiting time between distributed tasks, and the efficiency of task processing and management as well as the sequential processing of the overall tasks can be further increased.

In addition to the above-described contents, the specific effects of the present invention are described together with the specific matters for carrying out the invention below.

FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.
Figure 2 is a block diagram for explaining in detail the neural processing device of Figure 1.
Figure 3 is a block diagram for explaining the neural core SoC of Figure 2 in detail.
Figure 4 is a structural diagram for explaining in detail the global interconnection of Figure 3.
FIG. 5 is a block diagram for explaining the flow of control signals of the neural processing device of FIG. 1.
Figure 6 is a block diagram for explaining the neural processor of Figure 3 in detail.
FIG. 7 is a diagram illustrating a hierarchical structure of a neural processing device according to some embodiments of the present invention.
Figure 8 is a block diagram for explaining the neural core of Figure 6 in detail.
Figure 9 is a block diagram for explaining the LSU of Figure 8 in detail.
Figure 10 is a block diagram for explaining in detail the processing unit of Figure 8.
Figure 11 is a block diagram for explaining the L0 memory of Figure 8 in detail.
Figure 12 is a block diagram for explaining in detail the local memory bank of Figure 11.
Figure 13 is a block diagram for explaining the flow of data and control signals of the neural processing device of Figure 1.
Figure 14 is a block diagram illustrating the relationship between the command processor and the task manager of Figure 13.
Figure 15 is a block diagram for explaining in detail the structure of the task manager of Figure 8.
Figure 16 is a block diagram for explaining the table passage of Figure 15 in detail.
Figure 17 is a block diagram for explaining the task passage of Figure 15 in detail.
Figure 18 is a block diagram specifically explaining the function of the runtime handle.
Figure 19 is an example diagram for explaining the process of processing a task descriptor included in the 2_1 queue in response to a progress signal.
Figure 20 is an example diagram for explaining the process of processing a task descriptor included in the 2_1 queue in response to a pre-received progress signal.
Figure 21 is an example diagram for explaining a process of processing a task descriptor included in the 2_1 queue in response to multiple progress signals.
Figure 22 is an example diagram for explaining a process of processing a task descriptor included in the second_1 queue (Q2_1) through multiple progress signals received at a counter.
Figure 23 is a block diagram for explaining the Dunn passage of Figure 15 in detail.
Figure 24 is a block diagram for explaining in detail the report managing module of Figure 23.
Figure 25 is a diagram explaining data exchanged between the core global and neural core of Figure 15.
Figure 26 is a diagram for explaining the types of task descriptors stored in the first queue, the second queue, and the check-in buffer.
FIG. 27 is a diagram illustrating a first queue, a second queue, and a check-in buffer of a neural processing device according to some embodiments of the present invention.
FIG. 28 is a diagram illustrating a first queue, a second queue, and a check-in buffer of a neural processing device according to some embodiments of the present invention.
Figure 29 is a block diagram for explaining in detail the structure of the neural processing device of Figure 1.
FIG. 30 is a diagram illustrating a hierarchical structure of a command processor and a task manager of a neural processing device according to some embodiments of the present invention.
FIG. 31 is a diagram illustrating a hierarchical structure of command processors and task managers of a neural processing device according to some embodiments of the present invention.
Figure 32 is a block diagram for explaining memory reconfiguration of the neural processing system of Figure 1.
Figure 33 is a block diagram illustrating an example of memory reconfiguration of the neural processing system of Figure 1.
Figure 34 is an enlarged block diagram of part A of Figure 32.
Figure 35 is a drawing for explaining in detail the first memory bank of Figure 34.
Figure 36 is a block diagram illustrating the software hierarchy structure of the neural processing device of Figure 1.
Figure 37 is a conceptual diagram for explaining the deep learning operation performed by the neural processing device of Figure 1.
Figure 38 is a conceptual diagram for explaining the learning and inference operations of the neural network of the neural processing device of Figure 1.
FIG. 39 is a flowchart illustrating a task management method of a neural processing device according to some embodiments of the present invention.
Figure 40 is a flowchart for detailing the steps of receiving a task descriptor in the task queue of Figure 39.

The terms or words used in this specification and the claims should not be interpreted as limited to their general or dictionary meanings. In accordance with the principle that the inventor can define the concept of a term or word in order to best explain his or her invention, they should be interpreted as meanings and concepts that are consistent with the technical idea of the present invention. In addition, the embodiments described in this specification and the configurations illustrated in the drawings are only one embodiment in which the present invention is realized, and do not represent the entire technical idea of the present invention, so it should be understood that there may be various equivalents, modifications, and applicable examples that can replace them at the time of this application.

The terms first, second, A, B, etc., used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and/or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

The terminology used in this specification and claims is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. It should be understood that the terms "comprise" or "have" in this application do not exclude in advance the possibility of the presence or addition of features, numbers, steps, operations, components, parts or combinations thereof described in the specification.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

In addition, each configuration, process, procedure or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory to each other.

Hereinafter, with reference to FIGS. 1 to 37, a neural processing device according to some embodiments of the present invention will be described.

FIG. 1 is a block diagram illustrating a neural processing system according to some embodiments of the present invention.

Referring to FIG. 1, a neural processing system (NPS) according to some embodiments of the present invention may include a first neural processing device (1), a second neural processing device (2), and an external interface (3).

The first neural processing device (1) may be a device that performs a calculation using an artificial neural network. The first neural processing device (1) may be, for example, a device specialized in performing a deep learning calculation task. However, the present embodiment is not limited thereto.

The second neural processing device (2) may be a device having the same or similar configuration as the first neural processing device (1). The first neural processing device (1) and the second neural processing device (2) may be connected to each other through an external interface (3) to share data and control signals.

Although two neural processing devices are illustrated in FIG. 1, the neural processing system (NPS) according to some embodiments of the present invention is not limited thereto. That is, the neural processing system (NPS) according to some embodiments of the present invention may include three or more neural processing devices connected to each other through an external interface (3). In addition, conversely, the neural processing system (NPS) according to some embodiments of the present invention may include only one neural processing device.

At this time, the first neural processing device (1) and the second neural processing device (2) may each be processing devices other than neural processing devices. That is, the first neural processing device (1) and the second neural processing device (2) may each be a graphics processing unit (GPU), a central processing unit (CPU), and other types of processing devices. For convenience, the first neural processing device (1) and the second neural processing device (2) are described as neural processing devices below.

Figure 2 is a block diagram for explaining in detail the neural processing device of Figure 1.

Referring to FIG. 2, the first neural processing device (1) may include a neural core SoC (10), a CPU (20), an off-chip memory (30), a first non-volatile memory interface (40), a first volatile memory interface (50), a second non-volatile memory interface (60), a second volatile memory interface (70), and a control interface (CIF) (80).

The neural core SoC (10) may be a system on chip device. The neural core SoC (10) may be an artificial intelligence operation unit and an accelerator. The neural core SoC (10) may be, for example, one of a GPU (graphics processing unit), an FPGA (field programmable gate array), and an ASIC (application-specific integrated circuit). However, the present embodiment is not limited thereto.

The neural core SoC (10) can exchange data with other external computational units through the external interface (3). In addition, the neural core SoC (10) can be connected to a nonvolatile memory (31) and a volatile memory (32) through the first nonvolatile memory interface (40) and the first volatile memory interface (50), respectively.

The CPU (20) may be a control device that controls the system of the first neural processing device (1) and executes the operation of the program. The CPU (20) may be inefficient in performing parallel simple operations that are widely used in deep learning as a general-purpose operation unit. Therefore, the neural core SoC (10) may perform operations for deep learning inference and learning tasks and thus have high efficiency.

The CPU (20) can exchange data with other external operation units through the external interface (3). In addition, the CPU (20) can be connected to a nonvolatile memory (31) and a volatile memory (32) through a second nonvolatile memory interface (60) and a second volatile memory interface (70), respectively.

The CPU (20) can also transfer a task to the neural core SoC (10) through a command. At this time, the CPU (20) can be a kind of host that gives instructions to the neural core SoC (10). That is, the neural core SoC (10) can efficiently perform parallel computational tasks such as deep learning tasks according to the instructions of the CPU (20).

Off-chip memory (30) may be memory placed outside the chip of the neural core SoC (10). Off-chip memory (30) may include non-volatile memory (31) and volatile memory (32).

Nonvolatile memory (31) may be a memory that retains stored information even when power is not supplied. Nonvolatile memory (31) includes, for example, ROM (Read-Only Memory), PROM (Programmable Read-Only Memory), EAROM (Erasable Alterable ROM), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory) (e.g., NAND Flash memory, NOR Flash memory), UVEPROM (Ultra-Violet Erasable Programmable Read-Only Memory), FeRAM (Ferroelectric Random Access Memory), MRAM (Magnetoresistive Random Access Memory), PRAM (Phase-change Random Access Memory), SONOS (silicon-oxide-nitride-oxide-silicon), RRAM (Resistive Random Access Memory), NRAM (Nanotube Random Access Memory), magnetic computer memory (e.g., hard disk, diskette drive, magnetic tape), optical disk drive, and 3D crosspoint. It may include at least one of the memory (3D XPoint memory), but the present embodiment is not limited thereto.

Unlike nonvolatile memory (31), volatile memory (32) may be a memory that continuously requires power to maintain stored information. Volatile memory (32) may include, for example, at least one of DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), and DDR SDRAM (Double Data Rate SDRAM). However, the present embodiment is not limited thereto.

The first nonvolatile memory interface (40) and the second nonvolatile memory interface (60) may each include at least one of, for example, PATA (Parallel Advanced Technology Attachment), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), SATA (Serial Advanced Technology Attachment), and PCIe (PCI Express). However, the present embodiment is not limited thereto.

The first volatile memory interface (50) and the second volatile memory interface (70) may each be at least one of, for example, SDR (Single Data Rate), DDR (Double Data Rate), QDR (Quad Data Rate), and XDR (eXtreme Data Rate, Octal Data Rate). However, the present embodiment is not limited thereto.

The control interface (80) may be an interface for transmitting a control signal between the CPU (20) and the neural core SoC (10). The control interface (80) may transmit a command of the CPU (20) and a response of the neural core SoC (10) to the command. The control interface (80) may be, for example, PCIe (PCI Express), but is not limited thereto.

Figure 3 is a block diagram for explaining the neural core SoC of Figure 2 in detail.

Referring to FIGS. 2 and 3, a neural core SoC (10) may include at least one neural processor (1000), a shared memory (2000), a DMA (Direct Memory Access) (3000), a non-volatile memory controller (4000), a volatile memory controller (5000), a command processor (7000), and a global interconnection (6000).

A neural processor (1000) may be a computational unit that directly performs computational tasks. When there are multiple neural processors (1000), computational tasks may be assigned to each neural processor (1000). Each neural processor (1000) may be connected to each other through a global interconnection (6000).

The shared memory (2000) may be a memory shared by multiple neural processors (1000). The shared memory (2000) may store data of each neural processor (1000). In addition, the shared memory (2000) may receive data from an off-chip memory (30), temporarily store the data, and transmit the data to each neural processor (1000). Conversely, the shared memory (2000) may receive data from a neural processor (1000), temporarily store the data, and transmit the data to the off-chip memory (30) of FIG. 2.

The shared memory (2000) may require relatively fast memory. Accordingly, the shared memory (2000) may include, for example, SRAM. However, the present embodiment is not limited thereto. That is, the shared memory (2000) may also include DRAM.

The shared memory (2000) may be a memory corresponding to the SoC level, i.e., L2 (level 2). Therefore, the shared memory (2000) may also be defined as L2 shared memory.

DMA (3000) can directly control the movement of data without the CPU (20) or neural processor (1000) having to control the input/output of data. Accordingly, DMA (3000) can control the movement of data between memories to minimize the number of interrupts of the CPU (20) or neural processor (1000).

DMA (3000) can control data movement between shared memory (2000) and off-chip memory (30). Through the authority of DMA (3000), a non-volatile memory controller (4000) and a volatile memory controller (5000) can perform data movement.

A nonvolatile memory controller (4000) can control a read or write operation to a nonvolatile memory (31). The nonvolatile memory controller (4000) can control the nonvolatile memory (31) through a first nonvolatile memory interface (40).

The volatile memory controller (5000) can control a read or write operation on the volatile memory (32). In addition, the volatile memory controller (5000) can perform a refresh operation on the volatile memory (32). The volatile memory controller (5000) can control the volatile memory (32) through the first volatile memory interface (50).

The command processor (7000) can be connected to the control interface (80). The command processor (7000) can receive a control signal from the CPU (20) through the control interface (80). The command processor (7000) can generate a task through the control signal received from the CPU (20) and transmit the task to each neural processor (1000). In addition, the command processor (7000) can receive a completion report for the task from each neural processor (1000).

A global interconnection (6000) can interconnect at least one neural processor (1000), a shared memory (2000), a DMA (3000), a non-volatile memory controller (4000), a command processor (7000), and a volatile memory controller (5000). Additionally, an external interface (3) can also be connected to the global interconnection (6000). The global interconnection (6000) can be a path along which data moves between at least one neural processor (1000), a shared memory (2000), a DMA (3000), a non-volatile memory controller (4000), a volatile memory controller (5000), a command processor (7000), and the external interface (3).

The global interconnection (6000) can transmit not only data but also control signals and signals for synchronization. In some embodiments of the present invention, the neural processing device can have each neural processor (1000) directly transmit and receive synchronization signals. Accordingly, the latency due to transmission of the synchronization signal generated by the command processor (7000) can be minimized.

That is, when there are multiple neural processors (1000), there may be a dependency of individual tasks in which the task of a certain neural processor (1000) must be completed before the next neural processor (1000) can start a new task. The completion and start of these individual tasks can be confirmed through a synchronization signal, but in the existing technology, the command processor (7000) or the host, i.e., the CPU (20), is exclusively responsible for receiving this synchronization signal and instructing the start of a new task.

However, as the number of neural processors (1000) increases and the design of task dependencies becomes more complex, the number of such synchronization signals increases exponentially, and the latency according to each synchronization signal can significantly reduce the efficiency of the task.

Therefore, the neural processing device according to some embodiments of the present invention can have each neural processor (1000) directly transmit a portion of a synchronization signal to another neural processor (1000) according to the dependency of the task, instead of the command processor (7000). In this case, compared to the method managed by the command processor (7000), multiple neural processors (1000) can perform the synchronization task in parallel, thereby minimizing the latency due to synchronization.

In addition, the command processor (7000) must perform task scheduling of the neural processors (1000) according to task dependency, and the overhead of such scheduling may also increase significantly as the number of neural processors (1000) increases. Therefore, in the neural processing device according to some embodiments of the present invention, the scheduling task is partially performed by each neural processor (1000), and thus the scheduling burden is reduced, and thus the performance of the device may be improved.

In addition, the neural processing device according to some embodiments of the present invention can monitor whether a task is completed, an event occurs, whether there is a delay in task execution, etc. in the neural core of each neural processor (1000), and by minimizing the intervention of the command processor (7000), the burden on the command processor (7000) is reduced, so that the performance of the device can be improved.

In addition, the neural processing device according to some embodiments of the present invention can selectively generate a completion report by setting whether to monitor tasks for each task, and can be configured to modify whether to generate a completion report when a report to the command processor (7000) is required. Accordingly, it is possible to report on tasks that require warnings without performing monitoring on all tasks, and stable monitoring of tasks can be possible while reducing the burden on the command processor (7000).

Figure 4 is a structural diagram for explaining in detail the global interconnection of Figure 3.

Referring to FIG. 4, the global interconnection (6000) may include a data channel (6100), a control channel (6200), and an L2 sync channel (6300).

The data channel (6100) may be a dedicated channel for transmitting data. At least one neural processor (1000), a shared memory (2000), a DMA (3000), a nonvolatile memory controller (4000), a volatile memory controller (5000), and an external interface (3) may exchange data with each other through the data channel (6100).

The control channel (6200) may be a dedicated channel for transmitting a control signal. At least one neural processor (1000), a shared memory (2000), a DMA (3000), a nonvolatile memory controller (4000), a volatile memory controller (5000), a command processor (7000), and an external interface (3) may exchange control signals with each other through the control channel (6200). In particular, the command processor (7000) may transmit various control signals to each neural processor (1000).

The L2 sync channel (6300) may be a dedicated channel for transmitting a synchronization signal. At least one neural processor (1000), a shared memory (2000), a DMA (3000), a nonvolatile memory controller (4000), a volatile memory controller (5000), a command processor (7000), and an external interface (3) may exchange synchronization signals with each other through the L2 sync channel (6300).

The L2 sync channel (6300) is set as a dedicated channel within the global interconnection (6000) so that synchronization signals can be transmitted quickly without overlapping with other channels. Accordingly, the neural processing device according to some embodiments of the present invention can smoothly perform synchronization work using the existing global interconnection (6000) without requiring new wiring work.

FIG. 5 is a block diagram for explaining the flow of control signals of the neural processing device of FIG. 1.

Referring to FIG. 5, the CPU (20) can transmit a control signal to the command processor (7000) through the control interface (80). At this time, the control signal may be a signal instructing to perform each operation, such as an operation task or a data load/store task.

The command processor (7000) can receive a control signal and transmit the control signal to at least one neural processor (1000) through a control channel (6200). Each control signal can be stored in the neural processor (1000) as a respective task.

Figure 6 is a block diagram for explaining the neural processor of Figure 3 in detail.

Referring to FIGS. 3 to 6, a neural processor (1000) may include at least one neural core (100), a local interconnection (200), an L1 sync path (300), an L1 shared memory (400), a core global (500), a task manager (600), and an L1 LSU (700).

At least one neural core (100) can perform the work of the neural processor (1000) by dividing it. For example, there can be eight neural cores (100). However, the present embodiment is not limited thereto. In FIGS. 3 and 5, multiple neural cores (100) are illustrated as being included in the neural processor (1000), but the present embodiment is not limited thereto. That is, the neural processor (1000) can be configured with only one neural core (100).

The neural core (100) can receive task information from the core global (500) and perform a task according to the task information. At this time, the task can be defined by a control signal, and the task can be any one of the memory operations. The memory operation can be, for example, any one of the micro DMA (μDMA), LP micro DMA (Low Priority μDMA), store μDMA (STμDMA), and preprocessing tasks.

The L1 shared memory (400) may be a memory shared by each neural core (100) within the neural processor (1000). The L1 shared memory (400) may store data of each neural core (100). In addition, the L1 shared memory (400) may receive data from the shared memory (2000) of FIG. 4, temporarily store the data, and transmit the data to each neural core (100). Conversely, the L1 shared memory (400) may receive data from the neural core (100), temporarily store the data, and transmit the data to the shared memory (2000) of FIG. 3.

The L1 shared memory (400) may be a memory corresponding to a neural processor level, i.e., L1 (level 1). The L2 shared memory, i.e., shared memory (2000), may be shared by the neural processor (1000), and the L1 shared memory (400) may be shared by the neural core (100).

The L1 LSU (700) can receive at least one of data, a control signal, and a synchronization signal from the outside through the global interconnection (6000). The L1 LSU (700) can transmit at least one of the data, the control signal, and the synchronization signal received to the L1 shared memory (400). Similarly, the L1 LSU (700) can transmit at least one of the data, the control signal, and the synchronization signal to the outside through the global interconnection (6000). In addition, the L1 LSU (700) can transmit and receive at least one of the data, the control signal, and the synchronization signal for each of the neural cores (100).

The neural core (100) can receive task information from the core global (500) and perform a task according to the task information. At this time, the task may be a computational task (operational task) or a task related to a memory operation. The task may be defined by a control signal. The task information is information about the task, and may be information about the type of the task, the form of the task, additional information about the task, etc.

The neural core (100) can transmit a completion signal when the task is completed to the core global (500).

The task manager (600) can receive a task from a control interconnection (CI). At this time, the control interconnection (CI) can be a general term for a transmission interface that transmits a task from a command processor (7000). That is, the control interconnection (CI) can include a control channel (6200) and a local interconnection (200).

The task manager (600) can receive a task, generate task information, and transmit it to the core global (500). In addition, the task manager (600) can receive a completion signal through the core global (500), generate a completion report accordingly, and transmit it to the command processor (7000) through the control interconnection (CI).

The core global (500) may be a wire structure that is hardware-connected within the neural core (100). Although not shown, the core global (500) may be a structure that connects all of the neural core (100), the L1 shared memory (400), the L1 LSU (700), and the task manager (600). Accordingly, the local interconnection (200) and the L1 sync path (300) may also be included in the core global (500). However, the present embodiment is not limited thereto.

The core global (500) can receive task information from the task manager (600) and transmit it to the neural core (100), and receive a completion signal for the same from the neural core (100). Then, the core global (500) can transmit the completion signal to the task manager (600).

A local interconnection (200) can interconnect at least one neural core (100), an L1 shared memory (400), a core global (500), a task manager (600), and an L1 LSU (700). The local interconnection (200) can be a path through which data moves between at least one neural core (100), an L1 shared memory (400), a core global (500), a task manager (600), and an L1 LSU (700). The local interconnection (200) can be connected to a global interconnection (6000) of FIG. 3 to transmit data.

The L1 sync path (300) can connect at least one neural core (100), an L1 shared memory (400), a core global (500), a task manager (600), and an L1 LSU (700) to each other. The L1 sync path (300) can be a path along which a synchronization signal of at least one neural core (100), an L1 shared memory (400), a core global (500), a task manager (600), and an L1 LSU (700) travels.

The L1 sync path (300) may be formed physically separately from the local interconnection (200). In the case of the local interconnection (200), unlike the global interconnection (6000), a sufficient channel may not be formed inside. In this case, the L1 sync path (300) may be formed separately so that transmission of a synchronization signal can be performed quickly and without delay. The L1 sync path (300) may be used for synchronization performed at a level one level lower than the L2 sync channel (6300) of the global interconnection (6000).

FIG. 7 is a diagram illustrating a hierarchical structure of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 7, the neural core SoC (10) may include at least one neural processor (1000). Each neural processor (1000) may transmit data to each other through a global interconnection (6000).

Each neural processor (1000) may include at least one neural core (100). The neural core (100) may be a processing unit unit optimized for a deep learning operation task. The neural core (100) may be a processing unit unit corresponding to one operation of a deep learning operation task. That is, the deep learning operation task may be expressed as a sequential or parallel combination of several operations. The neural core (100) may be a processing unit unit capable of processing one operation each, and may be the minimum operation unit that a compiler may consider for scheduling.

The neural processing device according to the present embodiment can achieve fast and efficient scheduling and execution of computational tasks by configuring the minimum computational unit considered from the compiler scheduling perspective and the hardware processing unit scale to be the same.

That is, if the hardware's divisible processing unit is too large compared to the computational task, inefficiency in the computational task may occur in the operation of the processing unit. On the other hand, it is not appropriate to schedule a processing unit smaller than the compiler's minimum scheduling unit, an operation, every time, as this may result in scheduling inefficiency and increase hardware design costs.

Therefore, the present embodiment can simultaneously satisfy the scheduling of fast computational tasks and efficient execution of computational tasks without wasting hardware resources by similarly adjusting the scale of the scheduling unit of the compiler and the scale of the hardware processing unit.

Figure 8 is a block diagram for explaining the neural core of Figure 6 in detail.

Referring to FIG. 8, the neural core (100) may include an LSU (Load/Store Unit) (110), an L0 memory (120), a weight buffer (130), an activation LSU (140), an activation buffer (150), and a processing unit (160).

The LSU (110) can receive at least one of data, a control signal, and a synchronization signal from the outside through the local interconnection (200) and the L1 sync path (300). The LSU (110) can transmit at least one of the data, the control signal, and the synchronization signal received to the L0 memory (120). Similarly, the LSU (110) can transmit at least one of the data, the control signal, and the synchronization signal to the outside through the local interconnection (200) and the L1 sync path (300).

Specifically, a micro-DMA task may be a task in which a neural core (100) loads a program or data from a shared memory (2000) or an off-chip memory (30) to an L0 memory (120). Unlike a general micro-DMA task, an LP micro-DMA task may be a load task for a program or data to be used later, rather than a current program or data. Since this task has a lower priority, it may be identified differently from a micro-DMA task. An ST micro-DMA task may be a store task in which data is stored from an L0 memory (120) of a neural core (100) to a shared memory (2000) or an off-chip memory (30). A pre-processing task may include a task in which data, such as a large lookup table, is pre-loaded from the CPU (20).

Figure 9 is a block diagram for explaining the LSU of Figure 8 in detail.

Referring to FIG. 9, the LSU (110) may include a local memory load unit (111a), a local memory store unit (111b), a neural core load unit (112a), a neural core store unit (112b), a load buffer (LB), a store buffer (SB), a load engine (113a), a store engine (113b), and a conversion index buffer (114).

The local memory load unit (111a) can fetch a load instruction for the L0 memory (120) and issue the load instruction. When the local memory load unit (111a) provides the issued load instruction to the load buffer (LB), the load buffer (LB) can sequentially transmit a memory access request to the load engine (113a) in the input order.

In addition, the local memory store unit (111b) can fetch a store instruction for the L0 memory (120) and issue the store instruction. When the local memory store unit (111b) provides the issued store instruction to the store buffer (SB), the store buffer (SB) can sequentially transmit a memory access request to the store engine (113b) in the input order.

The neural core load unit (112a) can fetch a load instruction for the neural core (100) and issue the load instruction. When the neural core load unit (112a) provides the issued load instruction to the load buffer (LB), the load buffer (LB) can sequentially transmit a memory access request to the load engine (113a) in the input order.

In addition, the neural core store unit (112b) can fetch a store instruction for the neural core (100) and issue the store instruction. When the neural core store unit (112b) provides the issued store instruction to the store buffer (SB), the store buffer (SB) can sequentially transmit a memory access request to the store engine (113b) in the input order.

The load engine (113a) can receive a memory access request and load data through the local interconnection (200). At this time, the load engine (113a) can quickly find data by using a conversion table of recently used logical addresses and physical addresses in the conversion index buffer (114). If the logical address of the load engine (113a) is not in the conversion index buffer (114), address conversion information can be found in another memory.

The store engine (113b) can receive a memory access request and retrieve data through the local interconnection (200). At this time, the store engine (113b) can quickly find data by using a conversion table of recently used logical addresses and physical addresses in the conversion index buffer (114). If the logical address of the store engine (113b) is not in the conversion index buffer (114), address conversion information can be found in another memory.

The load engine (113a) and the store engine (113b) can send a synchronization signal to the L1 sync path (300). At this time, the synchronization signal can mean that the work has been completed.

Again, referring to FIG. 8, the L0 memory (120) is a memory located inside the neural core (100) and can temporarily store all input data required for the neural core (100) to be processed by receiving them from the outside. In addition, the L0 memory (120) can temporarily store output data calculated by the neural core (100) for transmission to the outside.

The L0 memory (120) can transmit input activation (Act_In) to the activation buffer (150) by the activation LSU (140) and receive output activation (Act_Out). In addition to the activation LSU (140), the L0 memory (120) can directly transmit and receive data with the processing unit (160). That is, the L0 memory (120) can transmit and receive data with each of the PE array (163) and the vector unit (164). The L0 memory (120) may be a memory corresponding to a neural core level. In this case, the L0 memory (120) may be a private memory of the neural core.

The L0 memory (120) can transmit data such as activation or weight through a data path. The L0 memory (120) can send and receive a synchronization signal through a separate dedicated path, the L0 Sync Path. The L0 memory (120) can send and receive a synchronization signal through the L0 Sync Path with, for example, the LSU (110), the weight buffer (130), the activation LSU (140), and the processing unit (160).

The weight buffer (130) can receive the weight from the L0 memory (120). The weight buffer (130) can transfer the weight to the processing unit (160). The weight buffer (130) can temporarily store the weight before transferring the weight.

Input activation (Act_In) and output activation (Act_Out) can refer to the input and output values of a layer of a neural network. In this case, if the neural network has multiple layers, the output value of the previous layer becomes the input value of the next layer, so the output activation (Act_Out) of the previous layer can be utilized as the input activation (Act_In) of the next layer.

Weight can refer to a parameter that is multiplied by the input activation (Act_In) input from each layer. Weight is adjusted and determined during the deep learning learning stage, and can be used to derive output activation (Act_Out) through a fixed value during the inference stage.

The activation LSU (140) can transfer input activation (Act_In) from the L0 memory (120) to the activation buffer (150) and transfer output activation (Act_Out) from the activation buffer (150) to the on-chip buffer. That is, the activation LSU (140) can perform both load and store operations of activation.

The activation buffer (150) can provide input activation (Act_In) to the processing unit (160) and receive output activation (Act_Out) from the processing unit (160). The activation buffer (150) can temporarily store input activation (Act_In) and output activation (Act_Out).

The activation buffer (150) can quickly provide activation to a processing unit (160) with a large amount of computation, and quickly receive activation to increase the computation speed of the neural core (100).

The processing unit (160) may be a module that performs an operation. The processing unit (160) may perform not only a one-dimensional operation but also a two-dimensional matrix operation, i.e., a convolution operation. The processing unit (160) may receive an input activation (Act_In), multiply it by a weight, and then add it to generate an output activation (Act_Out).

Figure 10 is a block diagram for explaining in detail the processing unit of Figure 8.

Referring to FIGS. 8 and 10, the processing unit (160) may include a PE array (163), a vector unit (164), a column register (161), and a row register (162).

The PE array (163) can perform multiplication by receiving input activation (Act_In) and weight (Weight). At this time, the input activation (Act_In) and weight (Weight) can be operated through convolution in matrix form, respectively. Through this, the PE array (163) can generate output activation (Act_Out). However, the present embodiment is not limited thereto. The PE array (163) can generate any type of output other than output activation (Act_Out).

The PE array (163) may include at least one processing element (163_1). The processing elements (163_1) may be aligned with each other and may each perform a multiplication for one input activation (Act_In) and one weight (Weight).

The PE array (163) can generate a partial sum by adding the values for each multiplication. This partial sum can be utilized as an output activation (Act_Out). Since the PE array (163) performs a two-dimensional matrix multiplication, it can also be referred to as a two-dimensional matrix compute unit (2D matrix compute unit).

The vector unit (164) can perform one-dimensional operations. The vector unit (164) can perform deep learning operations together with the PE array (163). Through this, the processing unit (160) can be specialized for necessary operations. That is, the neural core (100) has operation modules that perform a large amount of two-dimensional matrix multiplication and one-dimensional operations, respectively, so that it can efficiently perform deep learning operations.

The column register (161) can receive a first input (I1). The column register (161) can receive the first input (I1), divide it, and provide it to each column of the PE array (163).

The low register (162) can receive the second input (I2). The low register (162) can receive the second input (I2) and divide it to provide it to each row of the PE array (163).

The first input (I1) can be an input activation (Act_In) or a weight (Weight). The second input (I2) can be a value other than the first input (I1) among the input activation (Act_In) or the weight (Weight). Alternatively, the first input (I1) and the second input (I2) can be values other than the input activation (Act_In) and the weight (Weight).

Figure 11 is a block diagram for explaining the L0 memory of Figure 8 in detail.

Referring to FIG. 11, the L0 memory (120) may include a scheduler (121) and at least one local memory bank (122).

When data is stored in the L0 memory (120), the scheduler (121) can receive data from the load engine (113a). At this time, the data can be allocated to a local memory bank (122) in a round robin manner. Accordingly, the data can be stored in at least one of the local memory banks (122).

Conversely, when data is loaded from L0 memory (120), the scheduler (121) can receive data from the local memory bank (122) and transfer it to the store engine (113b). The store engine (113b) can store the data externally through the local interconnection (200).

Figure 12 is a block diagram for explaining in detail the local memory bank of Figure 11.

Referring to FIG. 12, a local memory bank (122) may include a local memory bank controller (122_1) and a local memory bank cell array (122_2).

The local memory bank controller (122_1) can manage read and write operations through the address of data stored in the local memory bank (122). That is, the local memory bank controller (122_1) can manage the input/output of data as a whole.

The local memory bank cell array (122_2) may have a structure in which cells in which data is directly stored are aligned in rows and columns. The local memory bank cell array (122_2) may be controlled by the local memory bank controller (122_1).

FIG. 13 is a block diagram for explaining the flow of data and control signals of the neural processing device of FIG. 1, and FIG. 14 is a block diagram for explaining the relationship between the command processor and the task manager of FIG. 13.

Referring to FIGS. 13 and 14, the neural processor (1000) may include at least one neural core (100). Each neural processor (1000) may include a task manager (600) and an L1 LSU (700) therein. The task managers (600) may exchange control signals and their responses with the command processor (7000) through a control interconnection (CI).

In contrast, the L1 LSU (700) can exchange data through a data interconnection and memory (DIM). The data interconnection and memory (DIM) can include an interconnection for transmitting data and a memory where data is shared. Specifically, the data interconnection and memory (DIM) can include a local interconnection (200) and a data channel (6100). In addition, the data interconnection and memory (DIM) can include an L1 shared memory (400), a shared memory (2000), and a volatile memory (32). However, the present embodiment is not limited thereto.

The task manager (600) can be controlled by the command processor (7000). That is, the command processor (7000) can transmit a task to the task manager (600) through a control signal, and the task manager (600) can transmit a task completion report to the command processor (7000). At least one task manager (600) can be included in the neural processor (1000). In addition, if there are multiple neural processors (1000), the number of task managers (600) can increase. All of these multiple task managers (600) can be controlled by the command processor (7000).

Figure 15 is a block diagram for explaining in detail the structure of the task manager of Figure 8.

Referring to FIGS. 8, 9 and 15, the task manager (600) may include a table passage (610), a task passage (620) and a dump passage (630).

The table passage (610) can receive a table update request (TURQ) for updating a matching table of physical addresses and logical addresses from the control channel (6200) and transmit it to the core global (500). At this time, the table update request can be transmitted from the command processor (7000) through the control channel (6200).

The task passage (620) can receive a task from the control channel (6200), generate task information according to the task, and transmit the task information to the core global (500). At this time, the task can be transmitted from the command processor (7000) through the control channel (6200). The core global (500) can transmit the task information to the neural core (100). The neural core (100) can perform a task according to the transmitted task information, and transmit a completion signal to the core global (500).

The core global (500) can transmit a completion signal to the Done passage (630). The Done passage (630) can receive the completion signal and generate a completion report (DNrp) of the task. The Done passage (630) can transmit the completion report (DNrp) to the command processor (7000) through the control channel (6200).

Additionally, a table update request (TURQ) of a table passage (610) can be transmitted to a neural core (100) through a core global (500). At this time, a table of a transformation index buffer (114) inside an LSU (110) of a neural core (100) can be updated.

Figure 16 is a block diagram for explaining the table passage of Figure 15 in detail.

Referring to FIG. 16, the table passage (610) may include a table buffer (611) and first to mth update request queues (611a1 to 611am).

The table buffer (611) can store table update requests (TURQ) in which physical addresses and logical addresses are matched, transmitted from the command processor (7000). When the core global (500) fetches these table update requests (TURQ), each table update request (TURQ) can be stored in the first to mth update request queues (611a1 to 611am).

Each of the first to mth update request queues (611a1 to 611am) may store different types of table update requests (TURQs). For example, the different types of table update requests (TURQs) may include at least one of a neural core TLB update request, a micro DMA TLB update request, an LP micro DMA TLB update request, and an ST micro DMA TLB update request. However, the present embodiment is not limited thereto. In some embodiments, each of the first to mth update request queues (611a1 to 611am) may include the same type of table update requests (TURQs).

In addition, the first to mth update request queues (611a1 to 611am) may each be a general queue, i.e., a queue that accepts various types of requests. Accordingly, the first to mth update request queues (611a1 to 611am) may each accept requests regardless of type.

Each of the first to mth update request queues (611a1 to 611am) can transmit a table update request (TURQ) to the core global (500), and the request can be delivered to the LSU (110) through the core global (500). At this time, the table of the translation index buffer (114) inside the LSU (110) can be updated.

Figure 17 is a block diagram for explaining the task passage of Figure 15 in detail.

Referring to FIG. 17, a task passage (620) includes a task buffer (621), a task queue (622), and a runtime handle (RH).

The task buffer (621) can store a task according to a control signal transmitted from the command processor (7000). The task buffer (621) can store a task in the form of a task descriptor in a task queue (622) by a task fetching operation of the core global (500).

The task queue (622) is configured to sequentially store task descriptors, perform dependency checks on the stored task descriptors, and sequentially store task descriptors for which dependency checks have been completed. In an embodiment, the task queue (622) may include a first queue (Q1), a dependency checker (DPc), and a second queue (Q2).

The first queue (Q1) can store a task descriptor provided from the task buffer (621). The task buffer (621) can transfer the task descriptor to the first queue (Q1) and generate a transfer dump report (TRrp). The task buffer (621) can transfer the transfer dump report (TRrp) to the dump passage (630). The transfer dump report (TRrp) can be a report on a task transferred to the first queue (Q1).

The first queue (Q1) can store task descriptors by dividing them according to the type of the task descriptor. In Fig. 17, n first queues (Q1) are illustrated. In this case, n can be a natural number. That is, the first queue (Q1) can be at least one.

At this time, the first queue (Q1) may include the 1_1 to 1_nth queues (Q1_1 to Q1_n). The 1_1st queue (Q1_1) may store the first task descriptor (Tsk_d1), and the 1_2nd queue (Q1_2) may store the second task descriptor (Tsk_d2). The 1_nth queue (Q1_n) may store the nth task descriptor (Tsk_dn).

The first to nth task descriptors (Tsk_d1 to Tsk_dn) may be of different types or may be of the same type. Alternatively, some of the first to nth task descriptors (Tsk_d1 to Tsk_dn) may be of the same type and some may be of different types.

The dependency checker (DPc) can receive a dependency update request (DFURQ). The dependency update request (DFURQ) can notify a change in dependency as a completed task occurs according to a defined dependency between specific tasks. That is, each task descriptor can include a dependency field for which there is a dependency on a task. At this time, when a task included in the dependency field is completed, it should be updated in a form in which it is removed from the dependency field. Therefore, the dependency update request (DFURQ) can include an update request for the dependency field of the task descriptor.

The dependency checker (DPc) can sequentially transfer the first to nth task descriptors (Tsk_d1 to Tsk_dn) for which dependency checking has been completed to the second queue (Q2).

At this time, the second queue (Q2) may include the 2nd_1 to 2nd_n queues (Q2_1 to Q2_n). The 2nd_1 queue (Q2_1) may store the first task descriptor (Tsk_d1), and the 2nd_2nd queue (Q2_2) may store the second task descriptor (Tsk_d2). The 2nd_nth queue (Q2_n) may store the nth task descriptor (Tsk_dn). The number of the second queues (Q2) may be equal to the number of the first queues (Q1).

The first queue (Q1) of the task queue (622) can store the first to nth task descriptors (Tsk_d1 to Tsk_dn) before the dependency check, and the second queue (Q2) of the task queue (622) can store the first to nth task descriptors (Tsk_d1 to Tsk_dn) for which the dependency check has been completed.

The runtime handle (RH) can extract necessary information from each of the first to nth task descriptors (Tsk_d1 to Tsk_dn) stored in the second queue (Q2) to generate the first to nth task information (Tsk_d1' to Tsk_dn'). The runtime handle (RH) can transfer the first to nth task information (Tsk_d1' to Tsk_dn') to the core global (500). At this time, the first to nth task information (Tsk_d1' to Tsk_dn') may correspond to the first to nth task descriptors (Tsk_d1 to Tsk_dn), respectively. At this time, the first to nth task information (Tsk_d1' to Tsk_dn') may be identical to the first to nth task descriptors (Tsk_d1 to Tsk_dn), respectively. However, the present embodiment is not limited thereto.

The runtime handle (RH) can transmit check-in data (ChI) to the dump passage (630). The check-in data (ChI) can include first to nth task descriptors (Tsk_d1 to Tsk_dn). The check-in data (ChI) can be data that informs the dump passage (630) that the first to nth task information (Tsk_d1' to Tsk_dn') corresponding to the first to nth task descriptors (Tsk_d1 to Tsk_dn) have left the task passage (620) and been transferred to the core global (500) for processing. The dump passage (630) monitors whether the task descriptor according to the check-in data (ChI) is performed.

The first to nth task descriptors (Tsk_d1 to Tsk_dn) may be configured to include a wait field. The wait field may be an item pre-designated by software. A task descriptor with a wait field set is converted into task information at the check-in timing and is not transferred to the core global (500), but waits in the second queue (Q2). Here, the check-in timing means the point in time when all preceding task descriptors are transferred to the core global (500), and the check-in may mean transferring task information to the core global (500) and transferring the corresponding task descriptor to the throw passage (630).

A task descriptor with a wait field set has a wait state. The wait field can be a means of controlling the execution timing of a task, and the task's work flow and execution timing can be controlled through the runtime handle (RH).

Figure 18 is a block diagram specifically explaining the function of the runtime handle.

Referring to FIG. 18, the 2nd_1st queue to the 2nd_nth queue (Q2_1 to Q2_n) can store a plurality of task descriptors for which dependency checks have been completed. The 2nd_1st queue (Q2_1) can include the 1st_1st task descriptor (Tsk_d11) to the 1_kth task descriptor (Tsk_d1k). The 1st_1st task descriptor (Tsk_d11) to the 1_kth task descriptor (Tsk_d1k) can be sequentially stored in the 2nd_1st queue (Q2_1). Here, k is a natural number. Similarly, the second_2 queue (Q2_2) may include the second_1st task descriptor (Tsk_d21) to the second_kth task descriptor (Tsk_d2k), and the second_nth queue (Q2_n) may include the n_1st task descriptor (Tsk_dn1) to the n_kth task descriptor (Tsk_dnk). In an exemplary embodiment, the first_2nd task descriptor (Tsk_d12), the second_3rd task descriptor (Tsk_d23), and the n_kth task descriptor (Tsk_dnk) may have wait fields set, and the remaining task descriptors may not have wait fields set.

In the 2_1 queue (Q2_1), the 1_1 task descriptor (Tsk_d11) can be converted into task information at the check-in timing and transferred to the core global (500) without the wait field being set, and the corresponding check-in data (ChI) can be transferred to the dump passage (630). In contrast, the runtime handle (RH) can wait in the 2_1 queue (Q2_1) without checking in the 1_2 task descriptor (Tsk_d12) even if the 1_2 task descriptor (Tsk_d12) with the wait field set corresponds to the check-in timing.

As the 1_2 task descriptor (Tsk_d12) waits in the 2_1 queue (Q2_1) in a waiting state, other task descriptors (Tsk_d13 to Tsk_d1k) following the 1_2 task descriptor (Tsk_d12) are not checked in and continue to wait in the 2_1 queue (Q2_1).

The runtime handle (RH) can control the progress timing for all of the 2_1st to 2_nth queues (Q2_1 to Q2_n). The 2_1st and 2_2nd task descriptors (Tsk_d21, Tsk_d22) preceding the 2_3rd task descriptor (Tsk_d23) can be converted into task information at the check-in timing and transferred to the core global (500), and the corresponding check-in data (ChI) is transferred to the dump passage (630). In contrast, the 2_3rd task descriptor (Tsk_d23) cannot be checked in by the runtime handle (RH) and waits in the 2_2nd queue (Q2_2).

In addition, task descriptors preceding the n_k-th task descriptor (Tsk_dnk) can be converted into task information through the runtime handle (RH) and passed to the core global (500), and the corresponding check-in data (ChI) is passed to the dump passage (630), but the n_k-th task descriptor (Tsk_dnk) including the wait field waits in the 2nd_n queue (Q2_n).

That is, the runtime handle (RH) can determine the status of the task descriptor as a progress state or a waiting state by checking whether the task descriptor stored in the 2nd_1 to 2nd_n queues (Q2_1 to Q2_n) includes a waiting field.

The runtime handle (RH) can receive a progress signal (Run) from the command processor (7000). The progress signal (Run) can be provided from the command processor (7000) through a control interconnection (CI). The command processor (7000) can transmit the progress signal (Run) in response to transmitting a task including a waiting field, but the embodiment of the present invention is not limited thereto. The command processor (7000) can transmit the progress signal (Run) to the runtime handle (RH) at a predetermined cycle. The runtime handle (RH) can change a task descriptor in a waiting state to a progress state in response to the progress signal (Run). The runtime handle (RH) can be configured to receive the progress signal (Run) and store the progress signal (Run) for a predetermined period of time. The runtime handle (RH) may include at least one of a register (Rs) and a counter (Rc) that receives and stores a progress signal (Run), but the embodiment of the present invention is not limited thereto.

Hereinafter, with reference to FIGS. 19 to 21, the process in which the runtime handle (RH) performs wait-run control according to the progress signal (Run) will be described in more detail.

Figure 19 is an example diagram for explaining the process of processing a task descriptor included in the second_1 queue (Q2_1) in response to a progress signal.

Referring to FIG. 19, the process of processing task descriptors (Tsk_d11 to Tsk_d13) included in the second_1 queue (Q2_1) according to the change in time (t0 -> t3) can be confirmed. Here, the time (t0 to t3) is an exemplary time unit divided according to the processing status of each task descriptor, and is not limited to a specific time. In the second_1 queue (Q2_1), the first_2 task descriptor (Tsk_d12) includes a waiting field, but the remaining task descriptors do not include waiting fields.

At time 0 (t0), the 1_1 task descriptor (Tsk_d11) corresponds to the check-in timing. That is, the 1_1 task descriptor (Tsk_d11) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_1 task descriptor (Tsk_d11) that does not include a wait field as the progress status. The runtime handle (RH) can convert the 1_1 task descriptor (Tsk_d11) into 1_1 task information (Tsk_d11') and transfer it to the core global (500), and transfer the corresponding check-in data (ChI) to the dunned passage (630).

At the first time (t1), the 1_2 task descriptor (Tsk_d12) corresponds to the check-in timing. That is, the 1_2 task descriptor (Tsk_d12) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). However, the runtime handle (RH) can determine the 1_2 task descriptor (Tsk_d12) including the wait field as a wait state, and can put the 1_2 task descriptor (Tsk_d12) on hold in the 2_1 queue (Q2_1). In addition, the runtime handle (RH) can check whether a progress signal (Run) has been received in the register (Rs). At the first time (t1), the progress signal (Run) has not yet been received. Therefore, the runtime handle (RH) continues to maintain the wait state of the 1_2 task descriptor (Tsk_d12). As the 1_2 task descriptor (Tsk_d12) waits, the subsequent 1_3 task descriptor (Tsk_d13) also waits in the 2_1 queue (Q2_1).

At the second time (t2), a run signal (Run) is received, and the state of the register (Rs) may change according to the reception of the run signal (Run). In the embodiment, the state of the register (Rs) that has received the run signal (Run) is defined as an active state, and the state of the register (Rs) that has not received the run signal (Run) is defined as a basic state. For example, the run signal (Run) may be a 1-bit signal, and may be changed from the basic state (0) of the register (Rs) to the active state (1) in response to the run signal (Run). The runtime handle (RH) may release the waiting state of the 1_2 task descriptor (Tsk_d12) in response to the run signal (Run) and convert it to a running state. The runtime handle (RH) can convert the 1_2 task descriptor (Tsk_d12) into the 1_2 task information (Tsk_d12') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Dunn passage (630).

At the third time (t3), as the progress signal (Run) is utilized to release the waiting state of the 1_2 task descriptor (Tsk_d12), it can be seen that the state of the register (Rs) is restored from the active state to the basic state. The 1_3 task descriptor (Tsk_d13) corresponds to the check-in timing. The runtime handle (RH) can convert the 1_3 task descriptor (Tsk_d13) that does not include the waiting field into the 1_3 task information (Tsk_d13') and transmit it to the core global (500), and the corresponding check-in data (ChI) can be transmitted to the dunned passage (630).

Fig. 20 is an exemplary diagram for explaining a process of processing a task descriptor included in a second_1 queue (Q2_1) in response to a pre-received progress signal. Here, time (t0 to t3) is an exemplary time unit divided according to the processing status of each task descriptor, and is not limited to a specific time. In the second_1 queue (Q2_1), the first_2 task descriptor (Tsk_d12) includes a waiting field, but the remaining task descriptors do not include waiting fields.

Referring to Fig. 20, the process of processing task descriptors (Tsk_d11 to Tsk_d14) included in the second_1 queue (Q2_1) according to the change in time (t0 -> t3) can be confirmed.

At time 0 (t0), the 1_1 task descriptor (Tsk_d11) corresponds to the check-in timing. That is, the 1_1 task descriptor (Tsk_d11) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_1 task descriptor (Tsk_d11) that does not include a wait field as a progress state, and can convert the 1_1 task descriptor (Tsk_d11) into 1_1 task information (Tsk_d11') and transfer it to the core global (500), and the corresponding check-in data (ChI) can be transferred to the dump passage (630).

In addition, at time 0 (t0), it can be confirmed that the register (Rs) is active in response to the receipt of the progress signal (Run). The 1_1 task descriptor (Tsk_d11) that does not include a wait field proceeds with the check-in process regardless of the state of the register (Rs).

At the first time (t1), the 1_2 task descriptor (Tsk_d12) corresponds to the check-in timing. That is, the 1_2 task descriptor (Tsk_d12) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_2 task descriptor (Tsk_d12) including the wait field as a wait state, and can put the 1_2 task descriptor (Tsk_d12) on hold in the 2_1 queue (Q2_1). However, the runtime handle (RH) can release the wait state of the 1_2 task descriptor (Tsk_d12) through a previously received progress signal (Run). The runtime handle (RH) can convert the 1_2 task descriptor (Tsk_d12) into the 1_2 task information (Tsk_d12') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Dunn passage (630).

At the second time (t2), the 1_3 task descriptor (Tsk_d13) corresponds to the check-in timing. The runtime handle (RH) can convert the 1_3 task descriptor (Tsk_d13) that does not include a wait field into the 1_3 task information (Tsk_d13') and transfer it to the core global (500), and the corresponding check-in data (ChI) can be transferred to the dunned passage (630). In addition, at the second time (t2), the progress signal (Run) may be received in advance, and the register (Rs) may be converted to an active state. This active state of the register (Rs) may be maintained for several times, and the check-in process may be performed regardless of the state of the register (Rs) in the task descriptor that does not include a wait field.

At the third time (t3), it can be seen that the register (Rs) remains active. The 1_4th task descriptor (Tsk_d14) corresponds to the check-in timing. The runtime handle (RH) can convert the 1_4th task descriptor (Tsk_d14) that does not include a wait field into the 1_4th task information (Tsk_d14') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the dunned passage (630).

In some embodiments, the register (Rs) may be configured to receive multiple progress signals (Run). In an embodiment, the runtime handle (RH) may be processed so that each progress signal (Run) received in the multiple registers (Rs) individually corresponds to a task descriptor that is in a waiting state.

Fig. 21 is an exemplary diagram for explaining a process of processing a task descriptor included in a second_1 queue (Q2_1) in response to multiple progress signals. Here, time (t0 to t3) is an exemplary time unit divided according to the processing status of each task descriptor, and is not limited to a specific time. In the second_1 queue (Q2_1), the first_2 task descriptor (Tsk_d12) and the first_4 task descriptor (Tsk_d14) include waiting fields, but the remaining task descriptors do not include waiting fields.

Referring to Fig. 21, at time 0 (t0), the 1_1 task descriptor (Tsk_d11) corresponds to the check-in timing. That is, the 1_1 task descriptor (Tsk_d11) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_1 task descriptor (Tsk_d11) that does not include a wait field as a progress state, and can convert the 1_1 task descriptor (Tsk_d11) into 1_1 task information (Tsk_d11') and transfer it to the core global (500), and the corresponding check-in data (ChI) can be transferred to the dump passage (630).

In addition, at time 0 (t0), multiple progress signals (Run) can be received. For example, two progress signals (Run) can be received in the first register (Rs1) and the second register (Rs2), respectively. It can be confirmed that the first register (Rs1) and the second register (Rs2) are active by the received progress signals (Run). The 1_1 task descriptor (Tsk_d11) that does not include a waiting field causes the check-in process to proceed regardless of the states of the first and second registers (Rs1, Rs2).

At the first time (t1), the 1_2 task descriptor (Tsk_d12) corresponds to the check-in timing. That is, the 1_2 task descriptor (Tsk_d12) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_2 task descriptor (Tsk_d12) including the wait field as a wait state, and can put the 1_2 task descriptor (Tsk_d12) on hold in the 2_1 queue (Q2_1). However, the runtime handle (RH) can release the wait state of the 1_2 task descriptor (Tsk_d12) through the progress signal (Run) of the first register (Rs1) that has been received in advance. The runtime handle (RH) can convert the 1_2 task descriptor (Tsk_d12) into the 1_2 task information (Tsk_d12') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Dunn passage (630).

At the second time (t2), the 1_3 task descriptor (Tsk_d13) corresponds to the check-in timing. The runtime handle (RH) can convert the 1_3 task descriptor (Tsk_d13) that does not include a wait field into the 1_3 task information (Tsk_d13') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Don passage (630).

At the third time (t3), the 1_4th task descriptor (Tsk_d14) corresponds to the check-in timing. That is, the 1_4th task descriptor (Tsk_d14) corresponds to the most preceding task descriptor in the 2_1st queue (Q2_1). The runtime handle (RH) can determine the 1_4th task descriptor (Tsk_d14) including the wait field as a wait state, and can put the 1_4th task descriptor (Tsk_d14) on hold in the 2_1st queue (Q2_1). However, the runtime handle (RH) can release the wait state of the 1_4th task descriptor (Tsk_d14) through the progress signal (Run) of the 2nd register (Rs2) that was received in advance. The runtime handle (RH) can convert the 1_4th task descriptor (Tsk_d14) into the 1_4th task information (Tsk_d14') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Don passage (630).

In some embodiments, the runtime handle (RH) may include a counter (Rc) capable of receiving and processing multiple progress signals (Run). The runtime handle (RH) may individually utilize multiple progress signals (Run) received by the counter (Rc) to release the waiting state of the task descriptor.

Fig. 22 is an exemplary diagram for explaining a process of processing a task descriptor included in a second_1 queue (Q2_1) through a plurality of progress signals received by a counter. Here, time (t0 to t3) is an exemplary time unit divided according to the processing status of each task descriptor, and is not limited to a specific time. In the second_1 queue (Q2_1), the first_2 task descriptor (Tsk_d12) and the first_4 task descriptor (Tsk_d14) include waiting fields, but the remaining task descriptors do not include waiting fields.

Referring to Fig. 21, at time 0 (t0), the 1_1 task descriptor (Tsk_d11) corresponds to the check-in timing. That is, the 1_1 task descriptor (Tsk_d11) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_1 task descriptor (Tsk_d11) that does not include a wait field as a progress state, and can convert the 1_1 task descriptor (Tsk_d11) into 1_1 task information (Tsk_d11') and transfer it to the core global (500), and the corresponding check-in data (ChI) can be transferred to the dump passage (630).

In addition, at time 0 (t0), a progress signal (Run) can be received by the counter (Rc). It can be known that one progress signal (Run) has been received by the received progress signal (Run) by the counter (Rc). The 1_1 task descriptor (Tsk_d11) that does not include a waiting field causes the check-in process to proceed regardless of the state of the counter (Rc).

At the first time (t1), one progress signal (Run) can be received by the counter (Rc), and it can be seen that the state of the counter (Rc) has changed from 1 to 2.

At the first time (t1), the 1_2 task descriptor (Tsk_d12) corresponds to the check-in timing. That is, the 1_2 task descriptor (Tsk_d12) corresponds to the most preceding task descriptor in the 2_1 queue (Q2_1). The runtime handle (RH) can determine the 1_2 task descriptor (Tsk_d12) including the wait field as a wait state, and can put the 1_2 task descriptor (Tsk_d12) on hold in the 2_1 queue (Q2_1). However, the runtime handle (RH) can release the wait state of the 1_2 task descriptor (Tsk_d12) through the progress signal (Run) of the counter (Rc) that has been received in advance. The runtime handle (RH) can convert the 1_2 task descriptor (Tsk_d12) into the 1_2 task information (Tsk_d12') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Dunn passage (630).

At the second time (t2), the 1_3 task descriptor (Tsk_d13) corresponds to the check-in timing. The runtime handle (RH) can convert the 1_3 task descriptor (Tsk_d13) that does not include a wait field into the 1_3 task information (Tsk_d13') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Don passage (630).

At the third time (t3), the 1_4th task descriptor (Tsk_d14) corresponds to the check-in timing. That is, the 1_4th task descriptor (Tsk_d14) corresponds to the most preceding task descriptor in the 2_1st queue (Q2_1). The runtime handle (RH) can determine the 1_4th task descriptor (Tsk_d14) including the wait field as a wait state, and can put the 1_4th task descriptor (Tsk_d14) on hold in the 2_1st queue (Q2_1). However, the runtime handle (RH) can release the wait state of the 1_4th task descriptor (Tsk_d14) through the progress signal (Run) of the counter (Rc) that has been received in advance. The runtime handle (RH) can convert the 1_4th task descriptor (Tsk_d14) into the 1_4th task information (Tsk_d14') and pass it to the core global (500), and the corresponding check-in data (ChI) can be passed to the Don passage (630).

Additionally, the runtime handle (RH) can perform not only the Wait-Run control of individual task descriptors, but also the Pause-Resume control, which temporarily pauses and resumes the operation of at least one second queue.

In some embodiments, the runtime handle (RH) may temporarily stop checking in task descriptors from at least one secondary queue if an abnormality or overload is expected in the operation of the neural core (100).

In some embodiments, the runtime handle (RH) may perform stop-restart control based on a danger signal provided in the Dawn passage (630). Here, the danger signal may be an event according to at least one of a hardware or software error, a log-related event, and a task performed without a descriptor, and such an event may be provided in an event monitor (EM) of the Dawn passage (630) described below, but the embodiments of the present invention are not limited thereto.

Additionally, the runtime handle (RH) can perform stop-restart control in response to an overload of the task queue (622) or an overload of the report queue of the reporting manager (RM) described below.

Depending on the pause control of the runtime handle (RH), the task information corresponding to the task descriptor may be temporarily suspended from being transmitted to the core global (500) and processed. When the overload is resolved, the runtime handle (RH) may release the pause state and restart the processing of the waiting task descriptor.

Figure 23 is a block diagram for explaining the Dunn passage of Figure 15 in detail.

Referring to FIG. 23, the Dawn Passage (630) may include a check-in buffer (Cib), a dependency setter (DPs), a timeout monitor (ToM), an event monitor (EM), and a report managing module (631).

The check-in buffer (Cib) can receive check-in data (ChI). The check-in buffer (Cib) can include first to n-th check-in buffers (Cib_1 to Cib_n). The check-in buffer (Cib) can store first to n-th task descriptors (Tsk_d1 to Tsk_dn) stored in the check-in data (ChI). The first to n-th check-in buffers (Cib_1 to Cib_n) can store first to n-th task descriptors (Tsk_d1 to Tsk_dn), respectively. The check-in buffer (Cib) can perform check-in of the first to n-th task descriptors (Tsk_d1 to Tsk_dn) through this.

That is, the first check-in buffer (Cib_1) can store the first task descriptor (Tsk_d1), and the second check-in buffer (Cib_2) can store the second task descriptor (Tsk_d2). The n-th check-in buffer (Cib_n) can store the n-th task descriptor (Tsk_dn). The number of check-in buffers (Cib) can be equal to the number of the first queue (Q1) and the number of the second queue (Q2).

The check-in buffer (Cib) can receive a completion signal from the core global (500). At this time, the completion signal can include the first to nth completion signals (Tsk_d1d to Tsk_dnd). The first to nth completion signals (Tsk_d1d to Tsk_dnd) can be completion signals for the first to nth task descriptors (Tsk_d1 to Tsk_dn), respectively. The first to nth completion signals (Tsk_d1d to Tsk_dnd) can be received by the first to nth check-in buffers (Cib_1 to Cib_n), respectively. That is, the first check-in buffer (Cib_1) can receive the first completion signal (Tsk_d1d), and the second check-in buffer (Cib_2) can receive the second completion signal (Tsk_d2d). The nth check-in buffer (Cib_n) can receive the nth completion signal (Tsk_dnd).

Dependency setters (DPs) can generate a dependency update request (DFURQ) by receiving a completion signal from a check-in buffer (Cib). That is, depending on which task corresponding to which task descriptor is completed, the dependency setters (DPs) can generate a dependency update request (DFURQ). The dependency setters (DPs) can transmit the dependency update request (DFURQ) to the task passage (620).

Dependency setters (DPs) can check out each of the first to nth task descriptors (Tsk_d1 to Tsk_dn) according to the completion signal. Accordingly, the dependency setters (DPs) can generate a checkout report (COrp) regarding which tasks have been completed and checked out. The dependency setters (DPs) can transmit the checkout report (COrp) to the reporting managing module (631).

That is, check-in can be defined as a procedure that is registered before a task descriptor is processed, and check-out can be defined as a procedure that is deregistered when the task descriptor is fully processed.

As the dependency setter (DPs) sends a dependency update request (DFURQ) to the task passage (620), the dependency checker (DPc) of the task passage (620) can sequentially send the task descriptor according to the dependencies of the task descriptor.

In this embodiment, the overhead of communication with the command processor (7000) can be minimized by allowing the task manager (600) to directly perform dependency checking and setting, rather than having the command processor (7000) exclusively perform processing according to dependencies. Accordingly, the performance and speed of the neural processing device (1) according to this embodiment can be dramatically improved.

The timeout monitor (ToM) can receive a timeout detection signal (TOdec) from the check-in buffer (Cib). The timeout detection signal (TOdec) can be a signal whether the time from the check-in time to the check-out time exceeds a preset threshold time. Here, the check-out time can mean the time when the execution of the corresponding task is completed. The check-in buffer (Cib) can monitor the execution time of the task corresponding to the checked-in task descriptor. The check-in buffer (Cib) can compare the execution time of the task with the threshold time to determine whether to generate the timeout detection signal (TOdec). The check-in buffer (Cib) can generate the timeout detection signal (TOdec) if the execution time calculated from the check-in time exceeds the threshold time. In other words, if the execution of the task is not completed by the threshold time, the timeout detection signal (TOdec) is generated. The first to nth check-in buffers (Cib_1 to Cib_n) check whether each checked-in task descriptor is performed, and if the performance time exceeds a threshold time, a timeout detection signal (TOdec) is generated.

In an embodiment, the threshold time may be set individually for each task. In some embodiments, the threshold time may be set differently for each type of task. A task corresponding to a memory operation may be set to have a shorter threshold time than a task corresponding to a computation. However, the embodiments of the present invention are not limited thereto.

In addition, in the embodiment, whether or not to generate a timeout report may be individually set depending on the task. That is, the command processor (7000) may be set not to receive a timeout report for at least some of the tasks transmitted to the task manager (600). Since timeout monitoring may be set not to be performed for all tasks, the burden on the task manager (600) for timeout monitoring may be reduced. For example, a timeout report may be generated for a task corresponding to a computation, but a timeout report may not be generated for a task corresponding to a memory operation. However, the embodiment of the present invention is not limited thereto.

The timeout monitor (ToM) can generate a timeout report (TOrp) according to a timeout detection signal (TOdec). The timeout monitor (ToM) can transmit the generated timeout report (TOrp) to the reporting management module (631).

In an embodiment, the event monitor (EM) can detect whether an event has occurred and generate an event report (Erp) based on the event detection signal. The generated event report (Erp) can be provided to the reporting management module (631).

The reporting management module (631) can receive at least one of a transfer done report (TRrp), an event report (Erp), a checkout report (COrp), and a timeout report (TOrp) to generate a completion report (DNrp).

In the embodiment, the checkout report (COrp) corresponds to a report that enables the command processor (7000) to confirm that the task transferred to the neural core (100) has been normally processed and checked out.

The transfer completion report (TRrp) is a report that allows the command processor (7000) to confirm that a task was normally provided as a task passage (620) and a task descriptor was created.

An event report (Erp) is a report that enables the command processor (7000) to confirm that an event has occurred due to at least one of a hardware or software error, a log-related event, and a task performed without a descriptor.

A timeout report (TOrp) is a report that allows the command processor (7000) to determine that processing for a specific task is delayed longer than a set threshold time.

In an embodiment, a checkout report (COrp), a transfer dump report (TRrp), an event report (Erp), and a timeout report (TOrp) can be generated independently. For example, even if a timeout report (TOrp) is generated due to a delay in the execution of a task associated with a specific task descriptor, the checkout report (COrp) can be generated independently from the generation of the timeout report (TOrp) when the execution of the task is completed.

Here, the completion report (DNrp) can be generated based on the checkout report (COrp). Since the completion report (DNrp) includes at least the checkout report, it can transmit whether the task is normally performed to the command processor (7000). Through the generation and transmission of this completion report (DNrp), whether the task is normally performed is confirmed, and the delay in the performance of a specific task can be prevented from becoming prolonged depending on dependencies.

In addition, the completion report (DNrp) can be configured to further include at least one of a transfer done report (TRrp), an event report (Erp), and a timeout report (TOrp), and can comprehensively report to the command processor (7000) whether a task is performed, whether a timeout occurred, whether a transfer is completed, and whether an event occurred.

Figure 24 is a block diagram for explaining in detail the report managing module of Figure 23.

Referring to FIG. 24, the report managing module (631) may include a transfer report queue (TQ), an event report queue (EQ), a checkout report queue (CQ), a timeout report queue (TOQ), and a reporting manager (RM).

Transfer Done Report Queue (TQ) can receive Transfer Done Report (TRrp) and forward it to Reporting Manager (RM). Transfer Done Report (TRrp) can be received sequentially in Transfer Done Report Queue (TQ), and accumulated Transfer Done Report (TRrp) can be forwarded to Reporting Manager (RM) in first-in, first-out (FIFO) order.

The event report queue (EQ) can receive event reports (Erp) and transmit them to a reporting manager (RM). The event report queue (EQ) can receive event reports (Erp) sequentially, and the accumulated event reports (Erp) can be transmitted to a reporting manager (RM) in a first-in, first-out (FIFO) manner. The checkout report queue (CQ) can receive checkout reports (COrp) and transmit them to a reporting manager (RM). The checkout report queue (CQ) can receive checkout reports (COrp) sequentially, and the accumulated checkout reports (COrp) can be transmitted to a reporting manager (RM) in a first-in, first-out (FIFO) manner. In addition, the timeout report queue (TOQ) can receive timeout reports (TOrp) and transmit them to a reporting manager (RM). Timeout reports (TOrp) can be received sequentially in the timeout report queue (TOQ), and the accumulated timeout reports (TOrp) can be delivered to the reporting manager (RM) in a first-in, first-out (FIFO) order.

The reporting manager (RM) can receive at least one of a transfer dump report (TRrp), an event report (Erp), a checkout report (COrp), and a timeout report (TOrp), and generate a completion report (DNrp) through the same. The reporting manager (RM) can transmit the completion report (DNrp) to the command processor (7000).

In addition, the reporting manager (RM) can monitor the status of at least one of the event report queue (EQ), the checkout report queue (CQ), and the timeout report queue (TOQ). The reporting manager (RM) can monitor at least one of the event report acceptance status of the event report queue (EQ), the checkout report acceptance status of the checkout report queue (CQ), and the timeout report acceptance status of the timeout report queue (TOQ). If at least one of the event report queue (EQ), the checkout report queue (CQ), and the timeout report queue (TOQ) is confirmed to be saturated, the reporting manager (RM) can pause the operation of the task passage (620) through the runtime handle (RH).

Figure 25 is a diagram explaining data exchanged between the core global and neural core of Figure 15.

Referring to FIG. 25, the core global (500) can receive a table update request (TURQ) and transmit it to the LSU (110). In addition, the core global (500) can receive task information (Tsk_d') and transmit it to the neural core (100).

The neural core (100) can perform a task and generate a completion signal (Tsk_dd). The LSU (110) or the processing unit (160) can transmit the completion signal (Tsk_dd) to the core global (500). The core global (500) can include a signal scheduler (sgn_sch). The signal scheduler (sgn_sch) can receive the completion signal, schedule the transmission of the completion signal, and transmit it to the Done passage (630).

Figure 26 is a diagram for explaining the types of task descriptors stored in the first queue, the second queue, and the check-in buffer.

Referring to FIG. 26, the first to fourth queues (Q1_1 to Q1_4) of the first queue (Q1), the second to fourth queues (Q2_1 to Q2_4) of the second queue (Q2), and the first to fourth check-in buffers (Cib_1 to Cib_4) of the check-in buffer (Cib) can each store a specific type of task descriptor. The first to fourth queues (Q1_1 to Q1_4), the second to fourth queues (Q2_1 to Q2_4), and the first to fourth check-in buffers (Cib_1 to Cib_4) can each store different types of task descriptors.

For example, the first_1st queue (Q1_1), the second_1st queue (Q2_1), and the first check-in buffer (Cib_1) can store task descriptors for computation, and the first_2nd queue (Q1_2), the second_2nd queue (Q2_2), and the second check-in buffer (Cib_2) can store task descriptors for micro DMA. In addition, the first_3rd queue (Q1_3), the second_3rd queue (Q2_3), and the third check-in buffer (Cib_3) can store task descriptors for LP micro DMA, and the first_4th queue (Q1_4), the second_4th queue (Q2_4), and the fourth check-in buffer (Cib_4) can store task descriptors for ST micro DMA. However, the present embodiment is not limited thereto.

FIG. 27 is a diagram illustrating a first queue, a second queue, and a check-in buffer of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 27, the first to fourth queues (Q1_1 to Q1_4), the second to fourth queues (Q2_1 to Q2_4), and the first to fourth check-in buffers (Cib_1 to Cib_4) can each store a specific type of task descriptor. The first to fourth queues (Q1_1 to Q1_4), the second to fourth queues (Q2_1 to Q2_4), and the first to fourth check-in buffers (Cib_1 to Cib_4) can store the same type of task descriptor.

For example, the first_1 queue (Q1_1), the second_1 queue (Q2_1), and the first check-in buffer (Cib_1) can store task descriptors for the first computation, and the first_2 queue (Q1_2), the second_2 queue (Q2_2), and the second check-in buffer (Cib_2) can store task descriptors for the second computation. In addition, the first_3 queue (Q1_3), the second_3 queue (Q2_3), and the third check-in buffer (Cib_3) can store task descriptors for the third computation, and the first_4 queue (Q1_4), the second_4 queue (Q2_4), and the fourth check-in buffer (Cib_4) can store task descriptors for the fourth computation. However, the present embodiment is not limited thereto.

At this time, the first to fourth computations may be completely identical computations, or they may be computations of the same type but of different types in detail.

FIG. 28 is a diagram illustrating a first queue, a second queue, and a check-in buffer of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 28, the first to fourth queues (Q1_1 to Q1_4), the second to fourth queues (Q2_1 to Q2_4), and the first to fourth check-in buffers (Cib_1 to Cib_4) can each store multiple types of task descriptors. The first to fourth queues (Q1_1 to Q1_4), the second to fourth queues (Q2_1 to Q2_4), and the first to fourth check-in buffers (Cib_1 to Cib_4) can store different types of task descriptors, or can store the same type of task descriptors.

FIG. 29 is a block diagram for explaining in detail the structure of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 29, the neural core (101) may have a CGRA structure, unlike the neural core (100). The neural core (101) may include an instruction memory (111_1), a CGRA L0 memory (111_2), a PE array (111_3), and an LSU (Load/Store Unit) (111_4).

The instruction memory (111_1) can receive and store instructions. The instruction memory (111_1) can sequentially store instructions internally and provide the stored instructions to the PE array (111_3). At this time, the instructions can instruct the operation of the processing elements (111_3a) included in each PE array (111_3).

The CGRA L0 memory (111_2) is a memory located inside the neural core (101), and can receive all input data required for the neural core (101) to work from the outside and temporarily store them. In addition, the CGRA L0 memory (111_2) can temporarily store output data calculated by the neural core (101) in order to transmit it to the outside. The CGRA L0 memory (111_2) can serve as a cache memory of the neural core (101).

The CGRA L0 memory (111_2) can transmit and receive data with the PE array (111_3). The CGRA L0 memory (111_2) may be a memory corresponding to L0 (level 0) lower than L1. At this time, the L0 memory may be a private memory of a neural core (101) that is not shared. The CGRA L0 memory (111_2) may transmit data such as activation or weight and programs to the PE array (111_3).

The PE array (111_3) may be a module that performs operations. The PE array (111_3) may perform not only one-dimensional operations but also two-dimensional or more matrix/tensor operations. The PE array (111_3) may include a plurality of processing elements (111_3a) and a specific processing element (111_3b) therein.

The processing elements (111_3a) and the specific processing elements (111_3b) can be arranged in rows and columns. The processing elements (111_3a) and the specific processing elements (111_3b) can be arranged in m columns. In addition, the processing elements (111_3a) can be arranged in n rows, and the specific processing elements (111_3b) can be arranged in l rows. Accordingly, the processing elements (111_3a) and the specific processing elements (111_3b) can be arranged in (n+l) rows and m columns.

The LSU (111_4) can receive at least one of data, a control signal, and a synchronization signal from the outside through the local interconnection (200). The LSU (111_4) can transmit at least one of the data, the control signal, and the synchronization signal received to the CGRA L0 memory (111_2). Similarly, the LSU (111_4) can transmit at least one of the data, the control signal, and the synchronization signal to the outside through the local interconnection (200).

The neural core (101) may have a Coarse Grained Reconfigurable Architecture (CGRA) structure. Accordingly, each processing element (111_3a) and a specific processing element (111_3b) of the PE array (111_3) of the neural core (101) may be connected to at least one of the CGRA L0 memory (111_2), the instruction memory (111_1), and the LSU (111_4). That is, the processing element (111_3a) and the specific processing element (111_3b) do not have to be connected to all of the CGRA L0 memory (111_2), the instruction memory (111_1), and the LSU (111_4), but may be connected to some of them.

In addition, the processing element (111_3a) and the specific processing element (111_3b) may be different types of processing elements. Accordingly, among the CGRA L0 memory (111_2), the instruction memory (111_1), and the LSU (111_4), the element connected to the processing element (111_3a) and the element connected to the specific processing element (111_3b) may be different from each other.

The neural core (101) of the present invention having a CGRA structure can perform a high level of parallel operation, and can have low power consumption because direct data exchange between the processing element (111_3a) and a specific processing element (111_3b) is possible. In addition, optimization according to various operation tasks can also be possible by including two or more types of processing elements (111_3a).

For example, if a processing element (111_3a) is a processing element that performs a two-dimensional operation, a specific processing element (111_3b) may be a processing element that performs a one-dimensional operation. However, the present embodiment is not limited thereto.

FIG. 30 is a diagram for explaining the hierarchical structure of a command processor and a task manager of a neural processing device according to some embodiments of the present invention, and FIG. 31 is a diagram for explaining the hierarchical structure of a command processor and a task manager of a neural processing device according to some embodiments of the present invention.

Referring to FIGS. 30 and 31, when the number of task managers (600) increases, it may be difficult for the command processor (7000) to manage all of the task managers (600). Accordingly, the neural processing device (1) according to some embodiments of the present invention may have a hierarchy structure in which a master task manager (600M) manages a plurality of task managers (600) and a command processor (7000) manages the master task manager (600M).

Also, referring to FIG. 31, the lower levels of the master task manager (600M) may be subdivided in various ways. For example, the first sub-task manager (600s1) and the second sub-task manager (600s2) may each form a hierarchy. That is, one first sub-task manager (600s1) may manage at least one second sub-task manager (600s2), and one master task manager (600M) may manage at least one first sub-task manager (600s1). Additionally, several hierarchies may be added below the second sub-task manager (600s2).

That is, although three levels of a task manager (600), a master task manager (600M), and a command processor (7000) are illustrated in FIGS. 30 and 31, the number of levels may be four or more. That is, the depth of the hierarchy structure may vary depending on the number of task managers (600).

FIG. 32 is a block diagram illustrating memory reconfiguration of a neural processing system according to some embodiments of the present invention.

Referring to FIG. 32, the neural core SoC (10) may include first to eighth processing units (160a to 160h) and an on-chip memory (OCM). Eight processing units are illustrated as an example, but this is only an example and the number of processing units may vary.

On-chip memory (OCM) may include first to eighth L0 memories (120a to 120h) and shared memory (2000).

The first to eighth L0 memories (120a to 120h) can be used as dedicated memories of the first to eighth processing units (160a to 160h), respectively. That is, the first to eighth processing units (160a to 160h) and the first to eighth L0 memories (120a to 120h) can correspond to each other on a 1:1 basis.

The shared memory (2000) may include first to eighth memory units (2100a to 2100h). The first to eighth memory units (2100a to 2100h) may correspond to the first to eighth processing units (160a to 160h) and the first to eighth L0 memories (120a to 120h), respectively. That is, the number of memory units may be eight, which is the same as the number of processing units and L0 memories.

The shared memory (2000) can operate in either of two types of on-chip memory formats. That is, the shared memory (2000) can operate in either the L0 memory format or the global memory format. That is, the shared memory (2000) can implement two logical memories with one hardware.

When the shared memory (2000) is implemented in the L0 memory format, the shared memory (2000) can operate as a private memory of each of the first to eighth processing units (160a to 160h), such as the first to eighth L0 memories (120a to 120h). The L0 memory can operate at a relatively high-speed clock compared to the global memory, and the shared memory (2000) can also use a relatively faster clock when operating in the L0 memory format.

When the shared memory (2000) is implemented in a global memory format, the shared memory (2000) can operate as a common memory that is used by the first processing unit (100a) and the second processing unit (100b). At this time, the shared memory (2000) can be shared not only by the first to eighth processing units (160a to 160h) but also by the first to eighth L0 memories (120a to 120h).

Global memory can generally use a lower clock than L0 memory, but is not limited thereto. When the shared memory (2000) operates in a global memory format, the first to eighth processing units (160a to 160h) can share the shared memory (2000). At this time, the shared memory (2000) is connected to the volatile memory (32) of FIG. 2 through a global interconnection (6000), and can also operate as a buffer of the volatile memory (32).

The shared memory (2000) may operate at least in part in the L0 memory format and the remainder in the global memory format. That is, the entire shared memory (2000) may operate in the L0 memory format, or the entire shared memory (2000) may operate in the global memory format. Alternatively, a part of the shared memory (2000) may operate in the L0 memory format and the remainder in the global memory format.

FIG. 33 is a block diagram illustrating an example of memory reconfiguration of a neural processing system according to some embodiments of the present invention.

Referring to FIGS. 32 and 33, the first, third, fifth and seventh dedicated areas (AE1, AE3, AE5, AE7) of the first, third, fifth and seventh processing units (100a, 100c, 100e and 100g) may include only the first, third, fifth and seventh L0 memories (120a, 120c, 120e and 120g), respectively. In addition, the second, fourth, sixth and eighth dedicated areas (AE2, AE4, AE6, AE8) of the second, fourth, sixth and eighth processing units (100b, 100d, 100f and 100h), respectively, may include only the second, fourth, sixth and eighth L0 memories (120b, 120d, 120f and 120h), respectively. Additionally, the second, fourth, sixth and eighth dedicated areas (AE2, AE4, AE6, AE8) may include the second, fourth, sixth and eighth memory units (2100b, 2100d, 2100f, 2100h). The first, third, fifth and seventh memory units (2100a, 2100c, 2100e, 2100g) of the shared memory (2000) may be utilized as a common area (AC).

The common area (AC) may be a memory shared by the first to eighth processing units (160a to 160h). The second dedicated area (AE2) may include the second L0 memory (120b) and the second memory unit (2100b). The second dedicated area (AE2) may be an area in which the second L0 memory (120b) and the second memory unit (210b), which are hardware-separated, operate in the same manner and logically operate as one L0 memory. The fourth, sixth, and eighth dedicated areas (AE4, AE6, AE8) may also operate in the same manner as the second dedicated area (AE2).

The shared memory (2000) according to the present embodiment can be used by converting the area corresponding to each neural core into a logical L0 memory and a logical global memory with an optimized ratio. The shared memory (2000) can perform this ratio adjustment at run time.

That is, each neural core may perform the same task, but in some cases, it may perform different tasks. In this case, the capacity of L0 memory and the capacity of global memory required for the task performed by each neural core cannot but be different each time. Accordingly, if the configuration ratio of L0 memory and shared memory is fixed, like in the existing on-chip memory, inefficiency may occur depending on the computational task assigned to each neural core.

Therefore, the shared memory (2000) of the neural processing device according to the present embodiment can set the optimal ratio of L0 memory and global memory according to the computational task during runtime, and can improve the efficiency and speed of the computation.

Figure 34 is an enlarged block diagram of part A of Figure 32.

Referring to FIG. 32 and FIG. 34, the shared memory (2000) may include a first L0 memory controller (122_1a), a second L0 memory controller (122_1b), a fifth L0 memory controller (122_1e), a sixth L0 memory controller (122_1f), first to eighth memory units (2100a to 2100h), and a global controller (2200). Other L0 memory controllers not shown may also be included in the present embodiment, but their description is omitted for convenience.

The first L0 memory controller (122_1a) can control the first L0 memory (120a). In addition, the first L0 memory controller (122_1a) can control the first memory unit (2100a). Specifically, when the first memory unit (2100a) is implemented in a logical L0 memory format, control by the first L0 memory controller (122_1a) can be performed on the first memory unit (2100a).

The second L0 memory controller (122_1b) can control the second L0 memory (120b). In addition, the second L0 memory controller (122_1b) can control the second memory unit (2100b). That is, when the second memory unit (2100b) is implemented in a logical L0 memory format, control by the first L0 memory controller (122_1a) can be performed on the second memory unit (2100b).

The fifth L0 memory controller (122_1e) can control the fifth L0 memory (120e). In addition, the fifth L0 memory controller (122_1e) can control the fifth memory unit (2100e). That is, when the fifth memory unit (2100e) is implemented in a logical L0 memory format, control by the fifth L0 memory controller (122_1e) can be performed on the fifth memory unit (2100e).

The 6th L0 memory controller (122_1f) can control the 6th L0 memory (120f). In addition, the 6th L0 memory controller (122_1f) can control the 6th memory unit (2100f). That is, when the 6th memory unit (2100f) is implemented in a logical L0 memory format, control by the 6th L0 memory controller (122_1f) can be performed on the 6th memory unit (2100f).

The global controller (2200) can control all of the first to eighth memory units (2100a to 2100h). Specifically, the global controller (2200) can control the first memory unit (2100a) to the eighth memory unit (2100h) when the first to eighth memory units (2100a to 2100h) each logically operate in a global memory format (i.e., do not logically operate in an L0 memory format).

That is, the first to eighth memory units (2100a to 2100h) may be controlled by the first to eighth L0 memory controllers (122_1a to 122_1h) or by the global controller (2200) depending on the type of memory they are logically implemented as.

When the L0 memory controllers including the first, second, fifth, and sixth L0 memory controllers (122_1a, 122_1b, 122_1e, 122_1f) control the first to eighth memory units (2100a to 2100h), respectively, the first to eighth L0 memory controllers (122_1a to 141h) control the first to eighth memory units (2100a to 2100h) in the same manner as the first to eighth L0 memories (120a to 120h), and therefore, they can be controlled as dedicated memories of the first to eighth processing units (160a to 160h). Accordingly, the first to eighth memory units (2100a to 2100h) can operate at a clock frequency corresponding to the clock frequency of the first to eighth processing units (160a to 160h).

The L0 memory controllers including the first L0 memory controller (122_1a), the second L0 memory controller (122_1b), the fifth L0 memory controller (122_1e), and the sixth L0 memory controller (122_1f) may each include the LSU (110) of FIG. 8.

When the global controller (2200) controls at least one of the first to eighth memory units (2100a to 2100h), the global controller (2200) can control the first to eighth memory units (2100a to 2100h) as global memories of the first to eighth processing units (160a to 160h), respectively. Accordingly, at least one of the first to eighth memory units (2100a to 2100h) can operate at a clock frequency that is independent of the clock frequency of the first to eighth processing units (160a to 160h), respectively. However, the present embodiment is not limited thereto.

The global controller (2200) can connect the first to eighth memory units (2100a to 2100h) to the global interconnection (6000) of FIG. 3. The first to eighth memory units (2100a to 2100h) can exchange data with the off-chip memory (30) of FIG. 1 or exchange data with the first to eighth L0 memories (120a to 120h), respectively, by the global controller (2200).

Each of the first to eighth memory units (2100a to 2100h) may include at least one memory bank. The first memory unit (2100a) may include at least one first memory bank (2110a). The first memory bank (2110a) may be an area in which the first memory unit (2100a) is divided into a specific size. Each of the first memory banks (2110a) may be a memory element having the same size. However, the present embodiment is not limited thereto. In FIG. 29, four memory banks are illustrated as being included in one memory unit.

Similarly, the second, fifth and sixth memory units (2100b, 2100e, 2100f) may each include at least one second, fifth and sixth memory bank (2110b, 2110e, 2110f).

The following description is based on the first memory bank (2110a) and the fifth memory bank (2110e), and may be the same as other memory banks including the second and sixth memory banks (2110b, 2110f).

The first memory bank (2110a) may operate in a logical L0 memory format or in a logical global memory format, respectively. At this time, the first memory bank (2110a) may operate independently from other memory banks within the first memory unit (2100a). However, the present embodiment is not limited thereto.

When each memory bank operates independently, the first memory unit (2100a) may include a first region that operates in the same manner as the first L0 memory (120a) and a second region that operates in a different manner from the first L0 memory (120a). In this case, the first region and the second region do not necessarily coexist, and one region may occupy the entire first memory unit (2100a).

Similarly, the second memory unit (2100b) may include a third region that operates in the same manner as the second L0 memory (120b) and a fourth region that operates in a different manner from the second L0 memory (120b). In this case, the third region and the fourth region do not necessarily coexist, and either region may occupy the entire first memory unit (2100a).

At this time, the ratio of the first region and the second region may be different from the ratio of the third region and the fourth region. However, the present embodiment is not limited thereto. Accordingly, the ratio of the first region and the second region may be the same as the ratio of the third region and the fourth region. In other words, the memory configuration ratio in each memory unit may vary as much as desired.

In general, in the case of existing systems on chips, the on-chip memory, except for the high-speed L0 memory, was often configured with high-density, low-power SRAM. This is because SRAM has high efficiency in terms of chip area and power consumption compared to the required capacity. However, in the case of tasks that require more data quickly than the capacity of the predetermined L0 memory, the processing speed of the existing on-chip memory was bound to be significantly slow, and even in cases where the need for global memory was not great, there was no way to utilize the remaining global memory, resulting in inefficiency.

In contrast, the shared memory (2000) according to some embodiments of the present invention may be selectively controlled by one of the two controllers, depending on the case. In this case, the shared memory (2000) is not controlled entirely by only one of the two controllers, but may be independently controlled by memory unit or memory bank.

Through this, the shared memory (2000) according to the present embodiment can obtain the optimal memory configuration ratio according to the computational task during runtime, thereby performing faster and more efficient computational tasks. In the case of a processing unit specialized in artificial intelligence, the required sizes of L0 memory and global memory may vary depending on the specific application. Furthermore, even for the same application, the required sizes of L0 memory and global memory may vary depending on the layer when using a deep learning network. The shared memory (2000) according to the present embodiment can change the memory configuration ratio during runtime even when the computational step according to each layer changes, thereby enabling faster and more efficient deep learning tasks.

Fig. 35 is a drawing for explaining the first memory bank of Fig. 34 in detail. Fig. 35 illustrates the first memory bank (2110a), but other memory banks may also have the same structure as the first memory bank (2110a).

Referring to FIG. 35, the first memory bank (2110a) may include a cell array (Ca), a bank controller (Bc), a first path unit (P1), and a second path unit (P2).

The cell array (Ca) may include a plurality of memory elements (Cells) therein. The cell array (Ca) may have a plurality of memory elements arranged in a lattice structure. The cell array (Ca) may be, for example, a SRAM (Static Random Access Memory) cell array.

A bank controller (Bc) can control a cell array (Ca). The bank controller (Bc) can determine whether the cell array (Ca) will operate in an L0 memory format or a global memory format and control the cell array (Ca) accordingly.

Specifically, the bank controller (Bc) can determine whether to transmit/receive data in the direction of the first path unit (P1) or the direction of the second path unit (P2) during runtime. The bank controller (Bc) can determine the direction of transmitting/receiving data according to the path control signal (Spc).

Path control signals (Spc) can be generated by a pre-designed device driver or a compiler. Path control signals (Spc) can be generated according to the characteristics of the computational task. Alternatively, path control signals (Spc) can be generated by input received from a user. That is, a user can directly input path control signals (Spc) to select the most optimal memory configuration ratio.

The bank controller (Bc) can determine the transmission/reception path of data stored in the cell array (Ca) through the path control signal (Spc). Depending on how the bank controller (Bc) determines the transmission/reception path of data, the exchange interface of data can vary. That is, when the bank controller (Bc) exchanges data with the first path unit (P1), the first interface can be used, and when it exchanges data with the second path unit (P2), the second interface can be used. In this case, the first interface and the second interface can be different from each other.

Additionally, the address system in which data is stored may also vary. That is, when a specific interface is selected, read and write operations may be performed with the corresponding address system.

The bank controller (Bc) can operate at a specific clock frequency. For example, if the cell array (Ca) is an SRAM cell array, the bank controller (Bc) can operate at a general SRAM operating clock frequency.

The first path unit (P1) may be connected to the bank controller (Bc). The first path unit (P1) may directly exchange data of the cell array (Ca) with the first processing unit (100a). At this time, “directly” may mean that data is exchanged with each other without going through the global interconnection (6000). That is, the first processing unit (100a) may directly exchange data with the first L0 memory (120a), and the first processing unit (100a) may exchange data through the first path unit (P1) when the shared memory (2000) is logically implemented in the L0 memory format. The first path unit (P1) may include an L0 memory controller including the first L0 memory controller (122_1a) and the second L0 memory controller (122_1b) of FIG. 32.

The first path unit (P1) can configure a multi-cycle sync-path. That is, the operating clock frequency of the first path unit (P1) can be the same as the operating clock frequency of the first processing unit (100a). The first L0 memory (120a) can quickly exchange data with the same clock frequency as the operating clock frequency of the first processing unit (100a) in order to quickly exchange data at the same speed as the operation of the first processing unit (100a). The first path unit (P1) can also operate with the same clock frequency as the operating clock frequency of the first processing unit (100a).

At this time, the operating clock frequency of the first path unit (P1) may be a multiple of the operating clock frequency of the bank controller (Bc). In this case, a separate CDC (Clock Domain Crossing) operation for clock synchronization between the bank controller (Bc) and the first path unit (P1) is not required, and thus, a delay in data transmission may not occur. Accordingly, faster and more efficient data exchange may be possible.

In Fig. 35, for example, the operating clock frequency of the first path unit (P1) may be 1.5 GHz. This may be twice the frequency of 750 MHz of the bank controller (Bc). However, the present embodiment is not limited thereto, and any case in which the first path unit (P1) operates at an integer multiple of the clock frequency of the bank controller (Bc) may be possible.

The second path unit (P2) can be connected to the bank controller (Bc). The second path unit (P2) can exchange data of the cell array (Ca) with the first processing unit (100a) through the global interconnection (6000) rather than directly. That is, the first processing unit (100a) can exchange data with the cell array (Ca) through the global interconnection (6000) and the second path unit (P2). At this time, the cell array (Ca) can exchange data not only with the first processing unit (100a) but also with other neural cores.

That is, the second path unit (P2) may be a data exchange path between the cell array (Ca) and all neural cores when the first memory bank (2110a) is logically implemented in a global memory format. The second path unit (P2) may include the global controller (2200) of FIG. 23.

The second path unit (P2) can form an Async-Path. The operating clock frequency of the second path unit (P2) can be the same as the operating clock frequency of the global interconnection (6000). The second path unit (P2) can also operate at the same clock frequency as the operating clock frequency of the global interconnection (6000).

At this time, the operating clock frequency of the second path unit (P2) may not be synchronized with the operating clock frequency of the bank controller (Bc). In this case, a CDC (Clock Domain Crossing) operation for clock synchronization may be required between the bank controller (Bc) and the second path unit (P2). If the operating clock frequency of the bank controller (Bc) and the operating clock frequency of the second path unit (P2) are not synchronized with each other, the degree of freedom in designing the clock domain can be increased. Accordingly, the difficulty of hardware design is reduced, and hardware operation can be derived more easily.

The bank controller (Bc) can use different address systems when exchanging data through the first path unit (P1) and when exchanging data through the second path unit (P2). That is, the bank controller (Bc) can use the first address system through the first path unit (P1) and the second address system through the second path unit (P2). In this case, the first address system and the second address system can be different from each other.

The bank controller (Bc) does not necessarily need to exist for each memory bank. That is, since the bank controller (Bc) is not a part for scheduling but rather plays a role in transmitting a signal, it is not an essential part for each memory bank having two ports. Accordingly, one bank controller (Bc) can control multiple memory banks. Multiple memory banks can operate independently even if they are controlled by the bank controller (Bc). However, the present embodiment is not limited thereto.

Of course, a bank controller (Bc) may exist for each memory bank. In this case, the bank controller (Bc) can individually control each memory bank.

Referring to FIGS. 34 and 35, when the first memory unit (210a) exchanges data through the first path unit (P1), the first address system may be used, and when the data is exchanged through the second path unit (P2), the second address system may be used. Similarly, when the second memory unit (210b) exchanges data through the first path unit (P1), the third address system may be used, and when the data is exchanged through the second path unit (P2), the second address system may be used. At this time, the first address system and the third address system may be the same. However, the present embodiment is not limited thereto.

The first address system and the third address system can be used exclusively for the first processing unit (100a) and the second processing unit (100b), respectively. The second address system can be applied commonly to the first processing unit (100a) and the second processing unit (100b).

In Fig. 35, for example, the operating clock frequency of the second path unit (P2) may operate at 1 GHz. This may be a frequency that is not synchronized with the operating clock frequency of 750 MHz of the bank controller (Bc). That is, the operating clock frequency of the second path unit (P2) is not dependent at all on the operating clock frequency of the bank controller (Bc) and may be freely set.

A typical global memory inevitably causes delays due to CDC operations by using slow SRAM (e.g., 750 MHz) and a faster global interconnection (e.g., 1 GHz). In contrast, a shared memory (2000) according to some embodiments of the present invention can avoid delays due to CDC operations by utilizing a first path unit (P1) in addition to a second path unit (P2).

In addition, since a general global memory uses a single global interconnection (6000) for multiple neural cores, a decrease in the overall processing speed can easily occur when data transmission occurs simultaneously. In contrast, the shared memory (2000) according to some embodiments of the present invention has the possibility of using the first path unit (P1) in addition to the second path unit (P2), and thus can obtain the effect of appropriately distributing the data processing amount that is concentrated on the global controller (2200).

FIG. 36 is a block diagram illustrating a software hierarchy structure of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 36, the software layer structure of a neural processing device according to some embodiments of the present invention may include a DL framework (10000), a compiler stack (20000), and a backend module (30000).

DL framework (10000) may refer to a framework for a deep learning model network used by a user. For example, a neural network that has completed learning may be created using a program such as TensorFlow or PyTorch.

The compiler stack (20000) may include an adaptation layer (21000), a compute library (22000), a front-end compiler (23000), a back-end compiler (24000), and a runtime driver (25000).

The adaptation layer (21000) may be a layer that comes into contact with the DL framework (10000). The adaptation layer (21000) may quantize the user's neural network model generated in the DL framework (10000) and modify the graph. In addition, the adaptation layer (21000) may convert the type of the model into a required type.

The front-end compiler (23000) can convert various neural network models and graphs received from the adaptation layer (21000) into a predetermined intermediate representation (IR). The converted IR may be a preset representation that is easy to handle in the back-end compiler (24000) later.

The IR of the front-end compiler (23000) can be optimized in advance at the graph level. In addition, the front-end compiler (23000) can finally generate the IR by converting it into a layout optimized for hardware.

The backend compiler (24000) optimizes the IR converted by the frontend compiler (23000) and converts it into a binary file so that it can be used by the runtime driver. The backend compiler (24000) can generate optimized code by dividing jobs into scales that fit the details of the hardware.

The compute library (22000) can store template operations designed in a form suitable for hardware among various operations. The compute library (22000) provides various template operations that require hardware to the backend compiler (24000) so that optimized code can be generated.

The runtime driver (25000) can continuously perform monitoring during operation to drive a neural network device according to some embodiments of the present invention. Specifically, it can be responsible for executing an interface of the neural network device.

The backend module (30000) may include an ASIC (Application Specific Integrated Circuit) (31000), an FPGA (Field programmable gate array) (32000), and a C-model (33000). The ASIC (31000) may refer to a hardware chip determined according to a predetermined design method. The FPGA (32000) may be a programmable hardware chip. The C-model (33000) may refer to a model implemented by simulating hardware in software.

The backend module (30000) can perform various tasks and produce results using binary code generated through the compiler stack (20000).

FIG. 37 is a conceptual diagram illustrating a deep learning operation performed by a neural processing device according to some embodiments of the present invention.

Referring to FIG. 37, an artificial neural network model (40000) is an example of a machine learning model, and in machine learning technology and cognitive science, is a statistical learning algorithm implemented based on the structure of a biological neural network or a structure that executes the algorithm.

The artificial neural network model (40000) can represent a machine learning model having problem-solving capabilities by learning that nodes, which are artificial neurons that form a network by combining synapses like in a biological neural network, repeatedly adjust the weights of synapses so that the error between the correct output corresponding to a specific input and the inferred output is reduced. For example, the artificial neural network model (40000) can include any probability model, neural network model, etc. used in artificial intelligence learning methods such as machine learning and deep learning.

A neural processing device according to some embodiments of the present invention can perform a calculation by implementing the form of such an artificial neural network model (40000). For example, the artificial neural network model (40000) can receive an input image and output information about at least a part of an object included in the input image.

The artificial neural network model (40000) is implemented as a multilayer perceptron (MLP) composed of nodes in multiple layers and connections between them. The artificial neural network model (40000) according to the present embodiment can be implemented using one of various artificial neural network model structures including MLP. As illustrated in FIG. 36, the artificial neural network model (40000) is composed of an input layer (41000) that receives an input signal or data (40100) from the outside, an output layer (44000) that outputs an output signal or data (40200) corresponding to the input data, and n hidden layers (42000 to 43000) located between the input layer (41000) and the output layer (44000) that receive signals from the input layer (41000), extract characteristics, and transmit them to the output layer (44000) (where, n is a positive integer). Here, the output layer (44000) receives signals from the hidden layers (42000 to 43000) and outputs them to the outside.

The learning methods of the artificial neural network model (40000) include the supervised learning method, which learns to optimize problem solving through the input of teacher signals (correct answers), and the unsupervised learning method, which does not require teacher signals.

The neural processing device can directly generate learning data for learning the artificial neural network model (40000) through simulation. In this way, a plurality of input variables and corresponding plurality of output variables are respectively matched in the input layer (41000) and the output layer (44000) of the artificial neural network model (40000), and synapse values between nodes included in the input layer (41000), hidden layers (42000 to 43000), and the output layer (44000) are adjusted, so that learning can be performed so that a correct output corresponding to a specific input can be extracted. Through this learning process, it is possible to identify characteristics hidden in the input variables of the artificial neural network model (40000), and to adjust the synapse values (or weights) between the nodes of the artificial neural network model (40000) so that the error between the output variables calculated based on the input variables and the target output is reduced.

FIG. 38 is a conceptual diagram illustrating learning and inference operations of a neural network of a neural processing device according to some embodiments of the present invention.

Referring to Fig. 38, in the training phase, a plurality of training data (TD) may be forwarded to the artificial neural network model (NN) and then backwardized again. Through this, the weights and biases of each node of the artificial neural network model (NN) may be adjusted, and through this, learning may be performed to produce increasingly more accurate results. In this way, through the training phase, the artificial neural network model (NN) may be converted into a trained neural network model (NN_T).

In the inference phase, new data (ND) can be input to the retrained neural network model (NN_T). The trained neural network model (NN_T) can derive the result data (RD) through the already trained weights and biases by inputting the new data (ND). For this result data (RD), it may be important which training data (TD) was used for training in the training phase and how much training data (TD) was used.

Hereinafter, with reference to FIGS. 39 and 40, a task management method of a neural processing device according to some embodiments of the present invention will be described. The task management method according to the embodiment is a method performed in the neural processing device according to the above-described embodiment, and any overlapping parts with the above-described embodiment will be omitted or simplified. In addition, FIGS. 1 to 38 and the related descriptions may be referred to for the description of the present embodiment.

FIG. 39 is a flowchart illustrating a task management method of a neural processing device according to some embodiments of the present invention.

Referring to FIG. 39, a task management method according to an embodiment includes a step of receiving a task descriptor in a task queue (S100); a step of checking whether the task descriptor includes a waiting field (S200); and a step of queuing the task descriptor in the task queue if the task descriptor includes a waiting field (S300).

Additionally, referring to FIG. 39, the task management method according to some embodiments may further include a step (S400) of releasing a waiting state of a task descriptor in response to a progress signal provided from a command processor.

In step (S400), the progress signal may be a signal provided after the waiting state of the task descriptor.

Additionally, in step (S400), the progress signal may be a signal provided before the task descriptor waiting state.

In addition, referring to FIG. 39, a task management method according to some embodiments may further include a step (S500) of generating task information corresponding to a task descriptor whose standby state has been released or a task descriptor that does not include a standby field and transferring the generated task information to a core global; and a step (S600) of providing a task descriptor corresponding to the task information transferred to the core global as a dump passage as check-in data.

Figure 40 is a flowchart for detailing the steps of receiving a task descriptor in the task queue of Figure 39.

Referring to FIG. 40, in some embodiments, the step of receiving a task descriptor in a task queue (S100) includes the step of receiving a task from a command processor and generating a task descriptor for the task (S110), the step of storing the task descriptor in a first queue (S120), the step of checking dependencies of the task descriptor (S130), and the step of storing the task descriptor for which the dependency check is completed in a second queue (S140).

The above description is merely an illustrative description of the technical idea of the present embodiment, and those with ordinary skill in the art to which the present embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present embodiment.

Claims (20)

A task buffer that receives a task from a command processor and creates a task descriptor for the task;
A task queue in which the task descriptor received from the task buffer waits; and
Contains a runtime handle that creates task information corresponding to the task descriptor passed from the above task queue and passes it to the core global,
The above task descriptor is,
Contains a wait field that controls the execution timing of the above task,
The above runtime handle is,
By checking whether the above task descriptor includes the above waiting field for the above task, the status of the above task descriptor is determined as a progress state or a waiting state.
Task manager.
In the first paragraph,
The above runtime handle releases the waiting state of the task descriptor in response to a progress signal provided from the command processor.
Task manager.
In the second paragraph,
The above runtime handle includes a progress signal counter from which the progress signal is received.
Task manager.
In the third paragraph,
The above runtime handle releases the waiting state of the task descriptor through the above progress signal received in advance in the above progress signal counter.
Task manager.
In the third paragraph,
The above progress signal counter is configured to receive at least two progress signals,
Task manager.
In the second paragraph,
The above runtime handle generates task information corresponding to the task descriptor whose waiting state has been released or the task descriptor that does not include the waiting field and passes it to the core global, and provides the task descriptor corresponding to the task information passed to the core global as a check-in data to the dump passage.
Task manager.
In Article 6,
The above-mentioned Dunn passage receives a completion signal for the task information provided through the core global, and checks out the checked-in task descriptor in response to the completion signal to generate a completion report.
Task manager.
In Article 6,
The above runtime handle stops the transmission of task information to the core global based on a danger signal provided from the above-mentioned Dunn passage.
Task manager.
In the first paragraph,
The above task queue is,
A first queue receiving the task descriptor from the task buffer;
A dependency checker that receives a task descriptor from the first queue and performs a dependency check on the received task descriptor; and
A second queue comprising a second queue for receiving task descriptors for which dependency checks have been completed from the above dependency checker.
Task manager.
In Article 9,
The second queue includes a first task descriptor and a second task descriptor stored sequentially,
As the first task descriptor is waited in the second queue by the runtime handle, the second task descriptor also waits in the second queue.
Task manager.
A task manager that generates task information corresponding to a task descriptor;
A neural core that performs a task according to the above task information and generates a completion signal of the task; and
Including a core global that receives task information for the above task, transmits the task information to the neural core, and receives the completion signal of the task from the neural core;
The above task descriptor is,
Contains a wait field that controls the execution timing of the above task,
The above task manager,
By checking whether the above task descriptor includes the above waiting field for the above task, the status of the above task descriptor is determined as a progress state or a waiting state.
Neural processing unit.
In Article 11,
The above task manager,
A task passage that creates the above task descriptor and selectively creates the task information according to the above task descriptor and transmits it to the core global,
A task passage including a task descriptor that checks in the task descriptor from the task passage and checks out the task descriptor upon receiving the completion signal to generate a completion report.
Neural processing unit.
In Article 12,
The above task passage is,
A task buffer that receives a task from a command processor and creates a task descriptor for the task;
A task queue in which the task descriptor received from the task buffer waits; and
Contains a runtime handle that creates task information corresponding to the task descriptor passed from the above task queue and passes it to the core global,
The above runtime handle queues the task descriptor to the task queue if the task descriptor contains a wait field;
The above runtime handle releases the waiting state of the task descriptor in response to a progress signal provided from the command processor.
Neural processing unit.
In Article 13,
The above runtime handle generates task information corresponding to the task descriptor whose waiting state has been released or the task descriptor that does not include the waiting field and passes it to the core global, and provides the task descriptor corresponding to the task information passed to the core global as check-in data to the dunned passage.
Neural processing unit.
A step of receiving a task descriptor by a task queue of a neural processing device, wherein the task descriptor includes a wait field controlling execution timing of the task;
A step of checking whether the waiting field for the task of the task descriptor is included by the runtime handle of the neural processing device; and
A step of determining the state of the task descriptor as a progress state or a waiting state based on whether the waiting field is included or not by the runtime handle.
A method for managing a task of a neural processing device, comprising:
In Article 15,
Further comprising a step of releasing the waiting state of the task descriptor in response to a progress signal provided from the command processor of the neural processing device by the runtime handle.
A method for managing tasks in a neural processing device.
In Article 16,
The above progress signal is a signal provided after the waiting state of the above task descriptor.
A method for managing tasks in a neural processing device.
In Article 16,
The above progress signal is a signal provided before the waiting state of the task descriptor.
A method for managing tasks in a neural processing device.
In Article 16,
A step of generating task information corresponding to a task descriptor whose waiting state has been released or a task descriptor that does not include the waiting field and transferring the task information to the core global; and
Further comprising the step of providing a task descriptor corresponding to the task information transmitted to the core global as a check-in data to the Dunn passage.
A method for managing tasks in a neural processing device.
In Article 15,
The step of receiving the above task descriptor is:
A step of receiving a task from a command processor of the neural processing device and generating a task descriptor for the task;
A step of storing the task descriptor in a first queue of the neural processing device;
A step of checking the dependency of the above task descriptor; and
A step of storing the task descriptor for which the dependency check has been completed in the second queue of the neural processing device,
A method for managing tasks in a neural processing device.

Images