TWI869788B - Processing circuit and computation scheduling method of artificial intelligence model - Google Patents

Abstract

A processing circuit of an artificial intelligence (AI) model is provided and includes a memory, a memory management circuit, and an operation circuit. The memory management circuit reads a tensor from the external memory and stores the tensor in the memory. The operation circuit is configured to perform the following operations: performing an operation of a first type on a first and second sub-tensors and of the tensor to generate a first and second intermediate data, respectively; performing an operation of a second type on the first intermediate data and the second intermediate data to generate a third intermediate data; performing an operation of the first type on a third sub-tensor of the tensor to generate a fourth intermediate data; and performing an operation of the second type on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data.

Description

Processing circuit and operation scheduling method of artificial intelligence model

The present invention relates to an artificial intelligence model, and more particularly to a processing circuit and a calculation scheduling method of the artificial intelligence model.

In a system-on-chip (SoC), the total memory bandwidth is often fixed and used by multiple modules. When a module occupies too much memory bandwidth, it will cause other modules to be blocked from accessing memory, which will lead to reduced system performance. As a module in a system-on-chip, artificial intelligence (AI) models often need to process a large amount of data and have a large demand for memory bandwidth; therefore, reducing the bandwidth requirement of AI models has become an important topic.

In view of the deficiencies of the prior art, one purpose of the present invention is to provide a processing circuit and calculation scheduling method of an artificial intelligence model to improve the deficiencies of the prior art.

An embodiment of the present invention provides a processing circuit of an artificial intelligence model. The processing circuit is coupled to an external memory and includes a memory, a memory management circuit and an operation circuit. The memory management circuit is used to read a tensor from the external memory and store the tensor in the memory. The operation circuit is configured to perform the following operations: perform a first type of operation on a first sub-tensor of the tensor to generate a first intermediate data; perform the first type of operation on a second sub-tensor of the tensor to generate a second intermediate data; perform a second type of operation on the first intermediate data and the second intermediate data to generate a third intermediate data; perform the first type of operation on a third sub-tensor of the tensor to generate a fourth intermediate data; and perform the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data.

Another embodiment of the present invention provides a processing circuit of an artificial intelligence model. The processing circuit is coupled to an external memory and includes a memory. The processing circuit performs the following operations: read a tensor and a plurality of core parameters from the external memory, and store the tensor and the core parameters to the memory, wherein the tensor includes a first sub-tensor and a second sub-tensor, and the core parameters include a vector core parameter; perform a first vector operation on the first sub-tensor with reference to a first part of the vector core parameter to generate a first intermediate data; and perform a second vector operation on the second sub-tensor with reference to a second part of the vector core parameter to generate a second intermediate data. The first part of the vector core parameter is not equal to the second part of the vector core parameter.

Another embodiment of the present invention provides a method for scheduling operations of an artificial intelligence model, wherein the artificial intelligence model includes a first operator and a second operator. The operation scheduling method includes: dividing a tensor into H sub-tensors, where H is an integer greater than 1; dividing the first operator into H first sub-operators; dividing the second operator into H second sub-operators; determining a dependency relationship between the H first sub-operators and the H second sub-operators; sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain an operation order; and, determining when a processing circuit that executes the artificial intelligence model deletes a target data from a memory included in the processing circuit according to the operation order, the target data being an output data of one of the H first sub-operators and the H second sub-operators.

The technical means embodied in the embodiments of the present invention can improve at least one of the shortcomings of the prior art. Therefore, compared with the prior art, the present invention can reduce the memory usage and/or reduce the bandwidth requirement for the memory.

The features, implementation and effects of the present invention are described in detail below with reference to the accompanying drawings.

The technical terms used in the following descriptions refer to the customary terms in this technical field. If this manual explains or defines some of the terms, the interpretation of those terms shall be based on the explanation or definition in this manual.

The disclosure of the present invention includes a processing circuit of an artificial intelligence model and a calculation scheduling method. Since some components included in the processing circuit of the artificial intelligence model of the present invention may be known components individually, the following description will be omitted for details of the known components without affecting the full disclosure and feasibility of the device invention.

FIG1 is an example of an AI network, which can be viewed as a simple AI model or as part of a complex AI model. The AI network 100 is used to operate on input data Din to generate output data Dout. The AI network 100 of FIG1 includes three operators: a subtraction operator 110 (“SUB”), a convolution operator 120 (“CONV”), and an addition operator 130 (“ADD”). The subtraction operator 110 performs a subtraction operation on the tensor TS1 (i.e., the input data Din) to generate a tensor TS2. The convolution operator 120 performs a convolution operation on the tensor TS2 to generate a tensor TS3. The addition operator 130 performs an addition operation on the tensor TS3 to generate a tensor TS4 (ie, the output data Dout). In the example of FIG1 , the sizes (dimensional information) of the tensors TS1 , TS2 , TS3 , and TS4 are all [1, 3, 224, 224].

FIG2 is a flow chart of an embodiment of the method for scheduling computation of an artificial intelligence model of the present invention. The flow chart of FIG2 is executed by a chip development tool (eg, a computer) and includes the following steps.

Step S210: Split the tensor into H sub-tensors (or tiles), where H can be any dimension of the tensor (H is an integer greater than 1). More specifically, this step determines the value of H based on one of the dimensions of the output tensor of the last operator of the AI network 100, and then splits the tensor into H sub-tensors. Taking the AI network 100 of FIG. 1 as an example, since the size of the output tensor (i.e., tensor TS4) of the last operator (addition operator 130) is [1,3,224,224], H can be 3 or 224. The details of splitting the tensor will be explained below in conjunction with FIG. 3.

Step S220: Split the operator into H sub-operators. This step will be described together with step S210 below with reference to FIG. 3 .

Step S230: Determine the dependency relationship between multiple sub-operators. This step will be explained below with reference to FIG. 5 .

Step S240: Sort the sub-operators according to the dependency relationship between the sub-operators to obtain an operation order. This step will be explained below with reference to FIG. 7.

Step S250: Determine when the electronic device executing the artificial intelligence model (more specifically, the processing circuit of the electronic device) deletes a target data from the memory according to the operation sequence, the target data being the output data of one of the sub-operators (i.e., the intermediate data of the AI network 100). This step will be explained below with reference to FIG. 10 , FIG. 11 , FIG. 12A , and FIG. 12B .

Please refer to Figure 3, which is the result of the tensor and operator after segmentation in Figure 1. In the embodiment of Figure 3, the dimension according to which the operation of segmenting the tensor (i.e., step S210) is based on the second dimension of the tensor TS4 (i.e., H=3). Therefore, the subtraction operator 110 is divided into a subtraction sub-operator 110_1 ("SUB1"), a subtraction sub-operator 110_2 ("SUB2") and a subtraction sub-operator 110_3 ("SUB3"); the convolution operator 120 is divided into a convolution sub-operator 120_1 ("CONV1"), a convolution sub-operator 120_2 ("CONV2") and a convolution sub-operator 120_3 ("CONV3"); and the addition operator 130 is divided into an addition sub-operator 130_1 ("ADD1"), an addition sub-operator 130_2 ("ADD2") and an addition sub-operator 130_3 ("ADD3"). The tensor TS1 is divided into sub-tensors TS1_i1, TS1_i2 and TS1_i3 (respectively, the input sub-tensors of the subtraction sub-operator 110_1, 110_2 and 110_3, all of which have sizes of [1, 1, 224, 224] and correspond to the same dimension (e.g., the second dimension) of the tensor TS1). The tensor TS4 is divided into sub-tensors TS3_o1, TS3_o2 and TS3_o3 (respectively, the output sub-tensors of the addition sub-operator 130_1, 130_2 and 130_3, all of which have sizes of [1, 1, 224, 224] and correspond to the same dimension of the tensor TS4). Tensor TS3 is divided into sub-tensors TS3_i1, TS3_i2, and TS3_i3 (the input sub-tensors of addition sub-operator 130_1, addition sub-operator 130_2, and addition sub-operator 130_3, respectively, all of size [1,1,224,224], and corresponding to the same dimension of tensor TS3). The respective output sub-tensors (i.e., sub-tensors TS2_o1, sub-tensors TS2_o2, and sub-tensors TS2_o3) of convolution sub-operator 120_1, convolution sub-operator 120_2, and convolution sub-operator 120_3 are equal to sub-tensors TS3_i1, sub-tensors TS3_i2, and sub-tensors TS3_i3, respectively.

It should be noted that due to the problem of visual field enlargement, the output subtensor TS1_o1 (subtensor TS1_o2 or subtensor TS1_o3) of the subtraction operator 110_1 (subtraction operator 110_2 or subtraction operator 110_3) is not equal to the input subtensor TS2_i1 of the convolution operator 120_1 (convolution operator 120_2 or convolution operator 120_3). (subtensor TS2_i2 or subtensor TS2_i3); more specifically, the sizes of subtensors TS1_o1, TS1_o2, and TS1_o3 are all [1,1,224,224], but the sizes of subtensors TS2_i1 and TS2_i3 are both [1,2,224,224], and the size of subtensor TS2_i2 is [1,3,224,224].

As can be seen from Figure 3, since sub-tensor TS1_o1, sub-tensor TS1_o2 and sub-tensor TS1_o3 correspond to sub-tensor TS1_i1, sub-tensor TS1_i2 and sub-tensor TS1_i3 respectively, and sub-tensor TS1_i1, sub-tensor TS1_i2 and sub-tensor TS1_i3 correspond to the same dimension of tensor TS1 (for example, the second dimension), sub-tensor TS1_o1, sub-tensor TS1_o2 and sub-tensor TS1_o3 also correspond to the same dimension of tensor TS1.

The process of FIG. 2 can effectively manage the duration of target data in the memory of the electronic device, which helps to reduce memory usage and/or reduce the bandwidth requirement for the memory. The details will be explained below in conjunction with FIG. 9, FIG. 10, FIG. 11, FIG. 12A and FIG. 12B.

Based on the overlapping relationship between the sub-tensors split from the same tensor (see FIG3 ), a topological diagram of the connections between the sub-operators as shown in FIG4 can be obtained. Specifically, the topological diagram is obtained based on the overlapping relationship between the input sub-tensors of each sub-operator split from the same operator and the output sub-tensors of each sub-operator of the source operator on which it depends. As shown in the figure, the input sub-tensor TS2_i1 of the convolution sub-operator 120_1 includes the sub-tensor TS1_o1 and the sub-tensor TS1_o2; that is, the convolution sub-operator 120_1 needs to wait until the subtraction sub-operator 110_1 and the subtraction sub-operator 110_2 are both finished before it can start. Similarly, the convolution operator 120_2 needs to wait for the subtraction operator 110_1, the subtraction operator 110_2 and the subtraction operator 110_3 to finish before it can start; the convolution operator 120_3 needs to wait for the subtraction operator 110_2 and the subtraction operator 110_3 to finish before it can start. The addition operator 130_1, the addition operator 130_2 and the addition operator 130_3 need to wait for the convolution operator 120_1, the convolution operator 120_2 and the convolution operator 120_3 to finish before they can start.

Please refer to FIG. 5 , which is a detailed flow chart of an embodiment of step S230 of FIG. 2 , including the following steps. The details of FIG. 5 are explained below in conjunction with FIG. 4 .

Step S510: Determine the target sub-operator. For example, select the convolution sub-operator 120_1 as the target sub-operator.

Step S520: Determine the source sub-operator of the target sub-operator. Continuing with the above example, since the source of the input sub-tensor TS2_i1 of the convolution sub-operator 120_1 includes the sub-tensor TS1_o1 and the sub-tensor TS1_o2, the source sub-operator of the convolution sub-operator 120_1 is the subtraction sub-operator 110_1 and the subtraction sub-operator 110_2 (i.e., the output sub-tensor TS1_o1 of the subtraction sub-operator 110_1 and the output sub-tensor TS1_o2 of the subtraction sub-operator 110_2 are the input sub-tensors of the convolution sub-operator 120_1). Similarly, the source sub-operators of the convolution sub-operator 120_2 are the subtraction sub-operator 110_1, the subtraction sub-operator 110_2 and the subtraction sub-operator 110_3, the source sub-operators of the convolution sub-operator 120_3 are the subtraction sub-operator 110_2 and the subtraction sub-operator 110_3; and the source sub-operator of the addition sub-operator 130_1 is the convolution sub-operator 120_1.

Step S530: Determine whether the target sub-operator is dependent on the source sub-operator, that is, the source sub-operator is a dependent sub-operator of the target sub-operator. For example, subtraction sub-operator 110_1, subtraction sub-operator 110_2 and subtraction sub-operator 110_3 are dependent sub-operators of convolution sub-operator 120_2.

By taking each sub-operator of FIG. 4 as the target sub-operator in turn and repeating the process of FIG. 5 , the dependency relationship between the multiple sub-operators can be determined, as shown in FIG. 6 . The convolution sub-operator 120_1 depends on the subtraction sub-operator 110_1 and the subtraction sub-operator 110_2. The convolution sub-operator 120_2 depends on the subtraction sub-operator 110_1, the subtraction sub-operator 110_2 and the subtraction sub-operator 110_3. The convolution sub-operator 120_3 depends on the subtraction sub-operator 110_2 and the subtraction sub-operator 110_3. The adder operator 130_1, the adder operator 130_2 and the adder operator 130_3 are dependent on the convolution operator 120_1, the convolution operator 120_2 and the convolution operator 120_3 respectively.

Please refer to FIG. 7 , which is a detailed flow chart of an embodiment of step S240 in FIG. 2 , including the following steps: The flow chart of FIG. 7 is based on a depth first search algorithm.

Step S710: Find a sub-operator with indegree 0, and mark the sub-operator with indegree 0 as a target sub-operator and visited. A sub-operator with indegree 0 is a sub-operator that has no dependencies. Taking FIG6 as an example, the addition sub-operator 130_1, the addition sub-operator 130_2, and the addition sub-operator 130_3 are sub-operators with indegree 0, that is, the top-level sub-operators.

Step S720: Determine whether a sub-operator with in-degree 0 is found. If yes, execute step S730; if not, execute step S795. The following is an example of the addition sub-operator 130_1.

Step S730: Find the unvisited dependent sub-operators of the target sub-operator. As shown in FIG6 , since the convolution sub-operator 120_1 is the dependent sub-operator of the addition sub-operator 130_1 (ie, the addition sub-operator 130_1 depends on the convolution sub-operator 120_1), step S730 finds the convolution sub-operator 120_1.

Step S740: Determine whether the dependent sub-operator is found. If yes, execute step S750; if not, execute step S760.

Step S750: Mark the dependent sub-operator as the target sub-operator and has been visited, and then execute step S730. Continuing with the above example, in step S750, the convolution sub-operator 120_1 is marked as the target sub-operator and has been visited, and then when step S730 is executed again, the dependent sub-operator of the convolution sub-operator 120_1 is found (assuming that the subtraction sub-operator 110_1 is found). Then, step S730 and step S740 are executed again; at this time, because the subtraction sub-operator 110_1 does not depend on any sub-operator (that is, there is no dependent sub-operator, step S740 is no), the process goes to step S760.

Step S760: Add the target sub-operator to the queue 800. Continuing with the above example, the subtraction sub-operator 110_1 is added to the queue 800. Please refer to FIG8A and FIG8B, which show the changes in the content of the queue 800 (FIG8B follows FIG8A). As shown in the first column of FIG8A, the queue 800 only includes the subtraction sub-operator 110_1 ("SUB1").

Step S770: Determine whether the target sub-operator is a top-level sub-operator (i.e., a sub-operator with an in-degree of 0). If yes, execute step S710; if no, execute step S780. Continuing with the above example, since the subtraction sub-operator 110_1 is not a top-level sub-operator, the determination result of step S770 is no.

Step S780: Determine a parent sub-operator that is dependent on the target sub-operator (i.e., return to the parent sub-operator), and mark the parent sub-operator as the target sub-operator. Continuing with the above example, the process will return to the convolution sub-operator 120_1.

Step S790: Determine whether there are any unmarked dependent sub-operators. Continuing with the above example, at this time, because there are unmarked sub-operators (i.e., subtraction sub-operator 110_2) among the dependent sub-operators (subtraction sub-operator 110_1 and subtraction sub-operator 110_2) of the target sub-operator (convolution sub-operator 120_1), the determination result of step S790 is yes; then, the process executes the following steps: step S730 (find subtraction sub-operator 110_2) → step S740 (the determination result is yes) → step S75 0 (mark the subtraction operator 110_2 as visited) → step S730 (cannot find the dependent sub-operator of the subtraction operator 110_2) → step S740 (the judgment result is no) → step S760 (add the subtraction operator 110_2 to the queue 800 (as shown in the second column of Figure 8A)) → step S770 (the judgment result is no) → step S780 (mark the convolution operator 120_1 as the target sub-operator) → step S790. At this time, because all dependent sub-operators (i.e., subtraction sub-operators 110_1 and subtraction sub-operators 110_2) of the target sub-operator (i.e., convolution sub-operator 120_1) have been visited, the judgment result of step S790 is no, and therefore the convolution sub-operator 120_1 is added to the queue 800 in the next step S760 (as shown in the third column of FIG. 8A). After continuing to execute step S770, step S780 (marking the addition sub-operator 130_1 as the target sub-operator), step S790 and step S760 (the addition sub-operator 130_1 is added to the queue 800), the judgment result of step S770 is yes (because the addition sub-operator 130_1 is the top-level sub-operator), and the process returns to step S710 to select the next sub-operator with an in-degree of 0 (for example, the addition sub-operator 130_2).

The above steps S710 to S790 will be repeatedly executed (the process of adding all sub-operators in Figure 6 to queue 800 is shown in Figures 8A and 8B and will not be repeated) until all sub-operators with an in-degree of 0 have been visited (that is, the result of step S720 is no, and the process goes to step S795).

Step S795: Sequentially remove all sub-operators from the queue 800. Taking FIG. 8B as an example, the order in which the sub-operators are removed from the queue 800 (i.e., the operation order of the sub-operators) is: SUB1→SUB2→CONV1→…→CONV3→ADD3 (i.e., the opposite order of the order in which the sub-operators are added to the queue).

FIG9 is a functional block diagram of an embodiment of the electronic device of the present invention. The electronic device 900 includes a chip 901 and an external memory 902 (e.g., a dynamic random access memory (DRAM)). The chip 901 and the external memory 902 are coupled or electrically connected to each other. The chip 901 includes a processing circuit 910 and a processor 920. The processing circuit 910 and the processor 920 are coupled or electrically connected to each other.

The processor 920 controls the processing circuit 910 to realize the functions of the chip 901. The processor 920 can be a circuit or electronic component with program execution capability, such as a central processing unit, a microprocessor, a microprocessing unit, a digital signal processor, an application specific integrated circuit (ASIC), or an equivalent circuit thereof.

The processing circuit 910 may be an intelligence processing unit (IPU) or a neural-network processing unit (NPU). The processing circuit 910 includes an operation circuit 912 (e.g., including but not limited to a convolution engine and a vector engine), a temporary storage circuit 914 (e.g., including but not limited to a plurality of temporary storages), a memory management circuit 916 (e.g., direct memory access (DMA)), and a memory 918 (e.g., static random access memory (SRAM)). The temporary storage circuit 914 is used to store data required by the operation circuit 912 to perform convolution operations or vector operations. The memory 918 can store the output sub-tensors of each sub-operator in FIG. 3 .

The external memory 902 stores input data Din, core parameter Kp and output data Dout. The memory management circuit 916 is used to read the input data Din and core parameter Kp from the external memory 902 and store them in the memory 918, read at least a portion of the input data Din and at least a portion of the core parameter Kp from the memory 918 and store them in the temporary storage circuit 914, and store the output data Dout generated by the operation circuit 912 in the external memory 902.

The following is a detailed description of step S250 of FIG2 in conjunction with FIG9, FIG10, FIG11, FIG12A and FIG12B. FIG10 shows a list of the life spans of the output subtensors of some of the sub-operators of FIG3. The horizontal axis corresponds to the operation order of the aforementioned sub-operators (not necessarily corresponding to the actual length of time); more specifically, the output subtensor TS1_o1 of the subtraction sub-operator 110_1 is generated at the time point when the operation order is 0, and ends at the time point when the operation order is 5 (i.e., it will no longer be used by other sub-operators). Please note that sub-tensor TS3_o1, sub-tensor TS3_o2 and sub-tensor TS3_o3 are generated at time points of operation order 3, 6 and 8 respectively; however, because sub-tensor TS3_o1, sub-tensor TS3_o2 and sub-tensor TS3_o3 will not be deleted from memory 918 in advance (because each is part of the output data Dout), the life cycle list of Figure 10 does not show the three sub-tensors.

The details of step S250 of Figure 2 include allocating memory 918 according to the life cycle list of Figure 10, and the process of allocating memory 918 is shown in Figure 11. Figures 12A and 12B are schematic diagrams of an embodiment of the active list of the present invention. Please refer to Figures 10, 11, 12A and 12B for the following description. The active list is used to display the activity status of the sub-tensor in the memory 918 (more specifically, it displays the time points when the sub-tensor is stored in the memory 918 and deleted from the memory 918). Figure 11 includes the following steps.

Step S1110: Create a life cycle list. An example of a life cycle list is shown in FIG10 .

Step S1120: Search the life cycle list to find the sub-tensor that is currently active in the life cycle. For example, sub-tensor TS1_o1 becomes active at the time point when the operation sequence is 0, and the active period of sub-tensor TS1_o1 is from the time point when the operation sequence is 0 to the time point when the operation sequence is 5.

Step S1130: Add the active sub-tensor to the active list. As shown in FIG12A , the sub-tensor TS1_o1 is added to the active list when the life cycle is 0.

Step S1140: Allocate memory to the active sub-tensor, that is, arrange corresponding storage space in the memory 918. Continuing with the above example, as shown in FIG12A , part of the memory 918 is allocated to the sub-tensor TS1_o1 when the life cycle is 0.

Step S1150: Delete the sub-tensor that is no longer active in the active list. For example, because the sub-tensor TS2_o1 in FIG. 10 is no longer active after the time point when the operation sequence is 3, the sub-tensor TS2_o1 in FIG. 12A is deleted when the life cycle is 3.

Step S1160: Release the memory corresponding to the sub-tensor that is no longer active. In response to the deletion of the sub-tensor from the active list in the previous step, this step releases the corresponding storage space in the memory 918, so that the memory 918 can be used more timely and flexibly.

Step S1170: The life cycle is increased by 1.

If there is no sub-tensor that is no longer active in the current life cycle, skip steps S1150 and S1160 and directly execute step S1170.

Step S1180: Determine whether the life cycle is terminated (i.e., determine whether the operation sequence of FIG. 10 is terminated). If yes, terminate the process of FIG. 11; if no, execute step S1120 to continue to find active sub-tensors.

As described above, the active lists of FIG. 12A and FIG. 12B can be obtained according to the life cycle list of FIG. 10 and the process of FIG. 11. The life cycle of FIG. 12A and FIG. 12B corresponds to the operation sequence of FIG. 10. For example, in FIG. 10, subtensor TS1_o1 is generated at the time point of operation sequence 0 and ends at the time point of operation sequence 5; therefore, the life cycle of subtensor TS1_o1 in FIG. 12A is 0~4. Similarly, in FIG. 10, subtensor TS2_o2 exists between the time point of operation sequence 5 and the time point of operation sequence 6; therefore, in FIG. 12B, subtensor TS2_o2 only exists in life cycle 5. In this way, the developer or designer of the chip 901 can design or manage the memory 918 according to the active list of Figures 12A and 12B; therefore, the bandwidth requirement of the external memory 902 can be reduced (the overall performance of the external memory 902 can be improved) without increasing the memory 918 (in order to save costs), or the memory 918 can be further saved without increasing the memory bandwidth of the external memory 902.

In comparison, if the operator and tensor of Figure 1 are not split, the chip developer must pre-allocate one of the storage blocks of memory 918 to tensor TS2 (the total amount of data is equivalent to the sum of the data of sub-tensor TS1_o1, sub-tensor TS1_o2 and sub-tensor TS1_o3), and the storage block cannot be released until the convolution operator 120 ends; in addition, if the sub-operations after the operator is split are not sorted in order of operation, the same sub-tensor such as sub-tensor TS1_i1 needs to be repeatedly operated multiple times, which increases the data volume of the entire AI model, resulting in increased costs (because of increased demand for memory 918) or decreased performance (because of increased bandwidth requirements for external memory 902). In actual operation, since the number of operators and the size of tensors are very large, the effect that can be achieved by the present invention is quite significant.

The present invention can be extended to an AI model that includes more operators. Please refer to Figure 13, which is a schematic diagram of another AI model. The left side of Figure 13 shows that the AI model includes N operators (operator 1, operator 2, ..., operator N), and the right side of Figure 13 shows that an operator is divided into H sub-operators. The arrows represent the dependency relationship when the sub-operators are executed. For example, the 1st sub-operator of operator 2 must wait until the 3rd sub-operator of operator 1 is executed before it can be executed.

FIG14 is a schematic diagram of another AI model. AI network 1400 includes an addition operator 1410 and a convolution operator 1420, and the size of the tensor is [1,56,56,224]. After segmentation (as shown in the lower half of FIG14), the addition operator 1410 and the convolution operator 1420 are each segmented into 56 sub-operators ("ADD1" to "ADD56" and "CONV1" to "CONV56"). The size of the sub-tensor output by each addition sub-operator ("ADD1" to "ADD56") is [1,1,56,224]. However, not all convolution sub-operators ("CONV1" to "CONV56") have the same size of input sub-tensors. More specifically, the input subtensor size of the first convolution operator ("CONV1") is [1,2,56,224], while the input subtensor size of the second convolution operator ("CONV2") is [1,3,56,224] (i.e., the aforementioned field of view magnification problem).

Figures 15A and 15B are flow charts of an embodiment of an implementation method of the artificial intelligence model of the present invention. Figures 15A and 15B include the following steps.

Step S1505: The memory management circuit 916 reads a tensor (ie, input data Din) and a plurality of core parameters Kp from the external memory 902, and stores the tensor and the core parameters Kp in the memory 918.

Step S1510: The processing circuit 910 (more specifically, the operation circuit 912) performs a first type of operation on a first sub-tensor of the tensor to generate a first intermediate data, and the memory management circuit 916 stores the first intermediate data into the memory 918. Taking FIG. 4 as an example, the first sub-tensor may be an input sub-tensor (i.e., sub-tensor TS1_i1) of the subtraction sub-operator 110_1, the first type of operation may be a subtraction operation (a type of vector operation), and the first intermediate data may be an output sub-tensor (i.e., sub-tensor TS1_o1) of the subtraction sub-operator 110_1. Taking FIG. 14 as an example, the first sub-tensor may be an input sub-tensor of an addition sub-operator (e.g., “ADD1”), the first type of operation may be an addition operation (a type of vector operation), and the first intermediate data may be an output sub-tensor of the addition sub-operator (e.g., “ADD1”).

Step S1520: The processing circuit 910 (more specifically, the operation circuit 912) performs the first type of operation on a second sub-tensor of the tensor to generate a second intermediate data, and the memory management circuit 916 stores the second intermediate data in the memory 918. Taking FIG. 4 as an example, the second sub-tensor may be an input sub-tensor (i.e., sub-tensor TS1_i2) of the subtraction sub-operator 110_2, the first type of operation may be a subtraction operation, and the second intermediate data may be an output sub-tensor (i.e., sub-tensor TS1_o2) of the subtraction sub-operator 110_2. Taking Figure 14 as an example, the second sub-tensor can be the input sub-tensor of the addition sub-operator (e.g., "ADD2"), the first type of operation can be an addition operation, and the second intermediate data can be the output sub-tensor of the addition sub-operator (e.g., "ADD2").

Step S1530: The processing circuit 910 (more specifically, the operation circuit 912) performs a second type of operation on the first intermediate data and the second intermediate data to generate a third intermediate data, and the memory management circuit 916 stores the third intermediate data into the memory 918. Taking FIG. 4 and FIG. 14 as examples, the second type of operation may be a convolution operation (e.g., convolution sub-operator 120_1 or "CONV1"), and the third intermediate data may be a result of the convolution operation (e.g., sub-tensor TS2_o1 of FIG. 4).

Step S1540: The memory management circuit 916 deletes the third intermediate data from the memory 918. As shown in FIG10 , since the operations after the time point of the operation sequence 3 will no longer use the sub-tensor TS2_o1 (i.e., the sub-tensor TS2_o1 becomes inactive starting from the life cycle 3 of FIG12A ), the sub-tensor TS2_o1 can be deleted from the memory 918 to release part of the memory 918.

Step S1550: The processing circuit 910 (more specifically, the operation circuit 912) performs the first type of operation on a third sub-tensor of the tensor to generate a fourth intermediate data, and the memory management circuit 916 stores the fourth intermediate data in the memory 918. Taking FIG. 4 as an example, the third sub-tensor may be an input sub-tensor (i.e., sub-tensor TS1_i3) of the subtraction sub-operator 110_3, the first type of operation may be a subtraction operation, and the fourth intermediate data may be an output sub-tensor (i.e., sub-tensor TS1_o3) of the subtraction sub-operator 110_3. Taking FIG. 14 as an example, the third sub-tensor may be an input sub-tensor of an addition sub-operator (e.g., “ADD3”), the first type of operation may be an addition operation, and the fourth intermediate data may be an output sub-tensor of the addition sub-operator (e.g., “ADD3”).

Step S1560: The processing circuit 910 (more specifically, the operation circuit 912) performs the second type of operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data, and the memory management circuit 916 stores the fifth intermediate data into the memory 918. Taking FIG. 4 and FIG. 14 as examples, the second type of operation may be a convolution operation (e.g., the convolution sub-operator 120_2 or "CONV2"), and the fifth intermediate data may be the result of the convolution operation (e.g., the sub-tensor TS2_o2 of FIG. 4).

Step S1570: The memory management circuit 916 deletes the first intermediate data from the memory 918. As shown in FIG10 , since the operation after the time point of the operation sequence 5 will no longer use the sub-tensor TS1_o1 (i.e., the sub-tensor TS1_o1 becomes inactive starting from the life cycle 5 of FIG12B ), the sub-tensor TS1_o1 can be deleted from the memory 918 to release part of the memory 918.

Step S1580: The memory management circuit 916 deletes the fifth intermediate data from the memory 918. As shown in FIG10 , since the operations after the time point of the operation sequence 6 will no longer use the sub-tensor TS2_o2 (i.e., the sub-tensor TS2_o2 becomes inactive starting from the life cycle 6 of FIG12B ), the sub-tensor TS2_o2 can be deleted from the memory 918 to release part of the memory 918.

As shown in the discussion of Figures 15A and 15B, the developer of chip 901 can arrange the instructions executed by the processing circuit 910 (more specifically, the memory management circuit 916) (provided to the processing circuit 910 by the processor 920 in some embodiments) according to the usage status of the memory 918 in advance to achieve the purpose of fully utilizing the memory 918.

Please refer to Figure 16, which is a schematic diagram of an embodiment of the temporary storage circuit 914 and the storage content of the memory 918 of the present invention. In the example of Figure 16, the core parameter Kp stored in the memory 918 includes a subtraction core parameter Kp_s (a type of vector core parameter), a convolution core parameter Kp_c and an addition core parameter Kp_a (a type of vector core parameter).

The subtraction core parameter Kp_s includes subparameters Kp_s1, Kp_s2, and Kp_s3. The subtraction core parameter Kp_s is a parameter required by the subtraction operator 110 to perform a subtraction operation on the tensor TS1, and the subparameters Kp_s1, Kp_s2, and Kp_s3 can correspond to the subtraction suboperator 110_1, 110_2, and 110_3 of FIG. 4, respectively. That is, the operation circuit 912 refers to the sub-parameter Kp_s1 (Kp_s2 or Kp_s3) to operate the sub-tensor TS1_i1 (TS1_i2 or TS1_i3) when executing the subtraction sub-operator 110_1 (110_2 or 110_3). In some embodiments, the sub-parameter Kp_s1 is not equal to the sub-parameter Kp_s2 and the sub-parameter Kp_s3, and the sub-parameter Kp_s2 is not equal to the sub-parameter Kp_s3.

The addition core parameter Kp_a includes sub-parameters Kp_a1, Kp_a2, and Kp_a3. The addition core parameter Kp_a is a parameter required by the addition operator 130 to perform addition operations on the tensor TS3, and the sub-parameters Kp_a1, Kp_a2, and Kp_a3 can correspond to the addition sub-operator 130_1, the addition sub-operator 130_2, and the addition sub-operator 130_3 in FIG. 4, respectively. In other words, when executing the addition sub-operator 130_1 (130_2 or 130_3), the operation circuit 912 refers to the sub-parameter Kp_a1 (Kp_a2 or Kp_a3) to operate on the sub-tensor TS3_i1 (TS3_i2 or TS3_i3). In some embodiments, sub-parameter Kp_a1 is not equal to sub-parameter Kp_a2 and sub-parameter Kp_a3, and sub-parameter Kp_a2 is not equal to sub-parameter Kp_a3.

The convolution core parameter Kp_c includes sub-parameters Kp_c1, Kp_c2, and Kp_c3. Different from the subtraction operation and the addition operation, although the operator and the tensor are split, the convolution sub-operator 120_1, the convolution sub-operator 120_2, and the convolution sub-operator 120_3 still need to refer to the complete convolution core parameter Kp_c when performing the convolution operation.

Please refer to FIG. 17 , which is a detailed flow chart of step S1510 or step S1520 of FIG. 15A , including the following steps. Please refer to FIG. 16 for the following description.

Step S1710: The memory management circuit 916 reads a target portion of the vector core parameter from the memory 918 and stores the target portion in the temporary storage circuit 914. More specifically, for step S1510, the target portion may be the sub-parameter Kp_s1. For step S1520, the target portion may be the sub-parameter Kp_s2. As shown in FIG. 16 , the temporary storage circuit 914 stores the sub-parameter Kp_s1 and/or the sub-parameter Kp_s2 and other data (e.g., the sub-tensor to be operated). Since the first type of operation (e.g., vector operation) in step S1510 and step S1520 only needs to use a portion (i.e., the target portion) of the subtraction core parameter Kp_s, the temporary storage circuit 914 does not need to store the complete subtraction core parameter Kp_s to save storage space. In some embodiments, the memory management circuit 916 stores the sub-parameter Kp_s1 (Kp_s2) in the temporary storage circuit 914 before step S1510 (S1520), and deletes the sub-parameter Kp_s1 (Kp_s2) from the temporary storage circuit 914 after step S1510 (S1520) is completed to save storage space.

Step S1720: The operation circuit 912 performs a target vector operation on the target subtensor with reference to the target portion of the vector core parameter to generate target intermediate data. More specifically, for step S1510, the target subtensor may be subtensor TS1_i1, the target vector operation may be subtraction suboperator 110_1, and the target intermediate data may be subtensor TS1_o1. For step S1520, the target subtensor may be subtensor TS1_i2, the target vector operation may be subtraction suboperator 110_2, and the target intermediate data may be subtensor TS1_o2. For Figure 14, the target part can be the sub-parameter Kp_a1, the target sub-tensor can be the input sub-tensor of an addition sub-operator (for example, "ADD1"), the target vector operation can be the addition sub-operator, and the first intermediate data can be the output sub-tensor of the addition sub-operator.

As mentioned above, because the original tensor is split into multiple sub-tensors, vector operations on the sub-tensors only require reference to some of the core parameters.

A person skilled in the art can understand the details of step S1550 of FIG. 15B from the description of FIG. 17 , so it is not described in detail. For example, for step S1550, the target part may be sub-parameter Kp_s3, the target sub-tensor may be sub-tensor TS1_i3, the target vector operation may be subtraction sub-operator 110_3, and the target intermediate data may be sub-tensor TS1_o3.

Please refer to FIG. 18, which is a detailed flow chart of step S1530 of FIG. 15A, including the following steps. Please refer to FIG. 19 for the following description.

Step S1810: The memory management circuit 916 reads the convolution kernel parameter Kp_c from the memory 918 and stores the convolution kernel parameter Kp_c to the temporary storage circuit 914. As shown in FIG19, the temporary storage circuit 914 stores the convolution kernel parameter Kp_c and other data (e.g., the sub-tensor to be operated) before the convolution operation starts. Since the second type of operation (e.g., convolution operation) of step S1530 requires the complete convolution kernel parameter Kp_c, the temporary storage circuit 914 needs to store the complete convolution kernel parameter Kp_c at this time.

Step S1820: The operation circuit 912 performs the second type of operation on the first intermediate data and the second intermediate data with reference to the convolution core parameter Kp_c to generate the third intermediate data. The convolution operation on the tensor with reference to the convolution core parameter Kp_c is well known to those skilled in the art and will not be described in detail. In some embodiments, the memory management circuit 916 deletes the convolution core parameter Kp_c from the temporary storage circuit 914 after step S1820 is completed.

Please refer to FIG. 20 , which is a detailed flow chart of step S1560 of FIG. 15B , including the following steps. Please refer to FIG. 19 for the following description.

Step S2010: Step S2010 is similar to step S1810 and will not be described in detail. Please note that if the memory management circuit 916 does not delete the convolution kernel parameter Kp_c from the temporary storage circuit 914 after step S1530, step S2010 can be skipped.

Step S2020: The operation circuit 912 performs the second type of operation on the first intermediate data, the second intermediate data and the fourth intermediate data with reference to the convolution core parameter Kp_c to generate the fifth intermediate data. Step S2020 is similar to step S1820, so it will not be described again.

Please refer to FIG. 21 , which is a functional block diagram of an embodiment of the memory management circuit 916 of the present invention. The memory management circuit 916 includes at least two channels (channel 916a and channel 916b), each channel can be operated independently, and multiple channels can be operated simultaneously. Based on this feature, the present invention further divides the sub-operators and sub-tensors into several small blocks, so that the processing circuit 910 can use a multi-stage pipeline to operate the AI model to improve the performance of the chip 901.

Please refer to FIG. 22, which is a schematic diagram of the multi-stage pipeline of the present invention. FIG. 22 uses the subtraction sub-operator 110_1, the subtraction sub-operator 110_2 and the convolution sub-operator 120_1 as examples for explanation. In the example of FIG. 22, the subtraction sub-operator 110_1 is further divided into operation blocks 110_1a ("SUB1a") and operation blocks 110_1b ("SUB1b"), and the sub-tensor TS1_i1 is further divided into data blocks TS1_i1a and data blocks TS1_i1b; the subtraction sub-operator 110_2 is further divided into operation blocks 110_2a ("SUB2a") and operation blocks 110_ 2b ("SUB2b"), and the sub-tensor TS1_i2 is further divided into data blocks TS1_i2a and TS1_i2b; the convolution sub-operator 120_1 is further divided into operation blocks 120_1a ("CONV1a") and operation blocks 120_1b ("CONV1b"), and the sub-tensor TS2_i1 is further divided into data blocks TS2_i1a and TS2_i1b. In this way, when the channel 916a performs the operation related to the operation block 110_1a (between the time point T0 and the time point T3), the channel 916b can substantially perform the operation related to the operation block 110_1b (between the time point T1 and the time point T4) at the same time. Compared to a single-stage pipeline (i.e., not using multiple channels at the same time to run the AI model), using two channels simultaneously can save about half the time. Similarly, using N channels simultaneously takes about 1/N the processing time of a single-stage pipeline.

Please refer to Figure 23, which is a flow chart of an embodiment of the multi-stage pipeline operation of the present invention, including the following steps.

Step S2310: The memory management circuit 916 uses the first channel (e.g., channel 916a) to read the first data block (e.g., data block TS1_i1a) of the first sub-tensor (e.g., sub-tensor TS1_i1) from the memory 918, and stores the first data block into the temporary storage circuit 914. For example, step S2310 may correspond to between time point T0 and time point T1 in FIG. 22 (i.e., "SUB1a load" operation).

Step S2320: The operation circuit 912 performs a first type of operation (e.g., a subtraction operation) on the first data block (e.g., data block TS1_i1a) to generate a first portion of the first intermediate data (e.g., a portion of the subtensor TS1_o1). For example, step S2320 may correspond to between time point T1 and time point T2 in FIG. 22 (i.e., the "SUB1a compute" operation).

Step S2330: The memory management circuit 916 uses the second channel (e.g., channel 916b) to read the second data block (e.g., data block TS1_i1b) of the first sub-tensor from the memory 918, and stores the second data block into the temporary storage circuit 914. For example, step S2330 may correspond to between time point T1 and time point T2 in FIG. 22 (i.e., "SUB1b load" operation). In other words, step S2320 and step S2330 are executed at least partially simultaneously.

Step S2340: The memory management circuit 916 uses the first channel (e.g., channel 916a) to store the first portion of the first intermediate data (e.g., a portion of the subtensor TS1_o1) to the memory 918. For example, step S2340 may correspond to between time point T2 and time point T3 in FIG. 22 (i.e., "SUB1a store" operation).

Step S2350: The operation circuit 912 performs the first type of operation (e.g., subtraction operation) on the second data block (e.g., data block TS1_i1b) to generate a second portion of the first intermediate data (e.g., a portion of the subtensor TS1_o1). For example, step S2350 may correspond to between time point T2 and time point T3 in FIG. 22 (i.e., "SUB1b calculation" operation). In other words, step S2340 and step S2350 are at least partially executed simultaneously.

Step S2360: The memory management circuit 916 uses the second channel (e.g., channel 916b) to store the second portion of the first intermediate data (e.g., a portion of the subtensor TS1_o1) to the memory 918. For example, step S2360 may correspond to between time point T3 and time point T4 in FIG. 22 (i.e., “SUB1b store” operation).

A person having ordinary knowledge in this technical field can understand other operations after time point T4 in Figure 22 based on the description of Figure 23, so they will not be elaborated here.

Although the embodiments of the present invention are described above, these embodiments are not intended to limit the present invention. A person having ordinary knowledge in the technical field may modify the technical features of the present invention according to the explicit or implicit contents of the present invention. All such modifications may fall within the scope of patent protection sought by the present invention. In other words, the scope of patent protection of the present invention shall be subject to the scope of the patent application defined in this specification.

100,1400:AI network Din:input data Dout:output data 110,SUB:subtraction operator 120,1420,CONV:convolution operator 130,1410,ADD:addition operator TS1,TS2,TS3,TS4:tensor 110_1,110_2,110_3,SUB1,SUB2,SUB3:subtraction operator 120_1,120_2,120_3,CONV1,CONV2,CONV3,CONV56:convolution operator 130_1,130_2,130_3,ADD1,ADD2,ADD3,ADD56:addition operator TS1_i1,TS1_i2,TS1_i3,TS3_o1,TS3_o2,TS3_o3,TS3_i1,TS3_i2,TS3_i3,TS2_o1,TS2_o2,TS2_o3,TS1_o1,TS1_o2,TS1_o3,TS2_i1,TS2_i2,TS2_i3: subtensor 800: queue 900: electronic device 901: chip 902: external memory 910: processing circuit 920: processor 912: operation circuit 914: cache circuit 916: memory management circuit 918: memory Kp: core parameter Kp_s: subtraction core parameter Kp_c: Convolution core parameter Kp_a: Addition core parameter Kp_s1, Kp_s2, Kp_s3, Kp_a1, Kp_a2, Kp_a3, Kp_c1, Kp_c2, Kp_c3: Sub parameter 916a, 916b: Channel 110_1a, 110_1b, 110_2a, 110_2b, 120_1a, 120_1b, SUB1a, SUB1b, SUB2a, SUB2b, CONV1a, CONV1b: Operation block TS1_i1a, TS1_i1b, TS1_i2a, TS1_i2b, TS2_i1a, TS2_i1b: Data block T0, T1, T2, T3, T4: Time point S210,S220,S230,S240,S250,S510,S520,S530,S710,S720,S730,S740,S750,S 760,S770,S780,S790,S795,S1110,S1120,S1130,S1140,S1150,S1160,S1170,S 1180,S1505,S1510,S1520,S1530,S1540,S1550,S1560,S1570,S1580,S1710,S1720,S1810,S1820,S2010,S2020,S2310,S2320,S2330,S2340,S2350,S2360: Steps

FIG1 shows an example of an AI network; FIG2 is a flow chart of an embodiment of the operation scheduling method of the artificial intelligence model of the present invention; FIG3 is the result of the tensor and operator of FIG1 after segmentation; FIG4 shows a topological graph of the connection between sub-operators; FIG5 is a detailed flow chart of an embodiment of step S230 of FIG2; FIG6 shows the dependency relationship between multiple sub-operators; FIG7 is a detailed flow chart of an embodiment of step S240 of FIG2; FIG8A and FIG8B show schematic diagrams of an embodiment of the queue of the present invention; FIG9 is a functional block diagram of an embodiment of the electronic device of the present invention; FIG10 shows a schematic diagram of an embodiment of the life cycle list of the present invention; FIG11 shows a flowchart of memory allocation; FIG12A and FIG12B are schematic diagrams of an embodiment of the active list of the present invention; FIG13 is a schematic diagram of another AI model; FIG14 is a schematic diagram of another AI model; FIG15A and FIG15B are flowcharts of an embodiment of the execution method of the artificial intelligence model of the present invention; FIG16 is a schematic diagram of an embodiment of the temporary storage circuit and the storage content of the memory of the present invention; FIG17 is a detailed flow of step S1510 or step S1520 of FIG15A; FIG18 is a detailed flow of step S1530 of FIG15A; FIG19 is a schematic diagram of another embodiment of the temporary storage circuit and the storage content of the memory of the present invention; FIG. 20 is a detailed flow chart of step S1560 of FIG. 15B; FIG. 21 is a functional block diagram of an embodiment of the memory management circuit of the present invention; FIG. 22 is a schematic diagram of a multi-stage pipeline of the present invention; and FIG. 23 is a flow chart of an embodiment of the multi-stage pipeline operation of the present invention.

900: Electronic devices

901: Chip

902: External memory

910: Processing circuit

912: Operational circuit

914: Temporary circuit

916: Memory management circuit

918:Memory

920: Processor

Kp: core parameters

Din: Input data

Dout: output data

Claims (16)

A processing circuit of an artificial intelligence model is coupled to an external memory, comprising: a memory; a memory management circuit coupled to the memory, used to read a tensor from the external memory and store the tensor in the memory; and an operation circuit coupled to the memory management circuit, configured to perform the following operations: performing a first type of operation on a first sub-tensor of the tensor to generate a first intermediate data; performing the first type of operation on a second sub-tensor of the tensor to generate a second intermediate data; performing a second type of operation on the first intermediate data and the second intermediate data to generate a third intermediate data; performing the first type of operation on a third sub-tensor of the tensor , to generate a fourth intermediate data; and perform the second type of operation on the first intermediate data, the second intermediate data and the fourth intermediate data to generate a fifth intermediate data; wherein the memory management circuit determines an operation sequence of the first intermediate data and the second intermediate data based on the dependency relationship between the first type of operation and the second type of operation, stores the first intermediate data and the second intermediate data that will participate in the second type of operation into the memory, and based on the operation sequence of the first intermediate data, knows that the first intermediate data will no longer participate in the operation after the second type of operation is performed to generate the fifth intermediate data, and deletes the first intermediate data from the memory. A processing circuit as claimed in claim 1, wherein the first type of operation is one of an addition operation and a subtraction operation, and the second type of operation is a product operation. A processing circuit as in claim 1, wherein the first intermediate data, the second intermediate data, and the fourth intermediate data correspond to the same dimension of the tensor. As in the processing circuit of claim 3, wherein the fourth intermediate data is generated after the third intermediate data is generated. The processing circuit of claim 1 further comprises: a temporary storage circuit; wherein, when the operation circuit performs the first type of operation, the memory management circuit reads at least a portion of a core parameter from the memory to the temporary storage circuit, and the operation circuit only refers to the at least a portion of the core parameter to perform the first type of operation; wherein the first type of operation is one of a subtraction operation and an addition operation. A processing circuit as in claim 1, wherein the first intermediate data, the second intermediate data, and the fourth intermediate data are of the same size. The processing circuit of claim 1, wherein the memory management circuit includes a first channel and a second channel, and the first type of operation on the first sub-tensor of the tensor includes the following steps: (A) using the first channel to read a first data block of the first sub-tensor from the memory; (B) performing the first type of operation on the first data block to generate a first part of the first intermediate data; (C) using the second channel to read a first data block of the first sub-tensor from the memory (D) storing the first portion of the first intermediate data to the memory using the first channel; (E) performing the first type of operation on the second data block to generate a second portion of the first intermediate data; and (F) storing the second portion of the first intermediate data to the memory using the second channel; wherein the step (B) and the step (C) are at least partially performed simultaneously, and the step (D) and the step (E) are at least partially performed simultaneously. A processing circuit of an artificial intelligence model is coupled to an external memory and includes a memory, wherein the processing circuit performs the following operations: reading a tensor and a plurality of core parameters from the external memory and storing the tensor and the core parameters in the memory, wherein the tensor includes a first sub-tensor and a second sub-tensor, and the core parameters include a vector core parameter; performing a first vector operation on the first sub-tensor with reference to a first part of the vector core parameter to generate a first intermediate data; and performing a first vector operation on the first sub-tensor with reference to the vector core parameter. A second vector operation is performed on the second sub-tensor with reference to a second part of the vector core parameter to generate a second intermediate data; wherein the first part of the vector core parameter is not equal to the second part of the vector core parameter; wherein the tensor further includes a third sub-tensor, and the core parameters further include a convolution core parameter for a convolution operation, and the processing circuit further performs the following operations: The convolution operation is performed on the first intermediate data and the second intermediate data with reference to the convolution core parameter to generate a third intermediate data; and after the convolution operation , performing a third vector operation on the third sub-tensor with reference to a third part of the vector core parameter to generate a fourth intermediate data; wherein the convolution operation is a first convolution operation, and the processing circuit further performs the following operations: performing a second convolution operation on the first intermediate data, the second intermediate data, and the fourth intermediate data with reference to the convolution core parameter to generate a fifth intermediate data; wherein the first intermediate data is stored in the memory, and the processing circuit further performs the following operations: after the second convolution operation, performing a second convolution operation on the first intermediate data, the second intermediate data, and the fourth intermediate data to generate a fifth intermediate data; wherein the first intermediate data is stored in the memory, and the processing circuit further performs the following operations: after the second convolution operation, performing a second convolution operation on the first intermediate data, the second intermediate data, and the fourth intermediate data The first intermediate data and the second intermediate data to be involved in a second type of operation are deleted from the memory; wherein, the first intermediate data and the second intermediate data are stored in the memory in an operation order determined based on a dependency relationship between the first type of operation and the second type of operation, and the first intermediate data is deleted from the memory after it is known based on the operation order of the first intermediate data that the first intermediate data will no longer participate in the operation after the fifth intermediate data is generated by performing the second type of operation. A processing circuit as claimed in claim 8, wherein the first sub-tensor and the second sub-tensor correspond to the same dimension of the tensor. A processing circuit as claimed in claim 8, wherein the first vector operation and the second vector operation are one of an addition operation and a subtraction operation. The processing circuit of claim 8, wherein the processing circuit further comprises a memory management circuit, the memory management circuit comprises a first channel and a second channel, and the first vector operation on the first sub-tensor to generate the first intermediate data comprises the following steps: (A) using the first channel to read a first data block of the first sub-tensor from the memory; (B) performing an operation on the first data block to generate a first part of the first intermediate data; (C) using the second channel to read the first data block from the memory; (B) reading a second data block of the first subtensor from the first channel; (D) using the first channel to store the first portion of the first intermediate data to the memory; (E) performing the operation on the second data block to generate a second portion of the first intermediate data; and (F) using the second channel to store the second portion of the first intermediate data to the memory; wherein the step (B) and the step (C) are at least partially executed simultaneously, and the step (D) and the step (E) are at least partially executed simultaneously. A method for scheduling operations of an artificial intelligence model, the artificial intelligence model comprising a first operator and a second operator, the output of the first operator being the input of the second operator, the method comprising: dividing a tensor into H sub-tensors, H being an integer greater than 1; dividing the first operator into H first sub-operators; dividing the second operator into H second sub-operators; determining the H first sub-operators according to the input of each of the H second sub-operators A dependency relationship between the H first sub-operators and the H second sub-operators; sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain an operation order; and determining when a processing circuit executing the artificial intelligence model deletes a target data from a memory included in the processing circuit according to the operation order, the target data being an output data of one of the H first sub-operators and the H second sub-operators. As in claim 12, the step of determining the dependency relationship between the H first sub-operators and the H second sub-operators includes: determining a target sub-operator; determining a source sub-operator of the target sub-operator according to the input of the target sub-operator, wherein the output of the source sub-operator is the input of the target sub-operator; and determining that the target sub-operator is dependent on the source sub-operator. The operation scheduling method of claim 12, wherein the step of sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain the operation order includes: (A) determining a target sub-operator; (B) determining a source sub-operator on which the target sub-operator depends; (C) when the source sub-operator does not depend on any sub-operator, adding the source sub-operator to a queue; (D) repeating the steps (B) to (C) until all the source sub-operators on which the target sub-operator depends have been added to the queue; and (E) adding the target sub-operator to the queue. The operation scheduling method of claim 14, wherein the step of sorting the H first sub-operators and the H second sub-operators according to the dependency relationship to obtain the operation order further includes: (F) determining an upper-level sub-operator that depends on the target sub-operator; (G) using the upper-level sub-operator as the target sub-operator, repeating the steps (B) to (E); and (H) repeating the steps (F) and (G) until the target sub-operator is a top-level sub-operator. As in claim 14, the step of determining when to delete the target data from the memory according to the operation order includes: determining a sub-operator that uses the target data last according to the queue, the sub-operator being one of the H first sub-operators and the H second sub-operators; wherein the target data is deleted after the sub-operator completes the operation.

Images