TWI898651B - Memory device and operation mothod thereof - Google Patents

Abstract

A memory device and an operation method are provided. The memory device includes: an input data memory, a weight data memory, an output data memory, and a logic calculating circuit. The logic calculating circuit is coupled to the input data memory, the weight data memory, and the output data memory. The input data memory stores a plurality of input data. The weight data memory stores a plurality of weight values corresponding to the plurality of input data. The logic calculating circuit calculates an output data according to at least one input data and at least one weight data corresponding to the at least one input data and transmits the output data to the output data memory. The output data memory transmits the output data.

Description

Memory device and operating method thereof

The present invention relates to artificial intelligence technology, and more particularly to a memory and an operating method thereof.

When using a Large Language Model (LLM), data processing is typically performed using stacked static random access memory (SRAM) devices in conjunction with external arithmetic logic circuits. However, in LLMs with large data volumes, the access latency caused by communication between the stacked SRAM and the arithmetic logic circuits increases significantly, impacting overall performance.

In view of this, the present invention provides a memory device and an operating method thereof. By placing a logic operation circuit in the memory device, the communication transmission time between the logic operation circuit and the memory device is avoided, thereby improving the speed of executing data operations and reducing access latency.

The memory device of the present invention includes: an input data memory, a weight data memory, an output data memory, and a logical operation circuit. The logical operation circuit is coupled to the input data memory, the weight data memory, and the output data memory. The input data memory stores multiple input data. The weight data memory stores multiple weight values corresponding to the multiple input data. The logical operation unit calculates output data based on at least one input data and its corresponding at least one weight value, and stores the output data in the output data memory. The output data memory transmits the output data.

The operating method of the memory device of the present invention includes: storing multiple input data through an input data memory; storing multiple weight values corresponding to the multiple input data through a weight data memory; calculating output data based on at least one input data and its corresponding at least one weight value through a logical operation unit, and storing the output data in the output data memory; and transmitting the output data through the output data memory.

Based on the above, the memory and operating method provided by the present invention can avoid the communication transmission time between the logical operation circuit and the memory device by setting up a logical operation circuit in the memory device, significantly reducing access latency to meet the processing requirements of large language models (LLMs) with large data volumes.

In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

Some embodiments of the present invention are described in detail below with reference to the accompanying drawings. Reference symbols in the following description will identify identical or similar elements when the same symbols appear in different drawings. These embodiments are only a portion of the present invention and do not disclose all possible implementations of the present invention. Rather, these embodiments are merely examples within the scope of the present invention's patent application.

FIG1 is a schematic diagram of a memory device according to an embodiment of the present invention; FIG2 is a schematic diagram of the operation of a logic operation circuit according to an embodiment of the present invention. Please refer to FIG1 and FIG2 . In this embodiment, the memory device 10 may be, for example, a stacked static random access memory (SRAM). The memory device 100 includes an input data memory 110, a weight data memory 120, an output data memory 130, and a logic operation circuit 140. The logic operation circuit 140 is coupled to the input data memory 110, the weight data memory 120, and the output data memory 130. The input data memory 110 is used to store multiple input data. The weight data memory 120 is used to store multiple weight values corresponding to the multiple input data.

The logical operation unit 140 can calculate the output data Z based on the input data D1~DN and the corresponding weight values W1~WN, and store the output data Z to the output data memory 130. Specifically, the logical operation unit 140 includes a multiplier 141 and an accumulator 142. The multiplier 141 is coupled to the accumulator 142. As shown in Figure 2, the multiplier 141 can calculate multiple products of the input data D1~DN and the corresponding weight values W1~WN respectively. For example, the multiplier 141 can calculate the product Di*Wi of the input data Di and the weight value Wi, where i is any integer from 1 to N. Next, the accumulator 142 sums the products calculated by the multiplier 141 to calculate the output data Z. In other words, Z=D1*W1+ D1*W1+…+ DN*WN.

The logical operation unit 140 can store the output data Z in the output data memory 130. When the output data Z is needed later, the output data memory 130 can output the output data Z to the corresponding component. For example, the output data memory 130 can output the output data Z to the central processing unit (CPU) (not shown). In other words, the CPU can access the output data memory 130 to obtain the required data (i.e., the output data Z).

In this way, the memory device 10 of the present invention can perform data operations through the logic operation circuit 140 within the device to reduce access latency.

FIG3 is a schematic diagram of a memory device according to an embodiment of the present invention; FIG4 is a flow chart of an operating method of a memory device according to an embodiment of the present invention. Please refer to FIG3 and FIG4 . Memory device 30 includes an input data memory 310, a weight data memory 320, an output data memory 330, a logic operation circuit 340, a data read logic circuit 350, a control logic circuit 360, an input buffer 370, a weight buffer 380, an output buffer 390, an address bus BA, and a data bus BD. Logic operation circuit 340 includes a multiplier 341 and an accumulator 342. The data read logic circuit 350 includes an address generator 351 and a data extractor 352 .

In step S401, the input data memory 310 stores multiple entries of input data, and the weight data memory 320 stores multiple weight values corresponding to the multiple entries of input data. In step S402, the address generator 351 generates at least one first address A-D1-A-DN corresponding to at least one entry of input data D1-DN and at least one second address A-W1-A-WN corresponding to at least one weight value W1-WN of the at least one entry of input data D1-DN based on the address control signal CON-A.

Specifically, when multiple entries of input data are written into the input data memory 310, the address generator 351 may generate multiple addresses accordingly to ensure that each entry of input data written into the input data memory 310 has a unique address. Similarly, when multiple weight values corresponding to the multiple entries of input data are written into the weight data memory 320, the address generator 351 may generate multiple addresses accordingly to ensure that each weight value written into the weight data memory 320 has a unique address. In this way, the address generator 351 can generate at least one first address A-D1~A-DN and at least one second address A-W1~A-WN based on the address control signal CON-A generated by the control logic circuit 360 to indicate at least one input data D1~DN currently required and its corresponding at least one weight value W1~WN.

In step S403 , the address generator 351 transmits at least one first address A-D1-A-DN and at least one second address A-W1-A-WN to the input data memory 310 and the weight data memory 320 respectively via the address bus BA.

In step S404, data extractor 352 extracts at least one input data D1-DN and at least one weight value W1-WN from input data memory 310 and weight data memory 320, respectively, based on data control signal CON-D. Specifically, data control signal CON-D generated by control logic circuit 360 can be used to indicate the timing of data extraction. Data extractor 352 extracts input data D1-DN and weight values W1-WN based on data control signal CON-D to ensure accurate data reading.

Based on the above, the control logic circuit 360 can control the operations of the address generator 351 and the data extractor 352 via the address control signal CON-A and the data control signal CON-D, respectively, to ensure the data accuracy and timing relationship of the data read operation performed by the data read logic circuit 350.

In step S405, control logic circuit 360 determines whether data extractor 352 successfully retrieves the data. If so, the process proceeds to step S406. Otherwise, if data extractor 352 fails, the process returns to step S404, allowing data extractor 352 to re-retrieve input data D1-DN and weights W1-WN.

In one embodiment, control logic circuit 360 can determine whether data extraction by data extractor 352 is successful based on the first addresses A-D1-A-DN and the second addresses A-W1-A-WN. Specifically, control logic circuit 360 can verify whether the first addresses A-D1-A-DN and the second addresses A-W1-A-WN are within expected ranges. If so, the first addresses A-D1-A-DN and the second addresses A-W1-A-WN are correct, and data extractor 352 can successfully extract the corresponding input data D1-DN and weight values W1-WN. If they do not match, then one of the first addresses A-D1-A-DN and the second addresses A-W1-A-WN is an incorrect address. This means that the address generated by address generator 351 is incorrect, and data extractor 352 cannot successfully extract the corresponding input data D1-DN and/or weight values W1-WN. In other words, data extractor 352 has failed.

In one embodiment, the control logic circuit 360 can determine whether the data extractor 352 has successfully extracted the data based on the input data D1-DN and the weight values W1-WN. Specifically, the control logic circuit 360 can determine whether the input data D1-DN and the weight values W1-WN are valid data. For example, the control logic circuit 360 can check the valid flag bits of the input data D1-DN and the weight values W1-WN. If the valid flag bit is a logical value of 1, it indicates that the data is valid and the data extractor 352 has successfully extracted the data. If the valid flag bit is a logical value of 0, it indicates that the data is invalid and the data extractor 352 has failed to extract the data. In another embodiment, if the valid flag bit is a logical value of 0, it means that the data is valid data and the data extractor 352 has successfully extracted it. If the valid flag bit is a logical value of 1, it means that the data is invalid data and the data extractor 352 has failed to extract it.

In one embodiment, the control logic circuit 360 can determine whether the data extraction by the data extractor 352 is successful based on the performance of the input buffer 370 and the weight buffer 380. Specifically, the input buffer 370 and the weight buffer 380 need to have sufficient storage capacity and/or sufficient transmission rate to store and/or transmit the input data D1-DN and weight values W1-WN extracted by the data extractor 352. If the storage capacity and/or transmission rate of the input buffer 370 and the weight buffer 380 are insufficient, which may result in data delay or loss, the control logic circuit 360 may determine that the data acquisition by the data acquisition device 352 has failed. If the storage capacity and transmission rate of the input buffer 370 and the weight buffer 380 are sufficient, the control logic circuit 360 may determine that the data acquisition by the data acquisition device 352 has succeeded.

In one embodiment, the control logic circuit 360 can determine whether the data extractor 352 successfully extracts the data based on whether noise interference or transmission errors occur. If noise interference or transmission errors occur during the process of extracting the input data D1-DN and weight values W1-WN by the data extractor 352, the control logic circuit 360 can determine that the data extractor 352 has failed to extract the data. If noise interference or transmission errors do not occur during the process of extracting the input data D1-DN and weight values W1-WN by the data extractor 352, the control logic circuit 360 can determine that the data extractor 352 has successfully extracted the data.

In one embodiment, control logic circuit 360 can determine whether data acquisition by data extractor 352 is successful based on the correctness of address control signal CON-A and data control signal CON-D. Specifically, control logic circuit 360 can control the operations of address generator 351 and data extractor 352 using address control signal CON-A and data control signal CON-D, respectively, to ensure data accuracy and timing. Control logic circuit 360 can verify the correctness of address control signal CON-A and data control signal CON-D generated by it through testing methods such as simulation testing and verification testing, that is, to check whether address control signal CON-A and data control signal CON-D meet requirements. If the address control signal CON-A and/or the data control signal CON-D are incorrect, the control logic circuit 360 may determine that the data acquisition by the data acquisition device 352 has failed. If the address control signal CON-A and the data control signal CON-D are correct, the control logic circuit 360 may determine that the data acquisition by the data acquisition device 352 has succeeded.

In step S406, the data extractor 352 transmits at least one input data D1-DN and at least one weight value W1-WN to the input buffer 370 and the weight buffer 380 via the data bus BD. The input buffer 370 is used to temporarily store the input data D1-DN extracted from the input data memory 310. The weight buffer 380 is used to temporarily store the weight values W1-WN extracted from the weight data memory 320.

In step S407, the logic operation circuit 340 receives at least one input data D1-DN and at least one weight value W1-WN from the input buffer 370 and the weight buffer 380, respectively, and calculates the output data Z based on the at least one input data D1-DN and the at least one weight value W1-WN. Specifically, a register (not shown) of the input buffer 370 can load the input data D1-DN into a register (not shown) of the multiplier 341. Similarly, a register (not shown) of the weight buffer 380 can load the weight values W1-WN into the register of the multiplier 341. Next, multiplier 341 calculates multiple products of the input data D1-DN and their corresponding weights W1-WN. The register of multiplier 341 loads these products into the register of accumulator 342 (not shown). Accumulator 342 then sums these products to calculate output data Z.

In step S408, the logic operation circuit 340 outputs the output data Z to the output buffer 390. In step S409, the output data memory 330 transfers the output data Z from the output buffer 390. Specifically, a register (not shown) in the output buffer 390 can load the output data Z into a register (not shown) in the output data memory 330. The output data memory 330 can then transfer the output data Z to a corresponding component (e.g., a central processing unit).

In this way, the memory device 30 of the present invention can perform data operations through the logic circuit 340 within the device, thereby reducing access latency. Furthermore, the memory device 30 of the present invention can control the operation of the data read logic circuit 350 by controlling the address control signal CON-A and the data control signal CON-D generated by the control logic circuit 360 to ensure data accuracy and timing. Furthermore, the control logic circuit 360 can further determine whether the data read logic circuit 350 has successfully retrieved the data, thereby reconfirming the data's accuracy. Accordingly, the memory device 30 of the present invention can significantly reduce the communication transmission time between the logical operation circuit 340 and the input data memory 310, weight data memory 320, and output data memory 330, while ensuring the accuracy and timing of the data, and can meet the processing requirements of large language models (LLMs) with large amounts of data.

FIG5 is a flow chart showing an operating method of a memory device according to an embodiment of the present invention. The operating method can be implemented by the memory device 10 of FIG1 . Please refer to FIG1 and FIG5 . In step S501 , a plurality of input data are stored in the input data memory 110 . In step S502 , a plurality of weight values corresponding to the plurality of input data are stored in the weight data memory 120 . In step S503 , output data is calculated based on at least one input data and at least one weight value corresponding thereto by the logic operation unit 140 , and the output data is stored in the output data memory 130 . In step S504, the output data is transmitted via the output data memory 130.

In summary, the memory device and operating method provided by the present invention can significantly reduce the communication transmission time between the logic operation circuit and the input data memory, weight data memory, and output data memory, while ensuring the accuracy and timing of the data extracted by the data read logic circuit, thereby meeting the large data processing requirements of large language models (LLMs).

Although the present invention has been disclosed above by way of embodiments, they are not intended to limit the present invention. Any person having ordinary skill in the art may make slight modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope of the attached patent application.

10, 30: Memory 110, 310: Input Data Memory 120, 320: Weight Data Memory 130, 330: Output Data Memory 140, 340: Logical Operation Circuit 141, 341: Multiplier 142, 342: Accumulator 350: Data Read Logic Circuit 351: Address Generator 352: Data Extractor 360: Control Logic Circuit 370: Input Buffer 380: Weight Buffer 390: Output Buffer A-D1, A-DN, A-W1, A-WN: Address BA, BD: Buses CON-A, CON-D: Control signals D1, D2, Di, DN: Input data W1, W2, Wi, WN: Weight values S401, S402, S403, S404, S405, S406, S407, S408, S409, S501, S502, S503, S504: Steps Z: Output data

Figure 1 is a schematic diagram of a memory device according to an embodiment of the present invention. Figure 2 is a schematic diagram of the operation of a logic operation circuit according to an embodiment of the present invention. Figure 3 is a schematic diagram of a memory device according to an embodiment of the present invention. Figure 4 is a flow chart of a method for operating a memory device according to an embodiment of the present invention. Figure 5 is a flow chart of a method for operating a memory device according to an embodiment of the present invention.

10: Memory device

110: Input data memory

120: Weight data memory

130: Output data memory

140: Logical Operation Circuit

141: Multiplier

142: Accumulator

D1, DN: Input data

W1, WN: weight values

Z: Output data

Claims (14)

A memory device comprises: an input data memory for storing a plurality of input data; a weight data memory for storing a plurality of weight values corresponding to the input data; an output data memory; and a logic operation circuit coupled to the input data memory, the weight data memory, and the output data memory, wherein the logic operation unit calculates output data based on at least one input data and its corresponding at least one weight value, and stores the output data in the output data memory; the output data memory transmits the output data. The memory device further comprises: The data read logic circuit includes: an address generator for generating, based on an address control signal, at least one first address for the at least one input data and at least one second address corresponding to the at least one weight value of the at least one input data; and a data extractor for extracting, based on the data control signal, the at least one input data and the at least one weight value from the input data memory and the weight data memory, respectively. The memory device of claim 1, wherein the logic operation circuit comprises: a multiplier; and an accumulator coupled to the multiplier, wherein the multiplier calculates at least one product of the at least one input data and its corresponding at least one weight value, respectively, and the accumulator sums the at least one product to calculate the output data. The memory device of claim 1 further comprises: an address bus; and a data bus, wherein the address generator transmits the at least one first address and the at least one second address to the input data memory and the weight data memory, respectively, via the address bus, and the data extractor transmits the at least one input data and the at least one weight value via the data bus. The memory device of claim 3 further comprises: a control logic circuit configured to determine whether the data extractor has successfully extracted the data; in response to the control logic circuit determining that the data extractor has failed to extract the data, the data extractor re-extracts the at least one input data and the at least one weight value from the input data memory and the weight data memory, respectively. The memory device of claim 4, wherein: In response to the control logic circuit determining that the data acquisition is successful, the data acquisition transmits the at least one input data and the at least one weight value to the input buffer and the weight buffer, respectively, via the data bus. The memory device of claim 4, wherein the logic operation circuit receives the at least one input data and the at least one weight value from the input buffer and the weight buffer, respectively, to calculate the output data based on the at least one input data and the at least one weight value. The memory device of claim 4, wherein the control logic circuit determines whether the data extractor has successfully extracted the data based on the at least one first address and the at least one second address. The memory device of claim 4, wherein the control logic circuit determines whether the data extractor has successfully extracted the data based on the at least one input data and the at least one weight value. The memory device of claim 4, wherein the control logic circuit determines whether the data extractor is successful based on the performance of the input buffer and the weight buffer. The memory device of claim 4, wherein the control logic circuit determines whether the data acquisition is successful based on whether noise interference or transmission error occurs. The memory device of claim 4, wherein the control logic circuit determines whether the data extractor has successfully extracted the data based on the accuracy of the address control signal and the data control signal. The memory device of claim 1, wherein the logic operation circuit outputs the output data to an output buffer. The memory device of claim 1, wherein the memory device is a stacked static random access memory (Stacked SRAM). A method for operating a memory device comprises: storing a plurality of input data in an input data memory; storing a plurality of weight values corresponding to the input data in a weight data memory; calculating output data based on at least one input data and its corresponding at least one weight value in a logic operation unit, and storing the output data in the output data memory; transmitting the output data through the output data memory; generating at least one first address of the at least one input data and at least one second address corresponding to the at least one weight value of the at least one input data by an address generator of a data read logic circuit based on an address control signal; and The data acquirer of the data read logic circuit extracts the at least one input data and the at least one weight value from the input data memory and the weight data memory respectively based on the data control signal.