CN118349814A - Optimization method, computing system and storage medium for large model auxiliary verification - Google Patents

Abstract

The invention discloses an optimization method, a computing system and a storage medium for large model auxiliary verification, wherein the method comprises the following steps: the first large model M ₁ generates an output one by one based on the input, the output comprising n+1 semantic units; the second large model M ₂ processes the input and output of the first large model M ₁ to obtain the output of the second large model M ₂ so as to verify the correctness of the n+1st semantic unit output by the first large model M ₁; wherein the scale of the first large model M ₁ is smaller than the scale of the second large model M ₂. The method adopts a smaller model to generate semantic units one by one, and uses a large model to verify the generated semantic unit sequence; the method can give consideration to efficiency and accuracy, and can effectively improve the parallelism and the calculation efficiency of the large language model.

Description

A large model-assisted verification optimization method, computing system and storage medium

Technical Field

The present invention relates to the field of artificial intelligence, and in particular to an optimization method, a computing system and a storage medium for large model-assisted verification.

Background technique

The generative large language model is an autoregressive model that has good results in many fields such as machine translation and intelligent customer service. The model is divided into two stages: prefill and decode.

The input of the pre-filling stage is a matrix obtained by embedding the input text. After being processed by the model, a vector corresponding to the embedding of the first semantic unit (token) can be obtained. This vector will be used as the input of the decoding stage. After being processed by the model, a vector corresponding to the embedding of the second semantic unit (token) is obtained. Therefore, it is obvious that the large language model adopts the method of generating semantic units one by one. It can be concluded that the model needs to be executed k times to generate k semantic units. Therefore, the performance of the large language model is not only affected by the accuracy of the model, but also by the speed of generating the corresponding semantic units. The data shows that, assuming that the length of the input semantic unit is 2k, for the Llama2-7B model, the decode stage is executed on the A100 GPU, and each semantic unit is generated for about 15ms. For the Llama2-70B model, the decode stage is executed on the A100 GPU, and the number of semantic units generated each time is about 35ms, which is about twice the delay of the 7B model.

At present, the main methods for accelerating the reasoning of large language models are:

1. Model compression. Prune the model, that is, make it sparse, quantize the weights and activations, and distill the model. This method will bring better results, but it essentially changes the model, so the output results will change.

2. Operator acceleration: Optimizing the model’s computational flow. Although this method does not change the results, the improvement is limited.

3. Model-assisted verification: Use a large-parameter model to optimize the output of a small-parameter model. Ideally, this approach can achieve the inference results of a large model using the inference time of a small model.

However, the system is not well optimized for large model-assisted verification.

Summary of the invention

In view of the above-mentioned defects of the prior art, the present invention provides an optimization method, a computing system and a storage medium for large-model assisted verification. The optimization method for large-model assisted verification generates semantic units one by one by using a small-scale model, and verifies the generated semantic units by using a large-scale model, so as to improve the computational parallelism of large-model reasoning. To achieve the above technical objectives, the present invention provides:

A large model-assisted verification optimization method, comprising:

The first large model _M1 generates outputs one by one based on the inputs, wherein the outputs include n+1 semantic units;

The second largest model M ₂ processes the input and output of the first largest model M ₁ to obtain the output of the second largest model M ₂ to verify the correctness of the n+1th semantic unit output by the first largest model M ₁ ; wherein the scale of the first largest model M ₁ is smaller than that of the second largest model M ₂ .

A further improvement of the present invention is:

The input of the first large model M ₁ in the pre-filling stage of generating n+1 semantic units includes a matrix X formed by the combination of L semantic unit groups; the matrix formed by the combination of the embedding vectors a ₁ , a ₂ , a ₃ …… a _n corresponding to the first n semantic units output by the first large model M ₁ is Y;

The processing of the input and output of the first largest model M ₁ by the second largest model M ₂ includes a pre-filling stage and a decoding stage; in the pre-filling stage, the input of the second largest model M ₂ is a matrix formed by combining the matrix X and the matrix Y in the row direction; in the decoding stage, the input of the second largest model M ₂ is the matrix Y. In this stage, the second largest model M ₂ uses a lower triangular mask when calculating the matrix P = QK ^T , where Q is the query matrix and K is the key matrix.

A further improvement of the present invention is that the first large model _M1 and the second large model _M2 both include a plurality of converter modules, and the converter modules include an attention module and a feedforward network.

A further improvement of the present invention is that the types of the first large model _M1 include Llama2-7B; and the types of the second large model _M2 include Llama2-70B.

The present invention also provides a computing system, the computing system comprising:

a memory for storing computer-executable instructions; and

A processor is used to execute the computer executable instructions to implement the above-mentioned optimization method for large model assisted verification.

A further improvement of the present invention is that the processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

The present invention also provides a computer-readable storage medium, which stores executable instructions. When a computer executes the executable instructions, it can implement the optimization method for large model-assisted verification according to any one of claims 1-4.

The technical solution provided by the present invention has the following technical effects: using a smaller model to generate semantic units one by one, and using a large model to verify the generated semantic unit sequence; this method can take into account both efficiency and accuracy, and can effectively improve the parallelism and computational efficiency of the large language model.

The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of an optimization method for large model-assisted verification in the present invention;

FIG. 2 is a schematic diagram of a lower triangle mask used in the present invention.

Detailed ways

The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

As shown in FIG1 , an embodiment of the present invention provides an optimization method for large model-assisted verification, including:

The first large model M ₁ generates n+1 semantic units one by one; the embedding vectors corresponding to the first n semantic units are a ₁ , a ₂ , a ₃ …… a _n respectively; the input of the first large model M ₁ in the pre-filling stage of generating n+1 semantic units includes a matrix X formed by the combination of L semantic unit groups; the embedding vectors a ₁ , a ₂ , a ₃ …… a _n corresponding to the first n semantic units output by the first large model M ₁ are formed by the combination of Y;

The second largest model M ₂ processes the input and output of the first largest model M ₁ to obtain an output to verify the correctness of the n+1th semantic unit output by the first largest model M _1. The processing of the input and output of the first largest model M ₁ by the second largest model M ₂ includes a pre-filling stage and a decoding stage; in the pre-filling stage, the input of the second largest model M ₂ is a matrix formed by combining the matrix X and the matrix Y in the row direction; in the decoding stage, the input of the second largest model M ₂ is the matrix Y, and in this stage, the second largest model M ₂ uses a lower triangular mask when calculating the matrix P = QK ^T.

In this embodiment, the type of the first large model _M1 is Llama2-7B, and the type of the second large model _M2 is Llama2-70B. The scale and running time of the first large model _M1 are much smaller than those of the second large model _M2 .

In this embodiment, the first large model _M1 generates 3 to 4 semantic units and verifies them once using the second large model _M2 ; if the semantic units generated by the second large model _M2 are consistent with those of the first large model _M1 , it means that the verification is passed.

The following is an introduction to the correctness of the verification method with the attached drawings:

As shown in Figure 1, the input of the first large model _M1 in the pre-filling stage of generating n+1 semantic units includes a matrix X formed by the combination of L semantic unit groups; it generates n+1 semantic units in turn; the embedding vectors _a1 , _a2 , _a3 ... _an corresponding to the first n semantic units are combined to form a matrix Y; the dimension of the matrix L is L×dim; where dim is the dimension of the embedding vector.

In this embodiment, the first large model _M1 and the second large model _M2 both include several transformer modules, and the transformer modules include attention modules and feed-forward networks. The large model calculation process involves two calculation operations, namely, attention mechanism calculation and matrix multiplication of input and weight.

1. Correctness of matrix multiplication of input and weight

First, verify the correctness of the matrix multiplication of the input and the weight. Assume that the input is a matrix A ₀ composed of n embedding vectors a ₁ , a ₂ , a ₃ … a _n , and the expression of its multiplication with the weight matrix is:

y ₁ = a ₁ W ^T

y ₂ = a ₂ W ^T

…

y _n =a _n W ^T

Y＝A ₀ W ^T

Y is essentially composed of y ₁ ,......,y _n , that is:

Therefore, the results of the two methods are the same: the vectors are fed into the matrix multiplication calculation one by one and the matrix multiplication calculation is composed of all vectors. Therefore, the correctness of the matrix multiplication of input and weight is verified.

2. Correctness of Attention Mechanism Calculation

The following will verify the correctness of the attention mechanism calculation from the prefill stage and the decoding stage respectively.

In the prefill phase, M ₂ takes as input the matrix Y consisting of X and a ₁ , a ₂ , a ₃ …… a _n , and expands in the row direction of X to a size of (L+n)×dim. Based on the correctness of the matrix multiplication with the weights explained above, the corresponding Q, K, and V are correct and have a size of (L+n)×dim. Next, Q is multiplied by K.

P＝QK ^T

The size of the result P is (L+n)×(L+n), but due to the existence of the lower triangle mask, P is also composed of the correct results, that is:

Where P _x is the result of the calculation with input X, and P ₁ , ..., P _n , are the results of the calculation with input a ₁ , a ₂ , a ₃ ... _an . The corresponding O = PV is essentially matrix multiplication, so the correctness remains unchanged.

In the decoding stage, M ₂ inputs a matrix Y composed of a ₁ , a ₂ , a ₃ …… a _n . It is worth noting that there is no mask in the decoding stage, so in the calculation process of this embodiment, a mask needs to be added manually. The size of Q, K, V corresponding to Y is n×dim. After splicing with KV_cache (key-value cache), the size of K and V is (L+n)×dim, where Q is the query matrix, K is the key matrix, and V is the value matrix. Therefore, in the process of calculating P, a mask needs to be added manually to ensure the correctness of P.

P＝QK ^T

The size of P is n×(L+n). For an n×n square matrix such as P[0:n-1,L:L+n-1], a lower triangular mask is needed to correct the result. As shown in FIG2 , the lower triangular matrix below the oblique dotted line represents the mask, and this application does not limit the specific value of the mask.

An embodiment of the present invention further provides a computing system, the computing system comprising:

a memory for storing computer-executable instructions; and

A processor is used to execute the computer executable instructions to implement the above-mentioned large model-assisted verification system optimization method. The processor includes a graphics processing unit GPU, a neural network processing unit NPU and a field programmable gate array FPGA chip.

An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores executable instructions, and when a computer executes the executable instructions, the above-mentioned system optimization method for large model-assisted verification can be implemented.

The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims (7)

1. An optimization method for large model aided verification, comprising the steps of:

The first large model M ₁ generates an output one by one based on the input, the output comprising n+1 semantic units;

The second large model M ₂ processes the input and output of the first large model M ₁ to obtain the output of the second large model M ₂ so as to verify the correctness of the n+1st semantic unit output by the first large model M ₁; wherein the scale of the first large model M ₁ is smaller than the scale of the second large model M ₂.

2. The optimization method for large model aided verification of claim 1, wherein:

The input of the first large model M ₁ in the pre-filling stage for generating n+1 semantic units comprises a matrix X formed by combining L semantic unit groups; the matrix formed by combining the embedded vectors a ₁,a₂,a₃......a_n corresponding to the first n semantic units output by the first large model M ₁ is Y;

The processing of the input and output of the second large model M ₂ to the first large model M ₁ includes a pre-fill stage and a decode stage; in the pre-filling stage, the input of the second large model M ₂ is a matrix formed by combining a matrix X and a matrix Y in the row direction; the input of the second large model M ₂ is matrix Y during the decoding phase, at which stage the second large model M ₂ uses a lower triangular mask in computing the matrix p=qk ^T, where Q is the query matrix and K is the key matrix.

3. The optimization method for large model aided verification of claim 1, wherein: the first large model M ₁ and the second large model M ₂ each include a number of transducer modules including an attention module and a feed forward network.

4. The optimization method for large model aided verification of claim 1, wherein: the categories of the first large model M ₁ include L1ama2-7B; the second large model M ₂ includes Llama2-70B.

5. A computing system, the computing system comprising:

a memory for storing computer-executable instructions; and

A processor for executing the computer-executable instructions to implement the optimization method of large model-assisted verification according to any one of claims 1-4.

6. The computing system of claim 5, wherein the processor comprises a graphics processing unit GPU, a neural network processing unit NPU, and a field programmable gate array FPGA chip.

7. A computer readable storage medium storing executable instructions that when executed by a computer enable the optimization method of large model-assisted verification according to any one of claims 1-4.