CN111814448A - Pre-trained language model quantization method and device - Google Patents

Abstract

The invention discloses a pre-training language model quantization method and device, wherein a pre-training language model quantization method includes: first fine-tuning the pre-training language model on downstream tasks; The data in the weight matrices of all embedding layers and all linear layers except the classification layer are clustered, and the number of categories is set to 2 ⁿ , where n is the number of bits occupied by each data of the target model after compression; The quantized model is fine-tuned a second time on the downstream task under the condition of maintaining quantization, and finally a quantized network is obtained. The solutions provided by the embodiments of the present application show that the impact of the improvement of the underlying quantization solution on the quantization effect is greatly underestimated and ignored; it also shows that a simple k-means quantization without any skills can achieve a very good compression effect , indicating that the k-means compression method has a very large development space and application prospects.

Description

Pre-trained language model quantization method and device

technical field

The invention belongs to the field of language model quantization, and in particular relates to a pre-training language model quantization method and device.

Background technique

In the prior art, there have been some quantization methods for pre-trained language models, including 8-bit specific precision quantization and Hessian matrix-based mixed-precision quantization.

8-bit specific precision quantization: All layers of the model that need to be quantized are quantized to 8 bits, and then fine-tuned.

Mixed-precision quantization based on Hessian matrix: Use the information of the Hessian matrix of the parameters of each layer to determine the quantization accuracy of the layer. The larger the Hessian matrix, the higher the quantization accuracy of the layer with the larger the eigenvalue, and the lower the vice versa. Fine-tune after quantization.

The bottom-most quantization scheme in the above two methods is linear quantization. That is to say, each tensor to be quantized separately adopts linear quantization: first find the maximum and minimum values of the parameters in the tensor, and then divide the range into several parts. Divided into 2 ⁿ parts, that is, 2 ⁿ categories. The average value of all parameters belonging to each class is taken as the central value of the class, and each parameter is replaced by the central value of the class to which it belongs. This tensor is then replaced by a tensor that stores the center value of each class and a tensor that stores the class to which each parameter belongs.

During the process of realizing the present application, the inventor found that the existing solution has at least the following defects:

The compression effect of linear quantization is not very good: the performance of the quantized model drops more at low precision, which makes the model cannot be compressed to very low precision.

Linear quantization is not a good clustering method. The quantized vector is not a good representation of the parametric distribution of the original vector.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a pre-trained language model quantization method and apparatus, which are used to solve at least one of the above technical problems.

In a first aspect, an embodiment of the present invention provides a method for quantifying a pre-trained language model, including: performing a first fine-tuning of the pre-training language model on a downstream task; The data in the weight matrices of all other embedding layers and all linear layers are clustered, and the number of categories is set to 2 ⁿ , where n is the number of bits occupied by each data of the compressed target model; and the quantized The model is fine-tuned a second time on the downstream task while maintaining quantization, resulting in a quantized network.

In a second aspect, an embodiment of the present invention provides a pre-trained language model quantization device, including: a first fine-tuning module configured to perform the first fine-tuning of the pre-trained language model on downstream tasks; a clustering compression module, configured as Using k-means clustering, cluster the data in the weight matrices of all embedding layers and all linear layers of the fine-tuned model except the classification layer, and set the number of categories to 2 ⁿ , where n is the compressed target The number of bits occupied by each data of the model; and a second fine-tuning module, configured to perform a second fine-tuning of the quantized model on the downstream task under the condition of maintaining quantization, and finally obtain a quantized network.

In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the pretrained language model quantization method of any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program is When the instructions are executed by a computer, the computer is made to execute the steps of the pre-trained language model quantization method according to any embodiment of the present invention.

The solution provided by the method and device of the present application shows that the impact of the improvement of the underlying quantization solution on the quantization effect is greatly underestimated and ignored; it also shows that simple k-means quantization without any skills can achieve very good results The compression effect shows that the k-means compression method has a very large development space and application prospect.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a method for quantizing a pre-trained language model according to an embodiment of the present invention;

2 is an algorithm diagram of k-means quantization of a pre-trained language model quantization method provided by an embodiment of the present invention;

3 is a comparison diagram of the average scores of 8 GLUE tasks that perform linear and k-means quantization on a BERT model according to a specific embodiment of a pre-trained language model quantization method provided by an embodiment of the present invention;

4 is a schematic diagram of comparing the average scores of 8 GLUE tasks linearly and k-means quantized on the ALBERT model according to a specific embodiment of the pre-trained language model quantization scheme according to the embodiment of the present invention;

5 is a comparison diagram of the average scores of 8 GLUE tasks with k-means quantized BERT and ALBERT models according to a specific embodiment of a pre-trained language model quantization scheme according to an embodiment of the present invention;

FIG. 6 is a performance comparison of the BERT and ALBERT models with k-means quantization according to a specific embodiment of the pre-trained language model quantization scheme according to the embodiment of the present invention, each value refers to the average score of the quantization model and the score of the full-precision model. The percentage diagram of the ratio;

7 is a block diagram of a pre-trained language model quantization apparatus according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 , which shows a flowchart of an embodiment of the pre-trained language model quantization method of the present application. The pre-trained language model quantization method of the present embodiment can be applied to the allocation processing of user requests. There is no limit to this.

As shown in Figure 1, in step 101, the pre-trained language model is fine-tuned for the first time on the downstream task;

In step 102, use k-means clustering to cluster the data in the weight matrices of all embedding layers and all linear layers of the fine-tuned model except the classification layer, and set the number of categories to 2 ⁿ , where, n is the number of bits occupied by each data of the target model after compression;

In step 103, the quantized model is fine-tuned for the second time on the downstream task under the condition of maintaining quantization, and finally a quantized network is obtained.

In this embodiment, for each selected task, the following experiments will be performed in sequence: fine-tuning the pre-trained models for downstream tasks (eg, BERT and ALBERT); quantizing task-specific models; fine-tuning quantized models. Then, the performance of the resulting model is tested on the validation set for each selected task.

To avoid the influence of other tricks, we apply only two quantization schemes (linear and k-means) following a fixed-precision quantization strategy without using any tricks. We quantify all weights of embedding layers and fully connected layers (except classification layers). For each weight vector, after quantization, will be represented by the corresponding cluster index vector and mean vector, and each parameter of the weight vector will be replaced by the mean of the cluster to which it belongs.

After the model is quantized, we fine-tune it on the corresponding downstream tasks while maintaining the quantization. For forward traversal, we reconstruct each quantization layer by its cluster index vector and mean vector. For the backward pass, we update the quantization parameters by training the mean vector while updating the rest of the parameters normally. More specifically, the gradient of each parameter in the mean vector is calculated as the mean of the gradients of the parameters belonging to the corresponding cluster. Then, the mean vector is updated by the same backpropagation method.

In some optional embodiments, using k-means clustering to cluster the data in the weight matrices of all other embedding layers and all linear layers of the fine-tuned model except the classification layer includes: using k- The means++ initialization divides the data into 2 ^k clusters (k here has the same meaning as n in the previous text), and initializes 2 ^k means for the 2 ^k clusters; according to the relationship between each data and each mean, the Each data is classified into the nearest cluster; after each data is classified, the corresponding mean value is updated to the mean value of all the data in the cluster; and each data is repeatedly reclassified and the mean value is updated, until Convergence is satisfied or the preset maximum number of iteration rounds is reached.

In some optional embodiments, using k-means++ initialization to divide the data into ^2k clusters, and initializing ^2k means for the ^2k clusters includes: selecting a random data as the first mean; assign the remaining data as the likelihood of the next mean based on the smallest distance from the existing mean, and select the next mean according to the likelihood of said next mean; and repeat the likelihood calculation and mean selection, until all 2 ^k -means are generated.

Further optionally, the preset maximum number of iteration rounds is set to 3.

In some other optional embodiments, when the quantized network performs forward calculation, the original weight matrix is restored by the stored category of each data and the mean value of each category, that is, each data uses the corresponding category The mean value of the quantized network is replaced; and the quantized network uses the gradient descent method to update the network parameters in the backward calculation, especially the quantized weight matrix, we average the gradients of the elements of the same category as the mean of the category The gradient of , updates each mean.

Further optionally, the pre-trained language model is BERT (Bidirectional Encoder Representation from Transformers) or ALBERT.

The following describes some problems encountered by the inventor in the process of implementing the present invention and a specific embodiment of the finalized solution, so that those skilled in the art can better understand the solution of the present application.

In order to improve the compression rate of the pre-trained language model or improve the performance of the compressed model, most of the existing work is mainly achieved by introducing other techniques, such as variable precision compression, grouping compression, etc. This method has limited improvement effect, or at the same time leads to The operation time is increased by dozens of times. The improvement brought by the improvement of the underlying quantization scheme has been greatly underestimated, so people rarely try in this regard.

By changing the underlying quantization scheme from linear clustering to k-means clustering, we greatly improve the rationality of grouping, thereby enabling the pre-trained language model to be compressed to less than 15% of its original size and still maintain the original Over 90% of model performance.

Specific steps are as follows:

1) Fine-tune the pre-trained language model on specific downstream tasks;

2) Use k-means clustering to cluster the data in the weight matrix of all embedding layers and linear layers of the model except the classification layer, and the number of categories is 2 ⁿ (where n is the value of each data of the compressed target model. The number of bits occupied), use the k-means++ initialization method to initialize, and the maximum number of iterations of the k-means method is set to 3;

3) The quantized model is fine-tuned again on the corresponding downstream tasks under the condition of maintaining quantization, and finally a quantized network is obtained.

In addition, when the quantized network performs forward calculation, the original weight matrix is restored by the stored category of each data and the mean of each category, that is, each data is replaced by the mean of the corresponding category; , using the gradient descent method to update the network parameters, especially the quantized weight matrix, we average the gradients of the elements of the same category, and update each mean as the gradient of the mean of the category.

This scheme shows that the impact of the improvement of the underlying quantization scheme on the quantization effect has been greatly underestimated and ignored; at the same time, it also shows that a simple k-means quantization without any techniques can achieve a very good compression effect, indicating that k-means compression The method has a very large development space and application prospect.

The inventor's process of implementing the embodiments of the present application, as well as some experimental procedures and corresponding experimental data in the process are described below, so that those skilled in the art can better understand the technical solutions of the present application.

Recently, pre-trained language models like BERT have shown excellent performance on a variety of natural language processing tasks. However, the application of these models is limited due to the huge space they require. A widely studied and effective way to reduce network size is quantization. However, most of the works focusing on BERT quantization use relatively rudimentary linear clustering methods as the quantization scheme, and few works attempt to improve the quantization scheme. This greatly limits the performance of quantization. In this paper, we implement k-means quantization and compare its performance with linear quantization on BERT's fixed-precision quantization. By comparison, we verify that the performance improvement effect of the improvement of the underlying quantization scheme is greatly underestimated, and k-means quantization has great development potential. Furthermore, we also compare the performance of the two quantization schemes on the ALBERT model to explore the difference in robustness to quantization between different pretrained models.

Keywords: K-means quantization, linear quantization, pretrained language models, GLUE.

1 Introduction

A pre-trained self-attention based model (Transformer) has recently achieved state-of-the-art performance on various natural language processing (NLP) tasks such as sequence labeling and sentence classification. Among them, the BERT model based on the Transformer architecture has attracted more attention due to its excellent performance and generality. However, the memory and computational consumption of these models is prohibitive. Even relatively small versions of BERT models (e.g., the BERT-base model) contain over 100 million parameters. Over-parameterized features make it challenging to deploy BERT models on resource-constrained devices such as smartphones and robots. Therefore, compressing these models is an important requirement of the industry.

A popular and effective method for model compression is quantization. To reduce the size of the model, quantization uses fewer bits than the original 32 bits to represent the parameters of the model. With proper hardware, quantization can greatly reduce memory footprint while speeding up computation. There is a lot of work in computer vision focused on quantized models, and much less work on NLP. Experimental work on Transformer quantization successfully quantizes Transformer models to 8 or 4 bits while maintaining comparable performance. But to our knowledge, there are only two published works on BERT quantization. One of the papers applies 8-bit fixed-precision linear quantization to a BERT model and achieves a 4x compression ratio with little accuracy drop. Another paper improves quantization performance by group-wise mixed-precision linear quantization of Hessian matrices based on parameter tensors.

However, for the underlying quantization scheme, most of the above Transformer quantization work, especially the BERT quantization work, utilizes linear clustering, which is a major clustering method. Although it can be processed quickly and easily, the quantized results are not a good representation of the original data distribution. Although some BERT quantization works achieve higher compression ratios without improving the quantization scheme, the method they developed for packet quantization is rather time-consuming and significantly increases latency. Although it is believed that replacing linear clustering with better clustering methods can improve the performance of quantization models, the effect of quantization scheme improvements has been underestimated. Therefore, in this paper, we explore the effect of simply improving the quantization scheme from linear clustering to k-means clustering, and compare the performance of the two schemes. Furthermore, to see the impact on other pre-trained language models, we also compared the above two quantization schemes for the ALBERT model, an improved model of BERT.

Overall, we apply k-means and linear quantization on BERT and ALBERT and test their performance on the GLUE task set. In this way, we verify that a simple improvement of the quantization scheme can lead to a great improvement in performance, and that simple k-means clustering has great potential as a BERT quantization scheme. Furthermore, we show that the number of k-means iteration rounds plays an important role in k-means quantization. Through further comparison, we find that ALBERT is not as robust as BERT in terms of quantization, because parameter sharing reduces parameter redundancy.

2 Background: BERT and ALBERT

In this section, we briefly describe the architecture of the BERT and ALBERT models and indicate the version of the model we used in our experiments.

2.1 BERT

The BERT model is a special Transformer-based pretrained network. They mainly consist of an embedding layer, an encoder block and an output layer. There is no decoder block in the BERT model. Each encoder block contains one self-attention layer (including three parallel linear layers corresponding to query, key, and value) and 3 feedforward layers (each containing one linear layer).

(h是每个自我注意层中的头数)。令

表示相应的自注意力层的输入。因此，自注意力头的输出计算如下：For each self-attention layer, BERT utilizes a multi-head technique to further improve its performance. For each self-attention head, there are 3 weight matrices W _q , W _k and W _v , where W _q , W _k , W _v

(h is the number of heads in each self-attention layer). make

represents the input of the corresponding self-attention layer. Therefore, the output of the self-attention head is calculated as follows:

Then, for each self-attention layer, the outputs of all its self-attention heads are sequentially concatenated to generate the output of the corresponding layer.

Specifically, in our work, we perform the following experiments using a bert-base-uncased version of the BERT model with 12 encoder blocks and 12 self-attention heads per self-attention layer.

2.2 ALBERT

ALBERT makes three major improvements compared to BERT. First, the ALBERT model decomposes the parameters of the embedding layer into the product of two smaller matrices. Second, they employ cross-layer parameter sharing to improve parameter efficiency. These two improvements can significantly reduce the total number of parameters and make the model more efficient. In addition, parameter sharing can also stabilize network parameters. Third, they replaced the next-sentence prediction (NSP, next-sentence prediction) loss with sentence-order prediction (SOP, sentence-order prediction) loss during pre-training. This allows the model to focus on modeling inter-sentence coherence rather than topic prediction, and improves performance on multi-sentence encoding tasks.

Specifically, in this paper, we use the albert-base-v2 version of the ALBERT model, which also has 12 encoder blocks (all parameters are shared between layers) and 12 self-attentions per self-attention layer Power head.

3 Methodology

In this section, we first introduce the quantification process in our experiments and then explain the two quantification schemes we use in detail.

3.1 Overview

To compare linear and k-means quantization schemes on Transformer-based pretrained models, we test the performance of the quantized models on different downstream tasks. Specifically, for each selected task, the following experiments will be performed in sequence: fine-tuning the pre-trained models (BERT and ALBERT) on downstream tasks; quantizing task-specific models; fine-tuning quantized models. Then, the performance of the resulting model is tested on the validation set for each selected task.

To avoid the influence of other tricks, we apply only two quantization schemes (linear and k-means) following a fixed-precision quantization strategy without using any tricks. We quantify all weights of embedding layers and fully connected layers (except classification layers). For each weight vector, after quantization, will be represented by the corresponding cluster index vector and mean vector, and each parameter of the weight vector will be replaced by the mean of the cluster to which it belongs.

After the model is quantized, we fine-tune it on the corresponding downstream tasks while maintaining the quantization. For forward traversal, we reconstruct each quantization layer by its cluster index vector and mean vector. For the backward pass, we update the quantization parameters by training the mean vector while updating the rest of the parameters normally. More specifically, the gradient of each parameter in the mean vector is calculated as the mean of the gradients of the parameters belonging to the corresponding cluster. Then, the mean vector is updated by the same backpropagation method.

3.2 Linear Quantization

Suppose we need to quantize the vector v into k bits (k-bit quantization). We first search for its minimum value v _min and maximum value v _max . The range [v _min , v _max ] is then divided into ^2k clusters:

Define the function Q^ as

Its value is between 0 and 2k-1. Thus each parameter v _i belongs to the Q^(vi ) _th cluster. v _i will be replaced by the mean of the Q _^ (vi )th cluster, i.e. the mean of all parameters belonging to it. Therefore, the quantization function is:

where 1{statement} is equal to 1 when the statement in the braces is true, and 0 otherwise.

3.3 K-Means Quantization

Suppose we need to quantize the vector v into k bits (k-bit quantization). For k-means quantization, we utilize k-means clustering and k-means++ initialization to divide the vector v into ^2k clusters.

We first initialize the mean (μ_1, μ_2,..., ^μ_2k ) for each cluster (c_1, c_2, ..., c_2k) using the ^k -means++ initialization method. Then, classify each parameter _vi to its nearest cluster. After classifying all parameters in v, the mean is updated to the mean of all parameters belonging to them respectively. Then, iteratively reclassifies the parameters and updates the mean until the convergence conditions are met or the maximum number of iteration rounds is reached. Furthermore, the process of the k-means++ initialization method is as follows: first, a random parameter from the vector v is chosen as the first mean; then other parameters are assigned as the likelihood of the next mean according to the minimum distance from all existing means, and according to These possibilities select the next mean; finally, the likelihood calculation and mean selection are repeated until all ^2k centroids are generated. Please refer to Figure 2 for the specific algorithm.

To reduce the efficiency drop due to the improved quantization scheme, we set the maximum number of iteration rounds for k-means clustering to 3. After k-means clustering is completed, we use the resulting cluster number vector as the cluster index vector, and the mean of each cluster as the corresponding mean vector. Each parameter _vi will be replaced by the mean of the cluster to which it belongs.

4 experiments

In this section, we first introduce the dataset we used in our experiments, then explain the details of our experiments on BERT and ALBERT, and finally present our experimental results and the corresponding discussion.

4.1 Dataset

We test the performance of our quantized models on the General Language Understanding Evaluation (GLUE) task set. These include NLU tasks such as question answering, sentiment analysis, and text entailment. Specifically, we utilize 8 tasks (QNLI, CoLA, RTE, SST-2, MRPC, STS-B, MNLI and QQP) to test the performance of different quantization schemes. The evaluation metrics for each task are as follows: CoLA is the Matthews correlation coefficient (mcc); QNLI, RTE, SST2 and MNLI are the correct rate (acc); MRPC and QQP are the correct rate (acc) and F1 score; STS _- B is the Pearson and Spearman correlation coefficient (corr). We follow the default partition of the dataset. The dataset can be downloaded here: https://gluebenchmark.com/tasks.

4.2 Experimental Details

Before quantization, we fine-tune the bert-base-uncased version of the BERT model on 8 tasks using the Adam optimizer (initial learning rate 5e-5, linear update). For the ALBERT model, we first fine-tune the albert-base-v2 model on QNLI, CoLA, SST-2, MNLI and QQP. Then fine-tune on RTE, MRPC and STSB based on the fine-tuning results of MNLI. We fine-tune ALBERT using the linearly updated Adam optimizer and search for the initial learning rate for each task in {1e-5, 2e-5, 3e-5, 4e-5, 5e-5}.

Table 1. Fixed-precision linear quantization results of BERT on the GLUE task set.

Table 2. Fixed-precision k-means quantization results for BERT on the GLUE task set.

Table 3. Fixed-precision linear quantization results of ALBERT on the GLUE task set.

Table 4. Fixed-precision k-means quantization results for ALBERT on the GLUE task set.

Figure 3 shows a comparison of the mean scores on 8 GLUE tasks with linear and k-means quantization on the BERT model.

Figure 4 shows the average scores for 8 GLUE tasks comparing linear and k-means quantization on the ALBERT model.

After quantization, we further fine-tune the quantized model for the corresponding task. In particular, the learning rates of the quantized layers are multiplied by a factor of 10 (e.g., 5e-4 for all quantized BERT models), while the learning rates of other layers remain the same.

4.3 Experimental results and discussion

We mainly focus on 1-5 bit fixed precision quantization. Tables 1 and 2 show the results of linear and k-means quantification of BERT, respectively, and Figure 3 shows a further comparison between the mean scores of the two sets of experiments. Similarly, the results and comparisons of ALBERT are shown in Table 3, Table 4 and Figure 4, respectively.

4.3.1 BERT

The improvement brought by the improvement of the quantification scheme. As shown in Table 1, Table 2, and Figure 3, although the model performs worse at lower number of bits regardless of the quantization scheme, when using the same number of bits, it performs better on all 8 tasks and its average , the model quantized with k-means performs significantly better than the model using linear quantization. Looking at the average performance of the 8 tasks, just by improving the quantization scheme from linear mean to k-means, we can achieve a performance drop of 1-5 bit quantization compared to full precision respectively from (38.8%, 34.7%, 27.6%, 17.1%, 4.8%) decreased to (28.6%, 3.94%, 0.9%, 0.3%, -0.2%). The results show that large performance improvements can be achieved by improving the quantization scheme alone, suggesting that the room for improvement of the quantization scheme is greatly underestimated. To further illustrate this, we repeated several experiments using a grouped linear quantization scheme, which is an improvement on linear quantization and has higher performance than simple linear quantization. The results are shown in Table 5. Compared to the performance of grouped linear quantization, simple k-means quantization can achieve higher or comparable performance while saving a lot of time.

Potential for k-means quantization. As shown in Table 2, the model can be compressed well simply using k-means quantization with a fixed-precision strategy, and the quantized model still compresses well even at some particularly low quantization bits. For example, on task RTE, a model quantized to 3 bits using k-means quantization results in only a 2.16% performance drop. For most tasks, including QNLI, SST-2, MRPC, STS-B, MNLI, and QQP, the performance of the quantized model only drops significantly when compressed to 1 bit. It is worth noting that these results are achieved with simple k-means quantization with a maximum number of iterations of only 3, without using any other tricks, suggesting that k-means quantization has great potential for development.

Table 5. Comparison between k-means quantization and grouped linear quantization on BERT. The rightmost column is the average elapsed time for k-means quantization compared to grouped linear quantization on RTE and MRPC. (wherein, in group quantization, each matrix is divided into different groups, and each group is quantized separately. For forward traversal, the model needs to reconstruct each quantized group for each layer separately, rather than directly Reconstruct the entire weight matrix for each quantization layer. This explains why group quantization is time-consuming. Specifically, in our group quantization experiments, we divide each matrix into 128 groups.)

4.3.2 ALBERT

In general, two main conclusions from BERT experiments still hold. As shown in Table 3, Table 4 and Figure 4, we can also see the huge improvement brought by the improvement of the quantization scheme and the huge potential of k-means quantization. However, some unusual results are worth discussing.

The effect of the number of k-means iteration rounds. The first set of abnormal results comes from 1-bit quantification of QNLI, MRPC and STS-B. Although the results of k-means quantization are generally better than linear quantization, these three sets of results do not conform to this law. We believe that this is because the distribution of parameters is too complex for k-means to get good clustering results with only 3 iterations. To test the theory and further explore the effect of the number of iterations, we repeated the experiments on these anomalous results while expanding the maximum number of iterations to 5, 10, and 20. The corresponding results are shown in Table 6. The higher the number of iterations, the better the k-means quantization works, and eventually surpasses the results of linear quantization. However, the problem of overfitting still exists, and the quantization performance of both QNLI and STS-B decreases significantly when the maximum number of iterations is increased from 10 to 20. Therefore, in k-means quantization, the k-means maximum number of iterations is also an important hyperparameter that needs to be carefully searched.

Table 6. 1-bit quantization performance with different maximum number of iteration rounds for k-means on ALBERT.

Figure 5 shows a comparison of the mean scores for 8 GLUE tasks for the k-means quantized BERT and ALBERT models.

Figure 6 shows the performance comparison of the k-means quantized BERT and ALBERT models. Each value refers to the percentage of the quantized model's average score compared to the full-precision model's score.

0 for CoLA and 68.4 for MRPC. Another set of anomalous results are linear quantifications from CoLA and MRPC, which are binary classification tasks. We found that after fine-tuning, the quantized model always outputs "1". 0 and 68.4 are only determined by the data distribution on the validation set. In other words, after quantizing the model to 1-5 bits by linear quantization, the model almost fails and it is difficult to train on both tasks. In addition, we further experimented with quantizing the model to higher bits on two tasks, and found that the performance of the quantized model is no longer 0 and 68.4 from quantization to 6 bits.

Comparison between BERT and ALBERT. Furthermore, we compare the performance of k-means quantization for BERT and ALBERT, and the results are shown in Figures 5 and 6. Compared to BERTBERT which still maintains 96.1% of its original performance after k-means 2-bit quantization, ALBERT’s performance has dropped to 93.4% and 72.5% after k-means 4-bit and 3-bit quantization, respectively. ALBERT is therefore less robust in terms of quantization (in our work, robustness to quantization means the ability to quantize to lower number of bits while maintaining high performance). Considering that the main improvement of ALBERT of BERT is parameter sharing, and quantization can also be regarded as intra-layer parameter sharing, we speculate that parameter sharing and quantization have similar effects, which means that the redundant information removed by parameter sharing and quantization is partially overlapping. Considering that, after parameter sharing, compared with BERT, ALBERT removes a lot of redundant information (the total number of parameters drops from 108M to 12M), therefore, further application of quantization on ALBERT will easily damage useful information, resulting in ALBERT pair Quantization is less robust. However, from another point of view, parameter sharing has greatly reduced the number of parameters, so it can also be regarded as a model compression method. Furthermore, given that full-precision ALBERT outperforms 4-bit and 3-bit BERT models that occupy similar memory in the GPU, parameter sharing, even better compression performance can be achieved compared to quantization without any tricks . However, as a compression method, parameter sharing has non-negligible disadvantages: it can only reduce memory consumption, while most other compression methods can reduce both memory consumption and computational consumption (i.e. time overhead).

5 Conclusion

In this paper, we compare k-means quantization and linear quantization on BERT and ALBERT models and draw three main conclusions. First, we found that models quantized with k-means performed significantly better than models using linear quantization. Huge performance gains can be achieved simply by improving the underlying quantization scheme. Second, even when using a simple fixed-precision compression strategy and without any other tricks, using k-means quantization can compress the model to a relatively low number of bits while maintaining high performance. This shows that k-means quantization has great potential for development. Third, the number of iterative rounds of k-means plays an important role in quantifying the performance of the model and should be determined carefully. Furthermore, by comparing the k-means quantization results of BERT and ALBERT, we find that ALBERT is less robust to quantization than BERT. This suggests that parameter sharing and quantization have some similar effects. Therefore, further application of quantization on models that apply extensive parameter sharing will more easily corrupt useful information, resulting in significant performance degradation.

Please refer to FIG. 7 , which shows a block diagram of a pre-trained language model quantization apparatus according to an embodiment of the present invention.

As shown in FIG. 7 , the pre-trained language model quantization apparatus 700 includes a first fine-tuning module 710 , a clustering compression module 720 and a second fine-tuning module 730 .

The first fine-tuning module 710 is configured to fine-tune the pre-trained language model on the downstream task for the first time; the clustering and compression module 720 is configured to use k-means clustering to perform fine-tuning on the fine-tuned model except for the classification layer. The data in the weight matrices of all other embedding layers and all linear layers are clustered, and the number of categories is set to 2 ⁿ , where n is the number of bits occupied by each data of the compressed target model; and the second fine-tuning module 730, is configured to perform a second fine-tuning of the quantized model on the downstream task under the condition of maintaining quantization, and finally obtain a quantized network.

In some optional embodiments, the above clustering compression module 720 is further configured to: use k-means++ initialization to divide the data into ^2k clusters, and initialize ^2k means for the ^2k clusters; For the relationship between each data and each mean, classify each data into the nearest cluster; after each data is classified, update the corresponding mean to the average of all the data in the cluster; and repeat the process for each data The data are reclassified and the centroids are updated until convergence is met or a preset number of iterations is reached.

It should be understood that the modules recited in FIG. 7 correspond to various steps in the method described with reference to FIG. 1 . Therefore, the operations and features described above with respect to the method and the corresponding technical effects are also applicable to the modules in FIG. 7 , and details are not repeated here.

It should be noted that the modules in the embodiments of the present application are not used to limit the solution of the present application, for example, the receiving module may be described as a module that receives a voice recognition request. In addition, the relevant functional modules may also be implemented by a hardware processor, for example, the receiving module may also be implemented by a processor, which will not be repeated here.

In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the pre-training in any of the foregoing method embodiments Language model quantification methods;

As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

Perform the first fine-tuning of the pretrained language model on the downstream task;

Using k-means clustering, cluster the data in the weight matrices of all embedding layers and all linear layers of the fine-tuned model except the classification layer, and set the number of categories to 2 ⁿ , where n is the compressed target The number of bits occupied by each data of the model;

The quantized model is fine-tuned a second time on the downstream task under the condition of maintaining quantization, and finally a quantized network is obtained.

The non-volatile computer-readable storage medium can include a stored program area and a stored data area, wherein the stored program area can store an operating system and an application program required by at least one function; the stored data area can store a quantization device according to a pre-trained language model using the created data, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-transitory computer-readable storage medium may optionally include memory located remotely from the processor, the remote memory being connectable to the pretrained language model quantization apparatus through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is made to execute the above Any pretrained language model quantization method.

FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 8 , the device includes: one or more processors 810 and a memory 820 . In FIG. 8 , one processor 810 is used as an example. The apparatus for pre-training the language model quantization method may further include: an input device 830 and an output device 840 . The processor 810, the memory 820, the input device 830, and the output device 840 may be connected through a bus or in other ways, and the connection through a bus is taken as an example in FIG. 8 . The memory 820 is the aforementioned non-volatile computer-readable storage medium. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, to implement the pre-trained language model quantization method in the above method embodiment. The input device 830 may receive input numerical or character information, and generate key signal input related to user settings and function control of the pre-trained language model quantization device. The output device 840 may include a display device such as a display screen.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

As an embodiment, the above-mentioned electronic equipment is applied to a pre-trained language model quantization device, including:

at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

Perform the first fine-tuning of the pretrained language model on the downstream task;

Using k-means clustering, cluster the data in the weight matrices of all embedding layers and all linear layers of the fine-tuned model except the classification layer, and set the number of categories to 2 ⁿ , where n is the compressed target The number of bits occupied by each data of the model;

The quantized model is fine-tuned a second time on the downstream task under the condition of maintaining quantization, and finally a quantized network is obtained.

The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones (eg iPhone), multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as iPads.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players (eg iPod), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of the server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, the processing power, stability , reliability, security, scalability, manageability and other aspects of high requirements.

(5) Other electronic devices with data interaction function.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of quantizing a pre-trained language model, comprising:

carrying out first fine adjustment on a pre-training language model on a downstream task;

clustering the data in the weight matrixes of all other embedded layers and all linear layers of the trimmed model except the classification layer by using k-means clustering, and setting the class number to be 2ⁿWherein n is the bit number occupied by each data of the compressed target model;

and carrying out secondary fine adjustment on the quantized model on the downstream task under the condition of maintaining quantization to finally obtain a quantized network.

2. The method of claim 1, wherein the clustering the data in the weight matrices of all embedded layers and all linear layers of the trimmed model except the classification layer using k-means clustering comprises:

partitioning the data into 2 with k-means + + initialization^kAnd is said 2^kIndividual cluster initialization 2^kAn average value;

classifying each data into a nearest cluster according to the relation between each data and each mean value;

after each data is classified, updating the corresponding average value to the average value of all data of the cluster where the corresponding average value is located;

and repeatedly reclassifying each data and updating the mean value until convergence is met or the preset maximum iteration round number is reached.

3. The method of claim 2, wherein said partitioning said data into 2 using k-means + + initialization^kAnd is said 2^kIndividual cluster initialization 2^kThe mean values include:

selecting a random data from said data as a first mean;

distributing the residual data as the possibility of the next mean value according to the minimum distance from the existing mean value, and selecting the next mean value according to the possibility of the next mean value;

repeat likelihood calculation and mean selection until all 2 s are generated^kAnd (4) average value.

4. The method of claim 2, wherein the preset maximum number of iterations is 3.

5. The method according to claim 1, wherein the quantized network restores the original weight matrix by the stored category of each data and the mean value of each category when performing forward calculation, that is, each data is replaced by the mean value of the corresponding category;

when the quantized network is calculated backwards, a gradient descent method is used for updating network parameters, particularly quantized weight matrixes are used, the gradients of elements in the same category are averaged, and the average is used as the gradient of the average of the category to update each average.

6. The method of any of claims 1-5, wherein the pre-trained language model is BERT or ALBERT.

7. A pre-trained language model quantification apparatus comprising:

the first fine tuning module is configured to perform first fine tuning on the pre-training language model on a downstream task;

a clustering compression module configured to cluster data in the weight matrix of all embedded layers and all linear layers of the trimmed model except the classification layer by using k-means clustering, and set the category number to be 2ⁿWherein n is the bit number occupied by each data of the compressed target model;

and the second fine tuning module is configured to perform second fine tuning on the quantized model on the downstream task under the condition of maintaining quantization, so as to finally obtain a quantized network.

8. The apparatus of claim 1, wherein the cluster compression module is further configured to:

partitioning the data into 2 with k-means + + initialization^kAnd is said 2^kIndividual cluster initialization 2^kAn average value;

classifying each data into a nearest cluster according to the relation between each data and each mean value;

after each data is classified, updating the corresponding average value to the average value of all data of the cluster where the corresponding average value is located;

and repeatedly reclassifying each data and updating the mean value until convergence is met or the preset maximum iteration round number is reached.

9. A computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, cause the computer to perform the steps of the pre-trained language model quantification method of any one of claims 1-6.

10. An electronic device, comprising: at least one processor, and a memory communicatively coupled to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method of any one of claims 1 to 6.

Images