CN114037074A - A model pruning method, device, electronic device and storage medium - Google Patents

Abstract

The present disclosure provides a model pruning method, apparatus, electronic device, readable storage medium and computer program product, which relate to the fields of computer vision and deep learning. The specific implementation scheme is: according to the image feature vector matrix output by the target sublayer, determine the pruning mask corresponding to the multi-head self-attention module in the target sublayer, and the target sublayer is a sublayer of the visual converter model, The pruning mask is used to identify the self-attention head to be pruned in the multi-head self-attention module; the multi-head self-attention module is pruned by using the pruning mask. This scheme can determine the pruning mask used to identify the self-attention head to be pruned in the multi-head self-attention module, and use the pruning mask to perform the multi-head self-attention module of the target sub-layer of the visual converter model. Self-attention head pruning. Therefore, the resources required for deployment and computation of the vision converter model can be reduced.

Description

A model pruning method, device, electronic device and storage medium

technical field

The present disclosure relates to the field of artificial intelligence, and in particular to computer vision and deep learning technologies, which can be specifically used in scenarios such as computer vision and deep learning.

Background technique

The Transformer model is a deep neural network model mainly based on the self-attention mechanism, which was originally applied to Natural Language Processing (NLP). And because the Transformer model has a strong representation ability, the Transformer model has also been greatly developed in the field of computer vision in recent years.

Compared with the commonly used CNN-based models in the computer vision field, the deployment and operation of the Vision Transformer model generally consumes huge resources. Therefore, how to reduce the resources consumed by the deployment and operation of the Vision Transformer model has become an urgent problem to be solved in the application of the Vision Transformer model.

SUMMARY OF THE INVENTION

The present disclosure provides a model pruning method, apparatus, electronic device, readable storage medium, and computer program product, so as to reduce the resources consumed by the deployment and operation of the vision converter model.

According to an aspect of the present disclosure, a model pruning method is provided, and the method may include the following steps:

According to the image feature vector matrix output by the target sub-layer, determine the pruning mask corresponding to the multi-head self-attention module in the target sub-layer, the target sub-layer is the sub-layer of the visual converter model, and the pruning mask is used to identify the multi-head self-attention module. The self-attention head to be pruned in the attention module;

Using the pruning mask, the multi-head self-attention module is pruned.

According to a second aspect of the present disclosure, there is provided a model pruning device, which may include:

The pruning mask determination unit is used to determine the pruning mask corresponding to the multi-head self-attention module in the target sublayer according to the image feature vector matrix output by the target sublayer, and the target sublayer is the sublayer of the visual converter model, The pruning mask is used to identify the self-attention head to be pruned in the multi-head self-attention module;

The module pruning unit is used to prune the multi-head self-attention module using the pruning mask.

According to another aspect of the present disclosure, there is provided an electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method in any of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method in any of the embodiments of the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program/instruction, characterized in that, when the computer program/instruction is executed by a processor, the method in any of the embodiments of the present disclosure is implemented.

The technology of the present disclosure can determine the pruning mask for identifying the self-attention head to be pruned in the multi-head self-attention module, and use the pruning mask to perform multi-head self-attention on the target sub-layer of the visual converter model. The force module performs self-attention head pruning. Therefore, the resources required for deployment and computation of the vision converter model can be reduced.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:

1 provides a flowchart of a model pruning method in an embodiment of the present disclosure;

2 is a schematic structural diagram of a Vision Transformer model provided in an embodiment of the present disclosure;

3 is a schematic diagram of a self-attention head pruning process provided in an embodiment of the present disclosure;

4 is a flowchart of a method for determining a pruning mask provided in an embodiment of the present disclosure;

FIG. 5 provides a schematic diagram of a model pruning device according to an embodiment of the present disclosure;

FIG. 6 is a schematic diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

An embodiment of the present disclosure provides a method for model pruning. For details, please refer to FIG. 1 , which is a flowchart of a method for model pruning in an embodiment of the present disclosure. The method may include the following steps:

Step S101: According to the image feature vector matrix output by the target sublayer, determine the pruning mask corresponding to the multi-head self-attention module in the target sublayer, the target sublayer is the sublayer of the visual converter model, and the pruning mask is used for Identifies the self-attention head to be pruned in the multi-head self-attention module.

Step S102: Use the pruning mask to prune the multi-head self-attention module.

The model pruning method provided by the embodiments of the present disclosure can determine the pruning mask used to identify the self-attention head to be pruned in the multi-head self-attention module, and use the pruning mask to prune the visual converter model. The multi-head self-attention module of the target sublayer performs self-attention head pruning. Therefore, the resources required for deployment and computation of the vision converter model can be reduced.

For the Vision Transformer model, the model generally includes an image vector conversion layer, an encoding layer and a decoding layer (collectively referred to as Transformer Layer), and the encoding layer and the decoding layer respectively include a plurality of encoding sub-layers and decoding sub-layers. Specifically, the encoding sublayer is a sublayer in the encoding layer of the visual converter model, and the decoding sublayer is a sublayer in the decoding layer of the visual converter model.

The target object of the model pruning method provided by the embodiments of the present disclosure is any one or more encoding sublayers and decoding sublayers in the Vision Transformer model. That is, at least one sublayer may be arbitrarily selected as the target sublayer among the plurality of encoding sublayers and decoding sublayers. Specifically, the target sub-layer can be selectively determined from multiple encoding sub-layers and decoding sub-layers according to preset resource consumption and accuracy of model output required by the visual converter model.

The principle of pruning the self-attention heads in the multi-head self-attention module to reduce the resources consumed by the deployment and operation of the visual converter model will be described in detail below with reference to FIG. 2 . FIG. 2 is a schematic structural diagram of a Vision Transformer model provided in an embodiment of the present disclosure. It should be noted that only the image vector conversion layer and multiple coding sub-layers in the Visiontransformer model are shown in FIG. 2 .

The image vector conversion layer mainly performs linear transformation and/or pixel flattening arrangement on the input image, reshapes the input image into a vector, and inputs the vector into the coding layer; each coding sublayer in the coding layer corresponds to For an encoder, the encoder consists of a standard module, a multi-head attention module (Multi-Head Attention), and a multi-layer perceptron module (MLP, also called a multi-layer perceptron, generally has two layers).

In an actual usage scenario, an image is generally divided into multiple patches, and the resolutions of each patch are equal. Each small block corresponds to an input position of the model, and after passing through the image vector conversion layer, a feature vector equal to the number of small blocks is generated. Then the feature vector is input to multiple coding sub-layers in turn, and the feature vector output by the last coding layer will be input to the classifier, and then the classification result will be obtained. The classification result may be a probability value, for example, the probability of recognizing that the input image is a dog is 90%, and the probability of being a cat is 10%.

For a certain coding sublayer, the calculation amount of processing multiple patches at the same time can be derived from formulas (1)-(4). Specifically, formulas (1)-(3) respectively estimate the calculation process of the coding sublayer The calculation amount of the three main steps in the formula (4) represents the calculation amount of the entire coding sub-layer. Among them, N represents the number of input patches or the number of input feature vectors, and D is the size of an embedding dimension (embedding size/embedding dim), which is the head (also called self-attention) in the feature vector during the training process. The product of the number of force heads, a single self-attention computing head) and the dimension of each feature vector (dim, also known as the length of the feature vector), [N, D] represents a matrix of dimension (N, D), [D, D] represents a matrix whose dimension is (D, D). [N, D] and [N, N] are similar, and will not be repeated here.

4×([N, D]×[D, D])=>4ND ² ——(1)

[N,D]×[D,N]+[N,N]×[N,D]=>2N ² D——(2)

[N, D]×[D, 4D]+[N, 4D]×[4D, D]=>8ND ² ——(3)

12ND ² +2N ² D——(4)

From formulas (1)-(4), it can be known that by reducing the number of self-attention heads in the multi-head self-attention module, the calculation amount of the encoding sub-layer can be reduced. And for the multi-head self-attention module, there are differences in importance, for example, the image regions that the less important self-attention heads pay attention to are actually not useful for the final output. Therefore, pruning the less important self-attention heads can reduce the calculation amount of the encoding sub-layer under the premise of ensuring the accuracy of the output, and thus can reduce the consumption of the vision converter model deployment and operation. resource.

In addition, it can be known from equations (1)-(4) that for compressing the embedding dim (also called feature dim), the resources required for the deployment and operation of the visual converter model can also be reduced. However, for the multi-head self-attention module, each self-attention head has physical meaning. If only the dimension of embedding (vector) is reduced, it is very possible to prune the self-attention head with rich information. Therefore, pruning the multi-head self-attention module of the target sub-layer of the vision converter model can ensure the accuracy of the model output while reducing the resources consumed by the deployment and operation of the vision converter model. .

In order to accurately determine the self-attention head to be pruned, the pruning mask is obtained in the following manner in the embodiment of the present disclosure. For details, please refer to FIG. 3 , which is a A schematic diagram of the self-attention head pruning process.

Suppose an image is divided into 9 patches, and after the image vector conversion layer performs linear mapping or flattening operation (linear projection or flatten) on the 9 patches, 9 tokens are generated, and it is assumed that the size (scale) of the token (identity) ) is 384, and the matrix dimension composed of tokens is: [9, 384]. Input the token into the Transformer Layer. After the token passes through the Transformer Layer, the matrix dimension does not change until it reaches the final output layer. That is, the Transformer Layer outputs the corresponding image feature vector matrix. After Transformer Layer i (the i-th sublayer) outputs the corresponding image feature vector matrix, the pruning mask can be determined for the image feature vector matrix. The specific process is as follows:

First, a first vector matrix is obtained after the image feature matrix is separated in the color channel dimension. That is, the first vector matrix is a vector matrix obtained by performing a separation operation on the image feature matrix in the color channel dimension. Specifically, the matrix dimension of the image feature vector matrix is [9, 384], which is transformed into a vector matrix with the matrix dimension [9, 6, 64] after reshape (matrix transformation), and then the split (separation) operation is performed to obtain two The first vector matrix with matrix dimensions [9, 6, 32]. Specifically, N=9, H=6, and C=32 of the image feature vector matrix [N, H, C]. Among them, N represents the number of input patches or the number of input feature vectors, H represents the number of multi-attention heads in the multi-head self-attention module, and C represents the color channel of the image.

Second, the first vector matrix is averaged in the color channel dimension to obtain the second vector matrix and the third vector matrix. Specifically, the mean value operation is performed on the two eigenvector matrices to obtain two matrices with matrix dimensions [6, 32], which are the second vector matrix and the third vector matrix, respectively.

Third, take the mean value of the second vector matrix in the multi-head dimension corresponding to the multi-head self-attention module to obtain a fourth vector matrix. That is, a further mean value operation is performed on the mean value of the second vector matrix in the multi-head dimension to obtain a fourth vector matrix whose matrix dimension is [1, 32], and the fourth vector matrix is a vector globally.

Fourth, an expansion operation is performed on the fourth vector matrix to obtain a fifth vector matrix. Specifically, an expand (expand) operation is used to obtain a fifth vector matrix whose matrix dimension is [6, 32]. The specific implementation manner of the so-called expansion operation may be to perform a copy operation on the fourth vector matrix whose matrix dimension is [1, 32] to obtain a fifth vector matrix whose matrix dimension is [6, 32].

Fifth, the third vector matrix is spliced with the fifth vector matrix having the same dimension to obtain the sixth vector matrix. Specifically, a fifth vector matrix whose matrix dimension is [6, 32] is spliced with another third vector matrix whose matrix dimension is [6, 32] to obtain a sixth vector matrix whose matrix dimension is [6, 64] . The so-called splicing of matrices refers to column merging (left and right merging) for matrices with the same number of rows, and row merging (up and down merging) for matrices with the same number of columns. For example, for matrix a=np.array([1, 2]), matrix b=np.array([2,4]), perform left and right merging a=np.hstack((a,b)), get the output print( a)=[1, 2, 2, 4].

Sixthly, linearly change the sixth vector matrix in the multi-head dimension to obtain a seventh vector matrix with a matrix dimension of two. Specifically, the sixth vector matrix is input to a linear projection layer to obtain a seventh vector matrix whose matrix dimension is [6, 2].

Seventh, based on the seventh vector matrix, a pruning mask is determined.

Referring to FIG. 4 for a specific implementation manner of determining a pruning mask based on the seventh vector matrix in an embodiment of the present disclosure, FIG. 4 is a flowchart of a method for determining a pruning mask provided in an embodiment of the present disclosure.

Step S401: Normalize the seventh vector matrix to obtain distribution probabilities of the seventh vector matrix in different dimensions.

Step S402: According to the distribution probability, the vector of the specified dimension is used as the pruning mask.

In the embodiment of the present disclosure, according to the distribution probability, the vector of the specified dimension is used as the pruning mask, which can ensure that the mask matrix accurately identifies the self-attention head to be pruned in the multi-head self-attention module.

Referring to FIG. 3 again, in an embodiment of the present disclosure, a seventh vector matrix with a matrix dimension of [6, 2] can be passed through a normalization sublayer (log-softmax layer) to obtain a distribution probability of two dimensions. Among them, the dimension value of the multi-head dimension is 6 (H=6), and the dimension value of the other dimension is 2. After obtaining the distribution probability, for the probability distribution of the second dimension, the argsort operation can be used to obtain a vector of a specified dimension (subscript 1 in Figure 3) as a pruning mask.

In order to prune the self-attention heads with low importance to reduce the calculation amount of the target sub-layer on the premise of ensuring the accuracy of the output, in the embodiment of the present disclosure, after the pruning mask is determined, the The self-attention head is pruned in the following manner, specifically: first, the corresponding relationship between the mask value in the pruning mask and the self-attention head in the multi-head self-attention module is determined. Then, in the case where the mask with the mask value of zero is included in the pruning mask, based on the correspondence, the self-attention head corresponding to the mask with the mask value of zero is pruned. Among them, the mask value corresponds to each self-attention head in the multi-head self-attention module one-to-one.

Specifically, when the pruning mask is [1, 0, 1, 1, 1, 1], the second self-attention head in the multi-head self-attention module is pruned (combining the pruning mask with image feature vector matrix multiplication). If the pruning mask is [0, 1, 0, 1, 1, 1], the first and third self-attention heads in the multi-head self-attention module are pruned, and the rest of the self-attention heads are kept.

In order to conveniently and accurately determine the pruning mask, the embodiment of the present disclosure can input the image feature vector matrix into the mask derivation module to obtain the pruning mask. The mask derivation module is used in the training process of the visual converter model. A concurrently trained module for determining pruning masks.

The mask derivation module is a module that is trained synchronously when training the visual transformer model. Specifically, the training process of the visual converter model is as follows: input the sample images and corresponding annotations into the visual converter model to be trained, and obtain the output result of the visual converter model to be trained. According to the loss function value corresponding to the result and the annotation, the parameters in the visual converter model to be trained are continuously adjusted until the number of training times meets the preset number of times, thereby obtaining the visual converter model.

During the training phase of the visual converter model to be trained, the mask derivation module is also performing corresponding steps, and the parameters of the mask derivation module are constantly being adjusted. However, in the model training stage, in order to ensure the consistency of the data, the multi-head self-attention module will not be pruned, but the output of the self-attention head corresponding to the mask whose mask value is zero will be set to zero, such as The pruning mask is [0, 1, 0, 1, 1, 1], i.e. zeroing the outputs of the first and third self-attention heads in the multi-head self-attention module.

In addition, in the model training stage, after obtaining the distribution probability, instead of the probability distribution of the second dimension, the argsort operation is used to obtain a vector with a subscript of 1 as the pruning mask. Instead, first input the probability distribution of the second dimension into the Gumbel-softmax layer, turn the probability distribution into one-hot encoding, and then obtain the vector on channel 1 as the pruning mask.

As shown in FIG. 5 , an embodiment of the present disclosure provides a model pruning device, which includes:

The pruning mask determining unit 501 is used to determine the pruning mask corresponding to the multi-head self-attention module in the target sublayer according to the image feature vector matrix output by the target sublayer, and the target sublayer is the sublayer of the visual converter model , the pruning mask is used to identify the self-attention head to be pruned in the multi-head self-attention module;

The module pruning unit 502 is used for pruning the multi-head self-attention module by using the pruning mask.

In one embodiment, the pruning mask determining unit 501 may include:

The first vector-matrix processing subunit is used to average the first vector-matrix corresponding to the image feature vector matrix in the color channel dimension to obtain the second vector-matrix and the third vector-matrix, where the first vector-matrix is the image in the color channel dimension. The vector matrix obtained after the feature matrix is separated;

The second vector-matrix processing subunit is used to average the second vector-matrix in the multi-head dimension corresponding to the multi-head self-attention module to obtain the fourth vector-matrix;

The third vector-matrix processing subunit is used for splicing the third vector-matrix and the fifth vector-matrix with the same dimension to obtain a sixth vector-matrix, and the fifth vector-matrix is a vector-matrix obtained by extending the fourth vector-matrix ;

The fourth vector-matrix processing subunit is used to linearly change the sixth vector-matrix in the multi-head dimension to obtain the seventh vector-matrix with a matrix dimension of two;

The pruning mask determining subunit is used to determine the pruning mask based on the seventh vector matrix.

In one embodiment, the pruning mask determines the subunit, which may include:

The matrix normalization subunit is used to normalize the seventh vector matrix to obtain the distribution probability of the seventh vector matrix in different dimensions;

The pruning mask obtains the subunit, which is used to use the vector of the specified dimension as the pruning mask according to the distribution probability.

In one embodiment, the modular pruning unit 502 may include:

The correspondence determination subunit is used to determine the correspondence between the mask value in the pruning mask and the self-attention head in the multi-head self-attention module;

The module pruning subunit is used to prune the self-attention head corresponding to the mask with the mask value of zero based on the correspondence when the pruning mask includes a mask with a mask value of zero.

In one embodiment, the pruning mask determining unit 501 may include:

The mask derivation module subunit is used to input the image feature vector matrix into the mask derivation module to obtain the pruning mask. The mask derivation module is simultaneously trained during the training of the visual converter model and is used to determine the pruning mask. code module.

In one embodiment, the target sublayer includes at least one of an encoding sublayer and a decoding sublayer, the encoding sublayer is a sublayer in the encoding layer of the visual translator model, and the decoding sublayer is a sublayer of the visual translator model. Sublayers in the decoding layer.

In the technical solution of the present disclosure, the acquisition, storage and application of the user's personal information involved are all in compliance with the provisions of relevant laws and regulations, and do not violate public order and good customs.

According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

FIG. 6 shows a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 6 , the device 600 includes a computing unit 601 that can be executed according to a computer program stored in a read only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603 Various appropriate actions and handling. In the RAM 603, various programs and data necessary for the operation of the device 600 can also be stored. The computing unit 601 , the ROM 602 , and the RAM 603 are connected to each other through a bus 604 . An input/output (I/O) interface 605 is also connected to bus 604 .

Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc. ; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 609 allows the device 600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

Computing unit 601 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 601 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the model pruning method. For example, in some embodiments, the model pruning method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608 . In some embodiments, part or all of the computer program may be loaded and/or installed on device 600 via ROM 602 and/or communication unit 609 . When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the model pruning method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the model pruning method by any other suitable means (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. Such program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable model pruning device, such that the program codes, when executed by the processor or controller, perform the functions specified in the flowcharts and/or block diagrams /Operation is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, there is no limitation herein.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.

Claims (15)

1. A model pruning method, comprising: According to the image feature vector matrix output by the target sublayer, determine the pruning mask corresponding to the multi-head self-attention module in the target sublayer, the target sublayer is a sublayer of the visual converter model, and the pruning mask code is used to identify the self-attention head to be pruned in the multi-head self-attention module; Using the pruning mask, the multi-head self-attention module is pruned. 2. The method according to claim 1, wherein, determining the pruning mask corresponding to the multi-head self-attention module in the target sublayer according to the image feature vector matrix output by the target sublayer, comprising: The first vector matrix corresponding to the image feature vector matrix is averaged in the color channel dimension to obtain a second vector matrix and a third vector matrix, and the first vector matrix is to separate the image feature matrix in the color channel dimension. The vector matrix obtained by the operation; The second vector matrix is averaged in the multi-head dimension corresponding to the multi-head self-attention module to obtain a fourth vector matrix; The third vector matrix is spliced with the fifth vector matrix with the same dimension, and the sixth vector matrix is obtained, and the fifth vector matrix is the vector matrix obtained by expanding the fourth vector matrix; Linearly changing the sixth vector matrix in the multi-head dimension to obtain a seventh vector matrix with a matrix dimension of two; The pruning mask is determined based on the seventh vector matrix. 3. The method according to claim 2, wherein the determining the pruning mask based on the seventh vector matrix comprises: Normalizing the seventh vector matrix to obtain the distribution probabilities of the seventh vector matrix in different dimensions; According to the distribution probability, a vector of a specified dimension is used as the pruning mask. 4. The method according to any one of claims 1 to 3, wherein, using the pruning mask to prune the multi-head self-attention module, comprising: determining the correspondence between the mask value in the pruning mask and the self-attention head in the multi-head self-attention module; When the pruning mask includes a mask with a mask value of zero, based on the correspondence, the self-attention head corresponding to the mask with the mask value of zero is pruned. 5. The method according to claim 1, wherein, determining the pruning mask corresponding to the multi-head self-attention module in the target sublayer according to the image feature vector matrix output by the target sublayer, comprising: Inputting the image feature vector matrix into a mask derivation module to obtain the pruning mask, and the mask derivation module is simultaneously trained during the training process of the visual converter model for determining the pruning masked module. 6. The method of claim 1, wherein the target sublayer comprises at least one of an encoding sublayer and a decoding sublayer, the encoding sublayer being a sublayer in an encoding layer of the visual translator model , the decoding sublayer is a sublayer in the decoding layer of the visual converter model. 7. A model pruning device, comprising: A pruning mask determining unit, used for determining the pruning mask corresponding to the multi-head self-attention module in the target sublayer according to the image feature vector matrix output by the target sublayer, where the target sublayer is a visual converter model The pruning mask is used to identify the self-attention head to be pruned in the multi-head self-attention module; a module pruning unit for pruning the multi-head self-attention module by using the pruning mask. 8. The apparatus according to claim 7, wherein the pruning mask determining unit comprises: The first vector-matrix processing subunit is used to average the first vector-matrix corresponding to the image feature vector matrix in the color channel dimension to obtain the second vector-matrix and the third vector-matrix, where the first vector-matrix is in the color channel. The vector matrix obtained after the channel dimension performs the separation operation on the image feature matrix; A second vector-matrix processing subunit, configured to average the second vector-matrix in the multi-head dimension corresponding to the multi-head self-attention module to obtain a fourth vector-matrix; The third vector-matrix processing subunit is used for splicing the third vector-matrix and the fifth vector-matrix having the same dimension to obtain a sixth vector-matrix, where the fifth vector-matrix is the The vector matrix obtained by the expansion operation; a fourth vector-matrix processing subunit, used for linearly changing the sixth vector-matrix in the multi-head dimension to obtain a seventh vector-matrix with a matrix dimension of two; A pruning mask determining subunit, configured to determine the pruning mask based on the seventh vector matrix. 9. The apparatus according to claim 8, wherein the pruning mask determination subunit comprises: a matrix normalization subunit, used for normalizing the seventh vector matrix to obtain the distribution probabilities of the seventh vector matrix in different dimensions; The pruning mask obtaining subunit is used to use a vector of a specified dimension as the pruning mask according to the distribution probability. 10. The device according to any one of claims 7 to 9, wherein the modular pruning unit comprises: Correspondence determination subunit for determining the correspondence between the mask value in the pruning mask and the self-attention head in the multi-head self-attention module; The module pruning subunit is configured to, in the case that the pruning mask includes a mask with a mask value of zero, based on the correspondence, Attention head for pruning. 11. The apparatus according to claim 8, wherein the pruning mask determining unit comprises: A mask derivation module subunit, used for inputting the image feature vector matrix into a mask derivation module to obtain the pruning mask, and the mask derivation module is simultaneously in the training process of the visual converter model A trained module for determining the pruning mask. 12. The apparatus of claim 7, wherein the target sublayer comprises at least one of an encoding sublayer and a decoding sublayer, the encoding sublayer being a sublayer of an encoding layer of the visual translator model , the decoding sublayer is a sublayer in the decoding layer of the visual converter model. 13. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any one of claims 1 to 6 Methods. 14. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 6. 15. A computer program product comprising computer programs/instructions, characterized in that, when the computer program/instructions are executed by a processor, the steps of the methods of claims 1 to 6 are implemented.

Images