CN114819159A - Inference method, device, equipment and storage medium of deep learning model - Google Patents

Abstract

The application relates to an inference method, an inference device, inference equipment and a storage medium of a deep learning model, wherein the inference method comprises the following steps: acquiring a data set to be inferred; inputting the data set to be inferred into an inference convolution kernel to obtain a floating point type data inference result; acquiring an output scaling factor corresponding to the data set to be inferred; wherein the output scaling factor is determined from a maximum of an element in the data inference result; calculating the product of the data reasoning result and the output scaling factor to obtain a quantization result; and continuing the reasoning of the deep learning model according to the quantification result. The method and the device are used for solving the technical problem that in the model performance optimization process of the prior art, the integral error is large due to quantization.

Description

Inference method, device, device and storage medium for deep learning model

technical field

The present application relates to the technical field of deep learning networks, and in particular, to a reasoning method, apparatus, device and storage medium for a deep learning model.

Background technique

At present, deep learning has been widely used in all walks of life, and has achieved great results in areas that are difficult to solve by traditional algorithms. However, a problem with the current promotion of deep learning applications is that its operating costs are huge. Even when the computing power of GPUs (graphics processing units, graphics processing units) has been greatly improved, the deep learning models whose parameters have been increased year by year are also Eat up the bonus of GPU performance improvement. Therefore, the performance optimization of the model itself is a key to whether deep learning can be applied in the large-scale production process.

Quantization is one of the methods in the model performance optimization process. A key to quantization is to convert floating-point float32 precision input to integer int8 form. This conversion process will introduce corresponding errors. Most of today's solutions cannot adjust the errors in real time, resulting in a large overall error, which affects the final product order.

SUMMARY OF THE INVENTION

The present application provides an inference method, device, device and storage medium for a deep learning model, which are used to solve the technical problem that quantization leads to a large overall error in the process of model performance optimization in the prior art.

In a first aspect, the present application provides an inference method for a deep learning model, including:

Get the data set to be reasoned;

Inputting the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result;

Obtain the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result;

Calculate the product of the data inference result and the output scaling factor to obtain a quantization result;

Continue the inference of the deep learning model according to the quantization result.

Optionally, the obtaining the output scaling factor corresponding to the data set to be inferred includes:

From each element of the data inference result, determine the largest element;

The quotient of the preset value and the largest element is calculated to obtain the output scaling factor; wherein, the preset value is the upper limit of the value range corresponding to the data type of the data set to be inferred.

Optionally, inputting the data set to be inferred into an inference convolution kernel to obtain a floating-point data inference result, including:

obtaining the first scaling factor corresponding to the data set to be inferred;

obtaining the weight corresponding to the inference convolution kernel and the second scaling factor corresponding to the weight;

obtain the bias of the inference convolution kernel;

The floating-point data inference result is calculated according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the offset.

Optionally, calculating the floating-point data inference result according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the offset, including :

Calculate the product of the data set to be inferred and the weight to obtain a first intermediate result;

Dividing the first intermediate result by the first scaling factor and the second scaling factor to obtain the second intermediate result;

Calculate the sum of the second intermediate result and the offset to obtain the floating-point data inference result.

Optionally, the obtaining the first scaling factor corresponding to the data set to be inferred includes:

obtaining a floating-point input data set corresponding to the data set to be inferred;

The first scaling factor is obtained by dividing the data set to be inferred by the floating-point input data set.

Optionally, the obtained data set to be reasoned includes:

From the target video, determine the data set to be inferred corresponding to the current video frame;

The obtaining the output scaling factor corresponding to the data set to be inferred includes:

Judging whether scene switching occurs in the current video frame relative to the previous video frame, and obtaining a judgment result;

According to the judgment result, the output scaling factor corresponding to the data set to be inferred is determined.

Optionally, determining the output scaling factor corresponding to the data set to be inferred according to the judgment result includes:

If the judgment result indicates that scene switching occurs, determine the maximum element from the elements of the data inference result; calculate the quotient of the preset value and the maximum element to obtain the output scaling factor; wherein the preset value Set the value to the upper limit of the value range corresponding to the data type of the data set to be inferred;

If the judgment result indicates that no scene switching has occurred, the output scaling factor corresponding to the last input data set of the data set to be inferred is used as the output scaling factor of the data set to be inferred.

In a second aspect, the present application provides an inference device for a deep learning model, including:

The first acquisition module is used to acquire the data set to be reasoned;

a first inference module, configured to input the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result;

a second obtaining module, configured to obtain an output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result;

a quantization module for calculating the product of the data inference result and the output scaling factor to obtain a quantization result;

The second reasoning module is configured to continue the reasoning of the deep learning model according to the quantization result.

In a third aspect, the present application provides an electronic device, including: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory is used to store a computer program; the The processor is configured to execute the program stored in the memory to implement the inference method of the deep learning model described in the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the inference method of the deep learning model described in the first aspect is implemented.

Compared with the prior art, the above technical solutions provided by the embodiments of the present application have the following advantages: the method provided by the embodiments of the present application can obtain output scaling factors matching the data sets to be inferred for different data sets to be inferred , the output scaling factor can be dynamically generated according to the input data set to be inferred during the actual application, and then multiplied with the floating-point data inference result obtained by inputting the data set to be inferred to the inference convolution kernel, which can achieve int8 integer quantization The balance between speed and quality, that is, the method provided by the embodiments of the present application can not only improve the inference efficiency of the deep learning network model, but also ensure the inference quality, and can effectively alleviate the problem of relatively large overall errors.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

1 is a schematic diagram of mapping floating-point data to integer data according to an embodiment of the present application;

2 is a schematic flowchart of an inference method for a deep learning model provided by an embodiment of the present application;

3 is a schematic flowchart of a method for obtaining a floating-point data inference result provided by an embodiment of the present application;

4 is a schematic structural diagram of an int8 integer convolution kernel provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an inference apparatus for a deep learning model provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application.

The term "and/or" in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship.

In the training process of deep learning, for the overall target performance, the current mainstream method is to use floating-point float32 data for training, where float represents the floating-point data type, and the number 32 after float represents the number of bits occupied. However, specific to the deployment of the model, for production efficiency, methods such as int8 integer quantization will be applied to speed up inference; where int represents an integer data type, and the number 8 after int represents the number of bits occupied. As for the floating-point to integer mapping, the mainstream method is to use a scaling factor, as shown in Figure 1, multiply the floating-point data by this scaling factor, thereby expanding to [-128, 127] this integer interval, and then use the rounding method to further round the amplified floating-point data to obtain the final integer value of int8 type, for example: floating-point data-0.5 mapped integer of int8 type The value of the type is -128; the integer value of the int8 type after the floating point data 1.5 mapping is 128; the integer value of the int8 type after the floating point data 0 mapping is -Z; the int8 after the floating point data SZ mapping The integer value of the type is 0.

Subsequently, the integer value of type int8 is used as input and input into the inference convolution kernel (inferenceconv kernel) for inference to improve the inference efficiency of the model.

However, in practical applications, in order to ensure the overall speed, although the input data type is mapped to the int8 type, in the inference convolution kernel, in order to ensure a certain level of accuracy, in the inference process of the inference convolution kernel. , for example, in the addition operation after multiplication and the offset addition operation, the floating-point data result is still reserved, and in order to ensure that the output result is int8 type, it is necessary to infer the floating-point data in the result. On the basis, after the relu function and the round operation, if the value is between 0 and 1, after the round operation, the result is 0. Therefore, in the method provided by the prior art, in order to improve the speed, the accuracy is sacrificed, resulting in a large overall error.

In order to solve the problem of large overall error caused by quantization in the process of model performance optimization in the prior art, an embodiment of the present application provides an inference method for a deep learning model, and the inference method for a deep learning model can be applied to a server or a cloud ; wherein, the server or cloud stores a deep learning network model, and the deep learning network model includes an inference convolution kernel.

As shown in FIG. 2 , the inference method of a deep learning model provided by an embodiment of the present application specifically includes the following steps:

Step 201, obtaining a data set to be reasoned;

The data set to be inferred may be a matrix mapping from floating-point data to integer data types. For the specific method of mapping from floating point data to integer data type, reference may be made to the mapping method in the above-mentioned prior art. On the basis of each floating-point element of the floating-point input matrix, multiply the first scaling factor to map each floating-point element into an integer element to obtain an integer-type matrix, which is used as the data set to be inferred. Among them, the data set to be inferred is the input of the inference convolution kernel.

For example, taking the int8 integer convolution kernel running on the NVIDIA GPU as an example, the components of the convolution kernel include: input (activation), weight (weight) and bias (bias), where the input (activation) ) is the dataset to be reasoned in practical applications.

Step 202, input the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result;

Specifically, as shown in Figure 3, the method for inputting the data set to be inferred into the inference convolution kernel to obtain the floating-point data inference result mainly includes the following steps:

Step 301, obtaining the first scaling factor corresponding to the data set to be inferred;

The actual input data set is of floating point type. In order to improve the inference speed, the floating point input data set is converted into an input data set of integer type as the data set to be inferred. Multiply the floating-point input data set by the first scaling factor to obtain the data set to be inferred. Conversely, when the floating-point input data set corresponding to the data set to be inferred is known, divide the data set to be inferred by the data set to be inferred. Taking an input dataset of floats, get the first scaling factor.

Step 302, obtaining the weight corresponding to the inference convolution kernel and the second scaling factor corresponding to the weight;

Each inference convolution kernel corresponds to a weight, and the weight is usually constant for a certain inference convolution kernel, and the second scaling factor corresponding to the weight is also fixed.

Step 303, obtaining the bias of the inference convolution kernel;

Bias is also an important parameter in the inference convolution kernel. For a certain inference convolution kernel, the bias is also fixed.

Step 304: Calculate a floating-point data inference result according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor, and the offset.

Specifically, the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the offset are known, and the floating-point data inference result is calculated according to the operation inside the inference convolution kernel. Taking the int8 integer convolution kernel running on the NVIDIA GPU as an example, according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the offset, calculate the floating-point data inference results, including :

Calculate the product of the data set to be inferred and the weight to obtain the first intermediate result; divide the first intermediate result by the first scaling factor and the second scaling factor to obtain the second intermediate result; calculate the sum of the second intermediate result and the offset , and get the floating-point data inference result.

Assuming that the input is _{I Q} , the weight is _{W Q} , the first scaling factor corresponding to the input is SI , the second scaling factor corresponding to the weight is SW , and the bias is B, then the floating-point data inference result is I _Q * W _Q /S _I /S _W +B, at this time, the output inference result is a floating-point type.

Step 203, obtaining the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result;

Specifically, the largest element is determined from each element of the data inference result of the data set to be inferred; the quotient of the preset value and the largest element is calculated to obtain the output scaling factor.

In specific implementation, the preset value can be set to 127. Taking the input data set as the input matrix as an example, the data inference result is also a matrix. If the maximum value of each matrix element in the data inference result is MAX _output , the output scaling factor is So=127/MAX _output .

Calculating the output scaling factor with the maximum value of the matrix elements ensures that other matrix elements can also meet the needs.

In the embodiment of the present application, the output scaling factor matches the data set to be inferred, instead of multiplexing the same coefficient for each inference as in the prior art. In the embodiment of the present application, the floating-point data inference result is remapped to the int8 type through the output scaling factor, which not only ensures the inference efficiency of the deep learning model, but also ensures the inference quality.

Step 204, calculating the product of the data inference result and the output scaling factor to obtain a quantization result;

On the basis of the floating-point data inference result, multiply the output scaling factor to map the floating-point output result to the int8 type. Data of type int8 is faster to reason about than type float32.

Step 205, continue the reasoning of the deep learning model according to the quantization result.

Among them, the subsequent reasoning process of different deep learning models may be different. For example, the subsequent reasoning process can be through the relu function and the round operation.

In order to highlight the difference between the inference method of the deep learning model provided by the embodiment of the present application and the prior art, the inference method of the deep learning model provided by the prior art is described as follows:

Take a dataset and use this dataset to finetune the entire quantization process to get a coefficient that performs well on that dataset. Then when the model is deployed, this coefficient is reused for each inference. However, the problem with this approach is that the data encountered during the actual model inference process may be very different in data distribution from the data set used for fine-tuning. If this happens, the coefficients obtained from the data set used for fine-tuning can no longer be used, and if they are used forcibly, it will directly affect the quality of inference.

However, in this embodiment of the present application, for different data sets to be inferred, an output scaling factor matching the data set to be inferred can be obtained, and an output scaling factor can be dynamically generated according to the input data set to be inferred during actual application, and then the output scaling factor can be dynamically generated. Multiplying the floating-point data inference result obtained by inputting the data set to be inferred into the inference convolution kernel can achieve a balance between int8 integer quantization speed and quality, that is, the method provided by the embodiment of the present application can not only improve the deep learning network The inference efficiency of the model can be guaranteed, and the inference quality can be ensured, which can effectively alleviate the problem of relatively large overall errors.

In order to facilitate understanding of the inventive concept of the embodiments of the present application, an int8 integer convolution kernel running on an NVIDIA GPU is also taken as an example. For the deep learning network model, the components of the convolution kernel include: input (activation), weight (weight) and bias (bias), where the input (activation) is the data set to be reasoned in practical applications , the target input matrix. In order to ensure the overall speed and a considerable level of precision, in the implementation of nvidia's int8 kernel, the floating-point float form is retained in the final and offset addition process.

Since the _inputs and weights are of type int8, they each have a scaling factor, defined here as SI (first scaling factor) and SW (second scaling factor), respectively, while for inputs and weights in _int8 form , defined as I _Q and W _Q respectively, the offset in floating point form is defined as B, and for the final output in int8 form, it also has a corresponding scaling factor (output scaling factor), defined as S _O . After the reasoning process described in Figure 4, the output in the form of int8 can be obtained. For the inference process of the convolution kernel, it can be expressed as: (I _Q *W _Q /S _I /S _W +B)*S _O (1); it can be expanded into: I _Q *W _Q *S _O /S _I /S _W +B*S _O (2).

However, the scaling factor corresponding to the final output cannot be known in advance. Then here is another way of thinking. If the output is not in the form of int8, but in the form of floating point, the scaling factor of the output there is no longer needed, and it becomes possible to dynamically calculate the scaling factor.

In order to achieve this goal, first set So as 1, then formula (1) is expressed as:

I _Q *W _Q /S _I /S _W +B(3)

The result obtained by formula (3) is regarded as the floating-point data inference result. Among them, the scaling factor SW corresponding to the weight is unchanged for the _int8 integer convolution kernel, and at the same time, the input corresponding scaling factor _SI can be calculated.

After the floating-point data inference result is obtained, the output scaling factor is obtained by determining the maximum matrix element in each matrix element, and calculating the quotient of the preset value and the maximum matrix element.

In specific implementation, the preset value can be set to 127. Assuming that the maximum value of each matrix element in the data inference result is MAX _output , the output scaling factor is So=127/MAX _output .

In the actual application process, for the same video, when the characteristics of the input data input before and after do not change much (for example, the color changes little, the style is similar), the output scaling factor can continue to be reused, without the need to re-use each inference. To recalculate the scaling factor, this greatly reduces the time required for inference.

In the actual application process, in order to further improve the accuracy, it is possible to determine whether scene switching occurs according to the two frames of video before and after the video, so as to determine whether to multiplex the output scaling factor.

Specifically, if a scene switch occurs, the largest element is determined from each element of the data inference result; the quotient of the preset value and the largest element is calculated to obtain the output scaling factor; wherein, the preset value is the data of the data set to be inferred The upper limit of the value range corresponding to the type; for example, if the value range of the int data type is -128 to 127, the default value is 127; if no scene switching occurs, the last input of the data set to be inferred The output scaling factor corresponding to the dataset is used as the output scaling factor of the dataset to be inferred. The last input data set is an input data set determined according to the previous frame of the current video frame.

In addition, it should be noted that, in the specific implementation, each frame of video corresponds to an input matrix, the input matrix is the input data set, the input matrix can be a matrix characterizing the characteristics of the video frame, and the input matrix The data type is converted to the data type, and the converted result is used as the input of the inference convolution kernel.

The method for judging whether a scene switching occurs according to two adjacent video frames mentioned in the embodiments of the present application may adopt any method of judging whether a scene switching occurs in the prior art, which is not limited here.

In the embodiment of the present application, the output scaling factor is determined according to whether the scene is switched. If the scene is switched, the scaling factor is re-determined according to the data set to be inferred; if the scene is not switched, in order to minimize the time used for inference, Just reuse the scaling factor of the previous input dataset.

Based on the same concept, an inference device for a deep learning model is provided in an embodiment of the present application. For the specific implementation of the device, reference may be made to the description in the method embodiment section, and repeated descriptions will not be repeated. As shown in Figure 5, the device mainly includes:

The first acquisition module 501 is used to acquire the data set to be reasoned;

The first inference module 502 is configured to input the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result;

The second obtaining module 503 is configured to obtain the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result;

a quantization module 504, configured to calculate the product of the data inference result and the output scaling factor to obtain a quantization result;

The second reasoning module 505 is configured to continue the reasoning of the deep learning model according to the quantization result.

In this embodiment of the present application, for different data sets to be inferred, an output scaling factor matching the data set to be inferred can be obtained, and an output scaling factor can be dynamically generated according to the input data set to be inferred during actual application, and then combined with the data set to be inferred. The data set to be inferred is multiplied by the floating-point data inference results obtained by the inference convolution kernel, which can achieve a balance between int8 integer quantization speed and quality, that is, the method provided by the embodiment of the present application can not only improve the deep learning network model The inference efficiency can be guaranteed, and the inference quality can be guaranteed, which can effectively alleviate the problem of relatively large overall errors.

In a specific embodiment, the second obtaining module 503 is configured to determine the largest element from each element of the data inference result of the data set to be inferred; calculate the quotient of the preset value and the largest element, Get the output scaling factor.

In a specific embodiment, the first inference module 502 is configured to acquire a first scaling factor corresponding to the data set to be inferred; acquire a weight corresponding to the inference convolution kernel and a second scaling factor corresponding to the weight ; Obtain the bias of the inference convolution kernel; calculate the floating point type according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the bias data inference results.

In a specific embodiment, the first inference module 502 is configured to calculate the product of the data set to be inferred and the weight to obtain a first intermediate result; divide the first intermediate result by the first scaling factor and the second scaling factor to obtain the second intermediate result; calculate the sum of the second intermediate result and the offset to obtain the floating-point data inference result.

In a specific embodiment, the first inference module 502 is configured to obtain a floating-point input data set corresponding to the data set to be inferred; divide the data set to be inferred by the floating-point input data set to obtain the first scaling factor.

In a specific embodiment, the first acquisition module 501 is used to determine the data set to be inferred corresponding to the current video frame from the target video; the second acquisition module 503 is used to acquire the corresponding data set to be inferred. outputting a scaling factor; judging whether scene switching occurs in the current video frame relative to the previous video frame, and obtaining a judgment result; determining the output scaling factor corresponding to the data set to be inferred according to the judgment result.

In a specific embodiment, the second obtaining module 503 is configured to, if the judgment result indicates that a scene switch occurs, determine the maximum element from each element of the data inference result; calculate the preset value and the maximum element The quotient of , obtains the output scaling factor; wherein, the preset value is the upper limit value of the value range corresponding to the data type of the data set to be inferred; if the judgment result indicates that scene switching does not occur, then The output scaling factor corresponding to the previous input data set of the data set to be inferred is used as the output scaling factor of the data set to be inferred.

In the embodiment of the present application, the output scaling factor is determined according to whether the scene is switched. If the scene is switched, the scaling factor is re-determined according to the data set to be inferred; if the scene is not switched, in order to minimize the time used for inference, Just reuse the scaling factor of the previous input dataset.

Based on the same concept, the embodiment of the present application also provides an electronic device. As shown in FIG. 6 , the electronic device mainly includes: a processor 601, a memory 602, and a communication bus 603, wherein the processor 601 and the memory 602 communicate through The bus 603 performs communication with each other. The memory 602 stores a program that can be executed by the processor 601, and the processor 601 executes the program stored in the memory 602 to implement the following steps:

Get the data set to be reasoned;

Inputting the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result;

Obtain the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result;

Calculate the product of the floating-point data inference result and the output scaling factor to obtain a quantization result;

Continue the inference of the deep learning model according to the quantization result.

The communication bus 603 mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA for short) bus or the like. The communication bus 603 can be divided into an address bus, a matrix bus, a control bus, and the like. For ease of presentation, only one thick line is used in FIG. 6, but it does not mean that there is only one bus or one type of bus.

The memory 602 may include random access memory (Random Access Memory, RAM for short), and may also include non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor 601 .

The above-mentioned processor 601 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc., and may also be a digital signal processor (Digital Signal Processing, DSP for short) , Application Specific Integrated Circuit (ASIC for short), Field-Programmable Gate Array (FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

In yet another embodiment of the present application, a computer-readable storage medium is also provided, where a computer program is stored in the computer-readable storage medium, and when the computer program runs on a computer, the computer is made to execute the above-mentioned embodiments. An inference method for deep learning models is described.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored on or transmitted from one computer-readable storage medium to another computer-readable storage medium, eg, from a website site, computer, server, or matrix center via wire (eg, Coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, microwave, etc.) way to transmit to another website site, computer, server or matrix center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a matrix storage device such as a server, a matrix center, or the like that contains one or more of the available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes, etc.), optical media (eg, DVDs), or semiconductor media (eg, solid state drives), and the like.

It should be noted that, in this document, relational terms such as "first" and "second" etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these There is no such actual relationship or sequence between entities or operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The above descriptions are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. a kind of reasoning method of deep learning model, is characterized in that, comprises: Get the data set to be reasoned; Inputting the data set to be inferred into the inference convolution kernel to obtain a floating-point data inference result; Obtain the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result; Calculate the product of the data inference result and the output scaling factor to obtain a quantization result; Continue the inference of the deep learning model according to the quantization result. 2. The inference method of a deep learning model according to claim 1, wherein the obtaining the corresponding output scaling factor of the data set to be inferred comprises: From each element of the data inference result, determine the largest element; The quotient of the preset value and the maximum element is calculated to obtain the output scaling factor; wherein the preset value is the upper limit of the value range corresponding to the data type of the data set to be inferred. 3. The inference method of a deep learning model according to claim 1 or 2, wherein the described data set to be inferred is input into an inference convolution kernel to obtain a floating-point data inference result, comprising: obtaining the first scaling factor corresponding to the data set to be inferred; obtaining the weight corresponding to the inference convolution kernel and the second scaling factor corresponding to the weight; obtain the bias of the inference convolution kernel; The floating-point data inference result is calculated according to the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the offset. 4 . The inference method of a deep learning model according to claim 3 , wherein the data set to be inferred, the first scaling factor, the weight, the second scaling factor and the The bias is calculated, and the data inference result of the floating-point type is calculated, including: Calculate the product of the data set to be inferred and the weight to obtain a first intermediate result; Dividing the first intermediate result by the first scaling factor and the second scaling factor to obtain the second intermediate result; The sum of the second intermediate result and the offset is calculated to obtain the floating-point data inference result. 5. The inference method of a deep learning model according to claim 3, wherein the obtaining the first scaling factor corresponding to the data set to be inferred comprises: obtaining a floating-point input data set corresponding to the data set to be inferred; The first scaling factor is obtained by dividing the data set to be inferred by the floating-point input data set. 6. the inference method of deep learning model according to claim 1, is characterized in that, described acquisition to be inferred data set, comprises: From the target video, determine the data set to be inferred corresponding to the current video frame; The obtaining the output scaling factor corresponding to the data set to be inferred includes: Judging whether scene switching occurs in the current video frame relative to the previous video frame, and obtaining a judgment result; According to the judgment result, the output scaling factor corresponding to the data set to be inferred is determined. 7. The inference method of a deep learning model according to claim 6, wherein the determining the output scaling factor corresponding to the data set to be inferred according to the judgment result comprises: If the judgment result indicates that scene switching occurs, determine the maximum element from each element of the data inference result; calculate the quotient of the preset value and the maximum element to obtain the output scaling factor; wherein the preset value Set the value to the upper limit of the value range corresponding to the data type of the data set to be inferred; If the judgment result indicates that scene switching does not occur, the output scaling factor corresponding to the previous input data set of the data set to be inferred is used as the output scaling factor of the data set to be inferred. 8. An inference device for a deep learning model, comprising: The first acquisition module is used to acquire the data set to be reasoned; a first inference module, configured to input the data set to be inferred into an inference convolution kernel to obtain a floating-point data inference result; a second obtaining module, configured to obtain the output scaling factor corresponding to the data set to be inferred; wherein, the output scaling factor is determined according to the maximum value of the elements in the data inference result; a quantization module for calculating the product of the data inference result and the output scaling factor to obtain a quantization result; The second reasoning module is configured to continue the reasoning of the deep learning model according to the quantization result. 9. An electronic device, comprising: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other through the communication bus; the memory is used to store a computer program; the processor , for executing the program stored in the memory to implement the inference method of the deep learning model according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the inference method of the deep learning model according to any one of claims 1 to 7 is implemented. 11 .

Images