CN117648140A - Method, apparatus, device, medium and program product for evaluating model performance - Google Patents

Abstract

The present disclosure provides a method, device, equipment, medium and program product for evaluating model performance, and relates to artificial intelligence technology fields such as AI development and cloud platforms. An implementation manner of the method includes: in response to receiving a test request for the target class model, obtaining a model performance evaluation plug-in and an image file corresponding to the target class model; wherein the image file includes a test sample set and plug-in running mode information; starting image file, and use the model performance evaluation plug-in to evaluate the test sample set based on plug-in operation mode information to determine the performance evaluation indicators corresponding to the target class model.

Description

Methods, devices, equipment, media and program products for evaluating model performance

Technical field

The embodiments of the present disclosure relate to the computer field, specifically to the field of artificial intelligence technology such as AI development and cloud platforms, and in particular to a method, device, equipment, medium and program product for evaluating model performance.

Background technique

In practical applications, corresponding indicators are usually selected for model performance evaluation based on the specific test scenarios of the model.

At present, the performance evaluation methods in related technologies cannot fully cover all industry scenarios; moreover, model performance evaluation in specific scenarios still cannot be personalized and switched to an appropriate operating process to complete the model performance evaluation.

Contents of the invention

Embodiments of the present disclosure provide a method, device, equipment, media and program product for evaluating model performance.

In a first aspect, an embodiment of the present disclosure proposes a method for evaluating model performance, including: in response to receiving a test request for a target class model, obtaining a model performance evaluation plug-in and an image file corresponding to the target class model; wherein , the image file includes the test sample set and plug-in running mode information; start the image file, and use the model performance evaluation plug-in to evaluate the test sample set based on the plug-in running mode information to determine the performance evaluation indicators corresponding to the target class model.

In some examples, the model performance evaluation plug-in includes a model inference plug-in and a metric evaluation plug-in;

Use the model performance evaluation plug-in to evaluate the test sample set based on plug-in operation mode information to determine the performance evaluation indicators corresponding to the target class model, including:

The model inference plug-in processes the test samples in the test sample set according to the corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples; the indicator evaluation plug-in processes the inference results corresponding to the test samples according to the corresponding plug-in operation mode information and The label corresponding to the test sample determines the performance evaluation index corresponding to the target class model.

In some examples, plug-in running mode information: plug-in running environment information, plug-in running logic information and plug-in running parameter information.

In some examples, the plug-in running parameter information includes plug-in input parameters and plug-in output format;

The model inference plug-in processes the test samples in the test sample set according to the corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples, including:

Through the model inference plug-in according to its corresponding plug-in operation logic information, the following operation process is executed for the test samples in the test sample set: parse the test samples in the test sample set to obtain the parsed test samples; traverse the parsed test samples, and Convert the parsed test samples into plug-in input parameters; process the plug-in input parameters to obtain inference results that match the plug-in output format.

In some examples, plug-in operating parameter information includes indicator evaluation algorithms;

The indicator evaluation plug-in processes the inference results corresponding to the test sample and the label corresponding to the test sample according to the corresponding plug-in running mode information to determine the performance evaluation indicators corresponding to the target class model, including: using the indicator evaluation plug-in according to its corresponding The plug-in runs the logic information and performs the following operation process on the inference results: respectively parses the inference results corresponding to the test sample and the label corresponding to the test sample to obtain the parsed label and parsed inference result; according to the indicator evaluation algorithm, process the parsing The resulting labels and parsed reasoning results are used to determine the performance evaluation indicators corresponding to the target class model.

In some examples, the plug-in running parameter information includes a display method; the method also includes: displaying the performance evaluation indicators of the target class model through the display method.

In some examples, if the target class model includes multiple types of models, display the performance evaluation indicators of the target class model through a display method, including: displaying the performance evaluation indicators of the multiple types of models through a comparison chart corresponding to the display method.

In some examples, the method also includes: encapsulating the model service code source file, test sample set and plug-in running mode information of the target class model into an image file.

In a second aspect, an embodiment of the present disclosure proposes a device for evaluating model performance, including: an acquisition module, configured to obtain a model performance evaluation plug-in corresponding to the target class model in response to receiving a test request for the target class model. and image file; among them, the image file includes test sample set and plug-in running mode information; the performance evaluation module is used to start the image file, and evaluate the test sample set according to the plug-in running mode information through the model performance evaluation plug-in to determine the corresponding target class model performance evaluation indicators.

In some examples, the model performance evaluation plug-in includes a model inference plug-in and a metric evaluation plug-in;

The performance evaluation module includes: a model inference unit, used to process the test samples in the test sample set according to the corresponding plug-in operation mode information through the model inference plug-in, and obtain the inference results corresponding to the test samples; the performance evaluation unit, used to evaluate through indicators The plug-in processes the inference result corresponding to the test sample and the label corresponding to the test sample according to the corresponding plug-in running mode information, and determines the performance evaluation index corresponding to the target class model.

In some examples, plug-in running mode information: plug-in running environment information, plug-in running logic information and plug-in running parameter information.

In some examples, the plug-in running parameter information includes plug-in input parameters and plug-in output format;

The model inference unit is specifically used to: use the model inference plug-in to run logical information according to its corresponding plug-in, and execute the following operation process on the test samples in the test sample set: parse the test samples in the test sample set to obtain the parsed test sample; Traverse the parsed test samples and convert the parsed test samples into plug-in input parameters; process the plug-in input parameters to obtain inference results that match the plug-in output format.

In some examples, plug-in operating parameter information includes indicator evaluation algorithms;

The performance evaluation unit is specifically used to: use the indicator evaluation plug-in to run logical information according to its corresponding plug-in, and perform the following operation process on the inference results: respectively parse the inference results corresponding to the test sample and the label corresponding to the test sample, and obtain the parsed labels and parsed reasoning results; according to the indicator evaluation algorithm, the parsed labels and parsed reasoning results are processed to determine the performance evaluation indicators corresponding to the target class model.

In some examples, the plug-in running parameter information includes a display method; the device also includes: a display module, used to display the performance evaluation indicators of the target class model through the display method.

In some examples, if the target class model includes multiple types of models, the display module is specifically used to: display the performance evaluation indicators of the multiple types of models through a comparison chart corresponding to the display method.

In some examples, the device further includes: an encapsulation module, configured to encapsulate the model service code source file of the target class model, the test sample set, and the plug-in running mode information into an image file.

In a third aspect, embodiments of the present disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are At least one processor executes, so that at least one processor can execute the method as described in the first aspect.

In a fourth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause a computer to execute the method described in the first aspect.

In a fifth aspect, an embodiment of the present disclosure proposes a computer program product, which includes a computer program. When executed by a processor, the computer program implements the method described in the first aspect.

The embodiment of the present disclosure proposes a method, device, equipment, medium and program product for evaluating model performance. Through test requests for target class models, performance evaluation of different types of models can be achieved; after receiving the test request, Select the model performance evaluation plug-in and image file corresponding to the target class model; after starting the image file, use the model performance evaluation plug-in to evaluate the test sample set in the image file according to the plug-in running mode information in the image file to obtain the target class model Performance evaluation indicators can indicate the running process of the model performance evaluation plug-in evaluation test sample set through personalized customized plug-in operation mode information, thereby achieving personalized performance evaluation.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments taken with reference to the following drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. in:

Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

Figure 2 is a flowchart of one embodiment of a method for evaluating model performance according to the present disclosure;

Figure 3 is a schematic diagram of the plug-in development process;

Figure 4 is a flowchart of one embodiment of a method for evaluating model performance according to the present disclosure;

Figures 5-6 are application scenario diagrams of methods for evaluating model performance according to the present disclosure;

Figure 7 is a schematic diagram of one embodiment of an apparatus for evaluating model performance according to the present disclosure;

Figure 8 is a block diagram of an electronic device used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the present disclosure are included to facilitate understanding and should be considered to be exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

It should be noted that, as long as there is no conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other. The present disclosure will be described in detail below in conjunction with embodiments with reference to the accompanying drawings.

FIG. 1 illustrates an exemplary system architecture 100 in which embodiments of methods and apparatus for evaluating model performance of the present disclosure may be applied.

As shown in Figure 1, the system architecture 100 may include a client 101, a network 102 and a test platform 103. The network 102 is a medium used to provide a communication link between the client 101 and the test platform 103 . Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The client 101 interacts with the test platform 103 through the network 102 to receive or send test requests and so on.

The test platform 103 can be a platform that provides various model performance evaluations. For example, it matches the test request of the client 101 to the model performance evaluation plug-in and image file corresponding to the target class model, and uses the model performance evaluation plug-in to follow the image file. Obtain the performance evaluation index of the target class model.

Among them, the test platform 103 can be hardware or software. When the test platform 103 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or it can be implemented as a single server. When the test platform 103 is software, it may be implemented as multiple software or software modules (for example, used to provide distributed services), or it may be implemented as a single software or software module. There are no specific limitations here.

It should be noted that the method for evaluating model performance provided by the embodiment of the present disclosure is generally executed by the test platform 103. Correspondingly, the device for evaluating model performance is generally provided in the test platform 103.

It should be understood that the numbers of clients, networks and test platforms in Figure 1 are only illustrative. You can have any number of clients, networks, and test platforms depending on your implementation needs.

Continuing with reference to FIG. 2 , a process 200 is shown for one embodiment of a method for evaluating model performance in accordance with the present disclosure. The method for evaluating model performance may include the following steps:

Step 201: In response to receiving a test request for the target class model, obtain the model performance evaluation plug-in and image file corresponding to the target class model. The image file includes a test sample set and plug-in operating mode information.

In this embodiment, the execution subject of the method for evaluating model performance (for example, the test platform 103 shown in Figure 1) can receive the target class model sent by the user end (for example, the user end shown in Figure 1). When making a test request, obtain the model performance evaluation plug-in and image file corresponding to the target class model. The target class model can be a model type of a given type, which can be reflected by its name.

Here, the image file can include: a test sample set for testing, and information about how the plug-in runs. The plug-in running mode information can be used to define the plug-in running process, the environment required for plug-in running, parameters and algorithms involved in plug-in running, etc.

In one example, when receiving a test request for a target class model, the model performance evaluation plug-in and image file corresponding to the target class model can be obtained from the corresponding storage path.

It should be noted that the corresponding storage path can be the storage path in the custom specification for the plug-in developer or the storage path personalized by the test user in the image file.

In this embodiment, different model performance evaluation plug-ins are selected or switched through test requests, and the plug-in running mode information is used to guide the model performance evaluation plug-in to evaluate the test sample set to complete the performance evaluation of different types of models.

In an example, in Figure 3, test users can build a model performance evaluation plug-in on the test platform and import the test sample set. On the test platform, test users can initiate a test request and obtain the corresponding model performance evaluation plug-in based on the test request. Later, after testing, test users can view performance evaluation metrics to gain a deeper understanding of the performance of the target class model.

When obtaining the model performance evaluation plug-in, the matching model performance evaluation plug-in is selected based on the type of the target class model, so that it can be applied to various types of model performance evaluation scenarios and achieve personalized model performance evaluation.

Plug-in developers can develop and debug the model performance evaluation plug-in through their development terminal to ensure that the model performance evaluation plug-in can run normally in an offline environment. After the model performance evaluation plug-in is successfully debugged, the plug-in developer can upload the plug-in's code package, dependencies, configuration and other information to the test platform and implement management operations. Subsequently, after the plug-in verification is passed on the test platform, the plug-in will be released and put online so that it can be evaluated and used by test users.

When the plug-in version needs to be updated, you can also complete the development and debugging of the updated version of the plug-in first, and upload the updated version to the test platform. The test platform will verify the plug-in and then put the plug-in online so that test users can use the updated version. Plugins are evaluated and used.

Here, plug-in developers can provide a running environment (for example, a mirror) for plug-in operation, develop specifications for plug-in operation, etc.; among which, the specifications can be specifications for test sample sets, specifications for inference results, and parameters for input plug-ins ( For example, plug-in input parameters) and plug-in output parameters (for example, plug-in output format) specifications, evaluation algorithm specifications for custom performance evaluation indicators, etc. Here, the above specifications may be specifications of storage paths, formats, etc.

It should be noted that test users can provide test requirements to plug-in developers according to their specific testing needs, and plug-in developers develop new plug-ins and share them with the test platform to evaluate different types of models and ensure testing Platform scalability.

In this embodiment, the plug-in's running environment and the storage path of the plug-in's input and output parameter formats can be standardized by the plug-in developer or personalized by the test user and set in the image file.

Step 202: Start the image file, and use the model performance evaluation plug-in to evaluate the test sample set according to the plug-in operating mode information to determine the performance evaluation indicators corresponding to the target class model.

In this embodiment, the above execution subject first starts the image file; then, uses the model performance evaluation plug-in to process the test samples in the test sample set according to the plug-in running mode information to obtain the inference results corresponding to the test samples; then, according to the corresponding test samples The inference results and the label corresponding to the test sample are used to determine the performance evaluation index corresponding to the target class model.

Here, the performance evaluation index corresponding to the target class model can be a performance evaluation index specific to this type of model. This performance evaluation index can characterize the performance of the target class model.

Here, starting the image file may include: starting the plug-in running mode information as a container; and then attaching the test sample set to the container to achieve sharing or access of the test sample set.

In one example, the above method of evaluating the test sample set can be determined based on the operation method corresponding to the performance evaluation index corresponding to the target class model. For example, the root mean square error is obtained based on the square root of the mean square error.

In one example, the performance evaluation indicators of the target class model may include:

For example, the performance evaluation indicators corresponding to the classification model can include: Accuracy: the proportion of correctly classified samples to the total number of samples; Precision: the proportion of true cases to true cases and false positives; recall rate (Recall): the proportion of true examples to true examples and false negative examples; F1 Score: the harmonic mean of precision and recall, used to balance the two; ROC (Receiver Operating Characteristic) curve and AUC (Area UnderCurve) ): Receiver operating characteristic curve and area under the curve, used for binary classification problems.

Performance evaluation indicators corresponding to the regression model can include: Mean Squared Error (MSE): the average of the squared differences between the actual value and the predicted value; Root Mean Squared Error (RMSE): the MSE Square root; Mean Absolute Error (MAE): the average of the absolute difference between the actual value and the predicted value; R-squared (R-squared, R ² ): used to measure the degree to which the model explains the total variance.

The performance evaluation indicators corresponding to the target detection model can include: Average Precision (AP): the average accuracy under different IoU (intersection over union ratio) thresholds in the target detection task; mAP (mean Average Precision): AP in multiple The average value on each category; IoU (Intersection and Union Ratio): the ratio of the intersection and union of the target bounding box and the real target bounding box; FROC (Free-Response ROC) curve: the ratio between the false positive rate and the recall rate in the target detection task curve between.

Performance evaluation indicators corresponding to the image generation model can include: SSIM (Structural SimilarityIndex, structural similarity index): used to compare the structural similarity of the generated image and the original image; PSNR (Peak Signalto Noise Ratio, peak signal-to-noise ratio): metric generation The signal-to-noise ratio between the image and the original image; LPIPS (LearnedPerceptual Image Patch Similarity, perceived image similarity): measures the perceptual similarity of the image, which is more in line with human eye perception.

Performance evaluation indicators corresponding to natural language processing models can include: BLEU (Bilingual Evaluation Understudy): an automatic evaluation indicator for machine translation tasks; ROUGE (Recall-Oriented Understudy for Gisting Evaluation): an evaluation indicator for text summarization tasks; METEOR (Metric forEvaluation of Translation with Explicit ORdering): an automatic evaluation indicator for machine translation; Perplexity (perplexity): an indicator for language model performance evaluation.

In this embodiment, the above performance evaluation index can evaluate the model performance in different dimensions and give an evaluation value (ie, performance evaluation index). The larger a certain index is, the better the classification performance of the model is in a certain aspect. .

In this embodiment, the performance evaluation of the target class model can be completed through the performance evaluation indicators corresponding to the target class model.

The method for evaluating model performance provided by the embodiments of the present disclosure can achieve performance evaluation of different types of models through test requests for target class models; after receiving the test request, select a model performance evaluation plug-in corresponding to the target class model. and image files; after starting the image file, use the model performance evaluation plug-in to evaluate the test sample set in the image file according to the plug-in running mode information in the image file to obtain the performance evaluation indicators of the target class model, which can be customized through personalized plug-ins. The running mode information instructs the model performance evaluation plug-in to evaluate the running process of the test sample set, thereby achieving personalized performance evaluation.

Referring further to Figure 4, a process 400 is shown for one embodiment of a method for evaluating model performance in accordance with the present disclosure. The method for evaluating model performance may include the following steps:

Step 401: In response to receiving a test request for the target class model, obtain the model performance evaluation plug-in and image file corresponding to the target class model. The image file includes a test sample set and plug-in operating mode information.

In this embodiment, the execution body of the method for evaluating model performance (such as the test platform 103 shown in Figure 1) can obtain the model performance corresponding to the target class model when receiving a test request for the target class model. Evaluate plugins and image files.

Here, the model performance evaluation plug-in can include a model inference plug-in and an indicator evaluation plug-in. The model inference plug-in can be used to process the test sample set to obtain inference results. This indicator evaluation plug-in can be used to process inference results and labels to obtain performance evaluation indicators corresponding to the target class model.

In this embodiment, during the model testing process, plug-ins can be added accordingly according to the test user's specific test request to complete the test user's test request.

Step 402: The model inference plug-in processes the test samples in the test sample set according to the corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples.

In this embodiment, the above execution subject can use the model inference plug-in to process the test sample set according to the plug-in operating mode information corresponding to the model inference plug-in to obtain the inference result. The result of this inference can be a predicted value.

Here, the above-mentioned execution subject can set the plug-in running mode information of the model inference plug-in through the plug-in developer and/or the test user. The plug-in operation mode information of the model inference plug-in can be used to define the operation logic, operation parameters, operation environment, etc. involved in the process of obtaining inference results from the test sample set.

In this embodiment, by setting the plug-in running mode information of the model inference plug-in by the test user, the test user's specific test request can be satisfied.

Step 403: The indicator evaluation plug-in processes the inference result corresponding to the test sample and the label corresponding to the test sample according to its corresponding plug-in operation mode information to determine the performance evaluation index corresponding to the target class model.

In this embodiment, the execution subject uses the indicator evaluation plug-in to process the inference results and labels according to the plug-in operating mode information corresponding to the indicator evaluation plug-in, and determines the performance evaluation index corresponding to the target class model. The plug-in operation mode information of the indicator evaluation plug-in can be used to define the operation logic, operation parameters, operation environment, etc. involved in the process of obtaining performance evaluation indicators from the inference results.

Here, the above-mentioned execution subject can evaluate the plug-in running mode information of the plug-in through the plug-in developer and/or the indicators set by the test user.

In this embodiment, the test user's specific test request can be satisfied by setting indicators to evaluate the plug-in running mode information of the plug-in.

In this embodiment, the specific operations of step 40 have been introduced in detail in step 201 in the embodiment shown in FIG. 2 and will not be described again here.

As can be seen from Figure 4, compared with the embodiment corresponding to Figure 2, the method for evaluating model performance in this embodiment highlights the process of processing the test sample set through the model inference plug-in to obtain the inference results; then, through the indicator evaluation The plug-in evaluates the inference results and labels to obtain performance evaluation indicators, so that the plug-in can be plugged in to complete the specific testing needs of the test user.

In some optional implementations of this embodiment, plug-in running mode information: plug-in running environment information, plug-in running logic information and plug-in running parameter information.

In this implementation, the running mode of the model performance evaluation plug-in can be configured through at least one of the plug-in running environment information, plug-in running logic information, and plug-in running parameter information. The running mode can be used to define the model performance evaluation plug-in evaluation test. The running process of the sample set.

Here, the plug-in running environment information may be the environment information of the model performance evaluation plug-in running on a test platform (such as the test platform 103 shown in Figure 1). The plug-in running logic information can define the execution logic of the model performance evaluation plug-in to process the test sample set. The plug-in running parameter information can define parameter-related information during the operation of the model performance evaluation plug-in, such as parameter format, parameter type, algorithm corresponding to the parameter, and display method corresponding to the parameter.

In this implementation, the running process of the model performance evaluation plug-in evaluation test sample set is defined through plug-in running environment information, plug-in running logic information, and plug-in running parameter information.

In some optional implementations of this embodiment, the plug-in running parameter information includes plug-in input parameters and plug-in output format;

The model inference plug-in processes the test samples in the test sample set according to the corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples, including:

Through the model inference plug-in, according to the corresponding plug-in operation logic information, the following operation process is executed for the test samples in the test sample set:

Analyze the test samples in the test sample set to obtain the parsed test samples;

Traverse the parsed test samples and convert the parsed test samples into plug-in input parameters;

Process the plug-in input parameters and obtain inference results that match the plug-in output format.

In this implementation, the above-mentioned execution subject uses the model inference plug-in to run logical information according to its corresponding plug-in, parses the test samples in the test sample set, and obtains the parsed test samples; traverses the parsed test samples, and compiles the parsed test samples. The samples are converted into plug-in input parameters; the plug-in input parameters are processed to obtain inference results, and the inference results are converted into inference results that match the plug-in output format.

In an example, the running logic of the model inference plug-in can include:

1. Obtain the test sample set from the path of the test sample set and parse the test sample set; 2. Traverse the parsed test samples and execute: 3.1 Convert the parsed test samples into plug-in input parameters; 3.2 Initiate an inference request and get Inference results; 3.3 Convert the format of the inference results to the plug-in output format; 3.4 Write the inference results in the plug-in output format into the result file; 4. If there is other information that needs to be output, it can be written into the result file, which can be read by other plug-ins later. Pick.

Among them, input_func is a model inference input parameter script, used to convert test samples into plug-in input parameters; output_func is a model output conversion script, used to convert the format of inference results into plug-in output format.

It should be noted that the inference results are written into the result file; other data that can be used by subsequent evaluation indicator plug-ins, such as labels, can also be written into the result file or other files in file form.

Before parsing the test sample, you need to obtain the model service code source file according to the path of the model service code source file, deploy it to the model service address, and obtain the model performance evaluation plug-in; then, process the test sample set through the model inference plug-in to obtain the inference results.

The model service address is the address of the device where the model service is to be deployed (for example, the network address of the client).

Here, the above paths and addresses can be set in the plug-in running parameter information. For example, the path of the file used to store the test sample set and the plug-in input and output results can be a storage path customized by the plug-in developer or a storage path set by the test user in the plug-in running parameter information.

Correspondingly, in this example, if you need to collect the "average inference delay" indicator at the same time during the evaluation of the target class model, you can warm up the model to obtain the indicator evaluation results corresponding to the "average inference delay".

In this implementation, test users can meet their specific test requests by setting plug-in running mode information.

In some optional implementations of this embodiment, the plug-in operating parameter information includes an indicator evaluation algorithm;

The indicator evaluation plug-in processes the inference results corresponding to the test sample and the label corresponding to the test sample according to the corresponding plug-in running mode information to determine the performance evaluation indicators corresponding to the target class model, including:

Through the indicator evaluation plug-in, according to the corresponding plug-in operation logic information, the following operation process is performed on the inference results:

Analyze the inference result corresponding to the test sample and the label corresponding to the test sample respectively, and obtain the parsed label and the parsed inference result;

According to the index evaluation algorithm, the parsed labels and the parsed inference results are processed to determine the performance evaluation index corresponding to the target class model.

In this implementation, the above-mentioned execution subject can use the indicator evaluation plug-in to run the logic information according to its corresponding plug-in, respectively parse the inference results corresponding to the test sample and the label corresponding to the test sample, and obtain the parsed label and parsed label. Inference results; process the parsed labels and parsed inference results according to the indicator evaluation algorithm to determine the performance evaluation indicators corresponding to the target class model.

In an example, the running logic of the indicator evaluation plug-in can include the following running process:

(1.1). Obtain the inference results and labels from the result file; (1.2). Parse the inference results and labels respectively, and the file path of the inference result is the path of the result file; (1.3). Determine the model with the target class through the indicator evaluation plug-in Corresponding performance evaluation indicators.

It should be noted that when determining the performance evaluation index, it can also be determined by evaluating the inference results and labels through the index evaluation algorithm defined in the plug-in running parameter information.

In this implementation, personalized evaluation of inference results and labels is achieved through the indicator evaluation algorithm defined in the plug-in running parameter information.

In some optional implementations of this embodiment, the method further includes:

Encapsulate the model service code source file, test sample set and plug-in running mode information of the target class model into an image file.

In this implementation, before model performance evaluation, the method for evaluating model performance also includes: encapsulating the model service code source file, test sample set and plug-in running mode information of the target class model into an image file.

Among them, the model performance evaluation plug-in corresponding to the target class model can be a file class model or a mirror class model. During the indicator evaluation process, the shared model service code source file appears in the model performance evaluation process in the form of a model service. The model service code source file can be used to define the path of the model service code source file, the model service address, etc.

Test sample set: It includes tags. The format of the test sample set can be set in the image file by the plug-in developer or the test user. For example, the plug-in developer provides the storage path of the sample data package.

Performance evaluation indicators: Performance evaluation indicators calculated based on labels and inference results to evaluate the target class model, such as accuracy, prediction delay, etc.

Plug-in running mode information: Based on Python code composition, the execution process of the plug-in in the model performance evaluation process is defined. For example, plug-in running mode information includes plug-in running environment information. The plug-in running environment information is used to define the running environment (or resources, etc.) for plug-in running. It is generally composed of images, such as python38+ environment.

In some optional implementations of this embodiment, if the plug-in running parameter information includes a display method, the method further includes: displaying the performance evaluation index of the target class model through a display method.

In this implementation, the display methods that can be used include: numerical type, text type, and line drawing.

Here, the above display method may refer to the display method of some or all performance evaluation indicators of the target class model.

After obtaining the performance evaluation indicators, the performance evaluation indicators can be written into the corresponding storage target, such as an indicator database or list, based on the plug-in operating parameter information.

In one example, the line graph can display the performance evaluation results of the target class model through elements such as lines, for example, a line graph, a bar graph, etc.

In this implementation, the performance evaluation indicators are displayed through the display method included in the plug-in running parameter information.

In some optional implementations of this embodiment, if the target class model includes multiple types of models, the performance evaluation indicators of the target class model are displayed through a display method, including:

The performance evaluation indicators of multiple types of models are displayed through comparison charts corresponding to the display methods.

In this implementation, the performance evaluation indicators of multiple types of models are displayed through comparison charts corresponding to the display method. This comparison chart can display all or part of the performance evaluation indicators of multiple types of models together through the definition of the display method; or cross-analyze and display all or part of the performance evaluation indicators; or display all of the multiple types of models through a confusion matrix. Or some performance evaluation indicators, such as sample proportion, etc.

Here, cross-analysis is performed on all or part of the performance evaluation indicators, and some performance of the model can be demonstrated through cross-analysis results. For example, accuracy and delay can measure model delay and resource usage.

In one example, all performance indicators of multiple types of models are displayed through a comparison chart. In the comparison chart, the model name, a list of performance evaluation indicators, and a line chart of all performance evaluation indicators of each model can be presented.

In one example, at least some of the performance indicators among all performance indicators are displayed through a comparison chart, for example, a line chart or a bar chart or a curve chart (for example, P-R curvature or ROC curve), for example, accuracy rate.

In this implementation, multiple types of models are displayed through comparison charts, and during the display process, test users can select the content they need to display (for example, a certain type of model, certain performance evaluation indicators), etc. according to their needs, so that Can meet the specific testing needs of this test user.

Referring further to FIGS. 5-6 , application scenario diagrams of methods for evaluating model performance according to the present disclosure are shown. In this application scenario, the method used to evaluate model performance can include the following steps:

(1) Preparation before evaluation: Prepare model service code source files and test sample sets, and start and publish the model performance evaluation plug-in as an online service; (2) Start model service: Request the target class model and test sample set through test requests to Model reasoning link; (3) Model reasoning: The model reasoning plug-in first parses the test sample set to obtain the parsed test samples, converts the parsed test samples into plug-in input parameters (i.e., input_func), and formats the inference results Convert to plugin output format (i.e., output_func). (4) Performance evaluation: The metric evaluation plug-in parses the inference results and labels, and evaluates the inference results and labels through metric evaluation calculations (i.e., calculate_metrics_func) to obtain performance evaluation indicators.

After obtaining the performance evaluation index, the method may also include: displaying the performance evaluation index through display; after displaying the performance evaluation index, the method may also include middleware cleaning and destruction of the model service to evaluate the performance of the target class model. The resources occupied, that is, CPU (Central Processing Unit/Processor)/Memory/GPU (Graphics Processing Unit)).

It should be noted that when evaluating the performance of a target class model, the resource usage of the target class model can also be used to assist in evaluating the performance of the target class model. For example, a model that uses fewer resources will perform better than a model that uses more resources.

In this embodiment, the performance evaluation of the target class model is completed through pluggable plug-ins.

With further reference to Figure 7, as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for evaluating model performance. The device embodiment corresponds to the method embodiment shown in Figure 2, The device can be applied in various electronic devices.

As shown in Figure 7, the device 700 for evaluating model performance in this embodiment may include: an acquisition module 701 and a performance evaluation module 702. Among them, the acquisition module 701 is used to obtain the model performance evaluation plug-in and image file corresponding to the target class model in response to receiving a test request for the target class model; wherein the image file includes a test sample set and plug-in operating mode information; performance The evaluation module 702 is used to start the image file, and evaluate the test sample set according to the plug-in operation mode information through the model performance evaluation plug-in to determine the performance evaluation indicators corresponding to the target class model.

In this embodiment, in the device 700 for evaluating model performance: the specific processing of the acquisition module 701 and the performance evaluation module 702 and the technical effects they bring can be referred to steps 201-202 in the corresponding embodiment of Figure 2 respectively. Relevant instructions will not be repeated here.

In some examples, the model performance evaluation plug-in includes a model inference plug-in and a metric evaluation plug-in;

The performance evaluation module 702 includes: a model inference unit, used to process the test samples in the test sample set according to the corresponding plug-in operation mode information through the model inference plug-in, and obtain the inference results corresponding to the test samples; a performance evaluation unit, used to pass the indicator The evaluation plug-in processes the inference results corresponding to the test sample and the label corresponding to the test sample according to the corresponding plug-in running mode information, and determines the performance evaluation index corresponding to the target class model.

In some examples, plug-in running mode information: plug-in running environment information, plug-in running logic information and plug-in running parameter information.

In some examples, the plug-in running parameter information includes plug-in input parameters and plug-in output format;

The model inference unit is specifically used to: use the model inference plug-in to run logical information according to its corresponding plug-in, and execute the following operation process on the test samples in the test sample set: parse the test samples in the test sample set to obtain the parsed test sample; Traverse the parsed test samples and convert the parsed test samples into plug-in input parameters; process the plug-in input parameters to obtain inference results that match the plug-in output format.

In some examples, plug-in operating parameter information includes indicator evaluation algorithms;

The performance evaluation unit is specifically used to: use the indicator evaluation plug-in to run logical information according to its corresponding plug-in, and perform the following operation process on the inference results: respectively parse the inference results corresponding to the test sample and the label corresponding to the test sample, and obtain the parsed labels and parsed reasoning results; according to the indicator evaluation algorithm, the parsed labels and parsed reasoning results are processed to determine the performance evaluation indicators corresponding to the target class model.

In some examples, the plug-in running parameter information includes a display method; the device also includes: a display module, used to display the performance evaluation indicators of the target class model through the display method.

In some examples, if the target class model includes multiple types of models, the display module is specifically used to: display a comparison chart of performance evaluation indicators of multiple types of models.

In some examples, the device further includes: an encapsulation module, configured to encapsulate the model service code source file of the target class model, the test sample set, and the plug-in running mode information into an image file.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

Figure 8 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are examples only and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 8 , the device 800 includes a computing unit 801 that can execute according to a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and treatments. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. Computing unit 801, ROM 802 and RAM 803 are connected to each other via bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

Multiple components in the device 800 are connected to the I/O interface 805, including: an input unit 808, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc. ; and communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunications networks.

Computing unit 801 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processing processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as methods for evaluating model performance. For example, in some embodiments, a method for evaluating model performance may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 800 via ROM 802 and/or communication unit 809 . When the computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of the method for evaluating model performance described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the method for evaluating model performance in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described above may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip implemented in a system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or a combination thereof. These various embodiments may include implementation in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor The processor, which may be a special purpose or general purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. An output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that the program codes, when executed by the processor or controller, cause the functions specified in the flowcharts and/or block diagrams/ The operation is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, laptop disks, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and may be provided in any form, including Acoustic input, voice input or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., A user's computer having a graphical user interface or web browser through which the user can interact with implementations of the systems and technologies described herein), or including such backend components, middleware components, or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communications network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.

Computer systems may include clients and servers. Clients and servers are generally remote from each other and typically interact over a communications network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other.

Artificial intelligence is a discipline that studies computers to simulate certain human thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level technology and software-level technology. Artificial intelligence hardware technology generally includes sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing and other technologies; artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural speech processing technology and machine learning/depth Learning, big data processing technology, knowledge graph technology and other major directions.

It should be understood that various forms of the process shown above may be used, with steps reordered, added or deleted. For example, each step described in the present disclosure can be executed in parallel, sequentially, or in a different order. As long as the desired results of the technical solutions mentioned in the present disclosure can be achieved, there is no limitation here.

The above-mentioned specific embodiments do not constitute a limitation on the scope of the present disclosure. It will be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions are possible depending on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure shall be included in the protection scope of this disclosure.

Claims (19)

1. A method for evaluating model performance, including: In response to receiving a test request for the target class model, obtain a model performance evaluation plug-in and an image file corresponding to the target class model; wherein the image file includes a test sample set and plug-in operating mode information; Start the image file, and use the model performance evaluation plug-in to evaluate the test sample set according to the plug-in operating mode information to determine the performance evaluation index corresponding to the target class model. 2. The method according to claim 1, wherein the model performance evaluation plug-in includes a model inference plug-in and an indicator evaluation plug-in; The method of using the model performance evaluation plug-in to evaluate the test sample set according to the plug-in operating mode information and determine the performance evaluation index corresponding to the target class model includes: The test samples in the test sample set are processed by the model inference plug-in according to the corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples; The performance evaluation index corresponding to the target class model is determined by processing the inference result corresponding to the test sample and the label corresponding to the test sample according to the corresponding plug-in operation mode information by the index evaluation plug-in. 3. The method according to claim 1 or 2, wherein the plug-in operation mode information includes: plug-in operation environment information, plug-in operation logic information and plug-in operation parameter information. 4. The method according to claim 3, wherein the plug-in operation parameter information includes plug-in input parameters and plug-in output format; Processing the test samples in the test sample set through the model inference plug-in according to its corresponding plug-in operation mode information to obtain the inference results corresponding to the test samples includes: Through the model inference plug-in according to its corresponding plug-in operation logic information, the following operation process is executed for the test samples in the test sample set: Analyze the test samples in the test sample set to obtain the parsed test samples; Traverse the parsed test samples and convert the parsed test samples into the plug-in input parameters; Process the plug-in input parameters to obtain inference results that match the plug-in output format. 5. The method according to claim 3, wherein the plug-in operating parameter information includes an indicator evaluation algorithm; Processing the inference result corresponding to the test sample and the label corresponding to the test sample through the indicator evaluation plug-in according to its corresponding plug-in operation mode information to determine the performance evaluation index corresponding to the target class model, including: Through the indicator evaluation plug-in according to its corresponding plug-in operation logic information, the following operation process is executed for the inference results corresponding to the test sample: Analyze the inference result corresponding to the test sample and the label corresponding to the test sample respectively, and obtain the parsed label and the parsed inference result; According to the index evaluation algorithm, the parsed labels and the parsed reasoning results are processed to determine the performance evaluation index corresponding to the target class model. 6. The method according to claim 3, wherein the plug-in operation parameter information includes a display mode; the method further includes: The performance evaluation indicators of the target class model are displayed through the display method. 7. The method according to claim 6, wherein if the target class model includes multiple class models, the performance evaluation indicators of the target class model displayed through the display method include: The performance evaluation indicators of the multiple types of models are displayed through comparison charts corresponding to the display method. 8. The method of claim 1, wherein the method further comprises: Encapsulate the model service code source file, test sample set and plug-in running mode information of the target class model into an image file. 9. An apparatus for evaluating model performance, comprising: An acquisition module, configured to obtain a model performance evaluation plug-in and an image file corresponding to the target class model in response to receiving a test request for the target class model; wherein the image file includes a test sample set and plug-in operation mode information; A performance evaluation module is used to start the image file, evaluate the test sample set according to the plug-in operating mode information through the model performance evaluation plug-in, and determine the performance evaluation index corresponding to the target class model. 10. The device according to claim 9, wherein the model performance evaluation plug-in includes a model inference plug-in and an indicator evaluation plug-in; The performance evaluation module includes: A model inference unit, configured to process the test samples in the test sample set according to the corresponding plug-in operation mode information through the model inference plug-in to obtain the inference results corresponding to the test samples; A performance evaluation unit configured to process the inference results corresponding to the test sample and the label corresponding to the test sample through the index evaluation plug-in according to its corresponding plug-in operation mode information, and determine the performance evaluation index corresponding to the target class model. 11. The device according to claim 9 or 10, wherein the plug-in operation mode information includes: plug-in operation environment information, plug-in operation logic information and plug-in operation parameter information. 12. The device according to claim 11, wherein the plug-in operating parameter information includes plug-in input parameters and plug-in output format; The model inference unit is specifically used to perform the following operation process on the test samples in the test sample set through the model inference plug-in according to its corresponding plug-in operation logic information: parse the test samples in the test sample set, and obtain Parsed test samples; traverse the parsed test samples, and convert the parsed test samples into the plug-in input parameters; process the plug-in input parameters to obtain inference results that match the plug-in output format. 13. The device according to claim 11, wherein the plug-in operating parameter information includes an indicator evaluation algorithm; The performance evaluation unit is specifically configured to perform the following operation process on the inference results corresponding to the test sample through the indicator evaluation plug-in according to its corresponding plug-in operation logic information: respectively analyze the inference results corresponding to the test sample and the inference results corresponding to the test sample. Test the label corresponding to the sample to obtain the parsed label and the parsed inference result; process the parsed label and the parsed inference result according to the index evaluation algorithm to determine the performance evaluation index corresponding to the target class model. 14. The device according to claim 11, wherein the plug-in operating parameter information includes a display mode; the device further includes: a display module configured to display performance evaluation indicators of the target class model through the display mode. 15. The device according to claim 14, wherein if the target class model includes multiple class models, the display module is specifically configured to: display the multiple classes through a comparison chart corresponding to the display method. Model performance evaluation metrics. 16. The device according to claim 9, wherein the device further comprises: an encapsulation module, configured to encapsulate the model service code source file, test sample set and plug-in running mode information of the target class model into an image file. 17. An electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of claims 1-8. Methods. 18. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method of any one of claims 1-8. 19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.