CN118733450A - A model testing method, device, equipment and medium - Google Patents

Abstract

The disclosed embodiments relate to a model testing method, apparatus, device and medium, wherein the method comprises: constructing a test question based on a standard answer in a basic data set; constructing a test data set based on the test question and a prompt word template; obtaining a target model, and inputting the prompt words in the test data set into the target model to obtain the corresponding answer to be tested; performing a unit test on the answer to be tested to determine the unit test result; and determining the model test result of the target model based on the unit test result. The disclosed embodiments determine the model test result of the target model based on the unit test result, and use the unit test result as the test indicator of the target model. Compared with the prior art, considering the operability characteristics of special scenarios such as code processing, it can more objectively and effectively reflect the real capabilities of the model and greatly improve the accuracy of the model test.

Description

A model testing method, device, equipment and medium

Technical Field

The present disclosure relates to the field of computer technology, and in particular to a model testing method, device, equipment and medium.

Background Art

With the development of technology, large language models are increasingly used in the field of code. In related technologies, the testing of models is mainly achieved through data from non-real scenarios or real scenarios. The data testing of non-real scenarios is quite different from that of real scenarios. The test indicators of real scenario data testing are usually based on the similarity between the model results and the correct answers to determine whether the model results are correct. This method cannot effectively and objectively reflect the model capabilities in special scenarios, such as code processing-related scenarios. Therefore, it is difficult for model testing in related technologies to ensure authenticity while taking into account effectiveness, and it needs to be improved.

Summary of the invention

In order to solve the above technical problems, the present disclosure provides a model testing method, device, equipment and medium.

The present disclosure provides a model testing method, the method comprising:

Construct test questions based on standard answers in the basic data set;

Constructing a test data set based on the test questions and prompt word templates;

Obtaining a target model, and inputting the prompt words in the test data set into the target model to obtain the corresponding answer to be tested;

Performing unit testing on the answer to be tested and determining the unit testing result;

A model test result of the target model is determined based on the unit test result.

The present disclosure also provides a model testing device, the device comprising:

The question module is used to construct test questions based on the standard answers in the basic data set;

A data set module, used to construct a test data set based on the test questions and prompt word templates;

An input module, used to obtain a target model, and input the prompt words in the test data set into the target model to obtain a corresponding answer to be tested;

A unit test module, used to perform unit testing on the answer to be tested and determine the unit test result;

A result model is used to determine a model test result of the target model based on the unit test result.

An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the model testing method provided by the embodiment of the present disclosure.

The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the model testing method provided by the embodiment of the present disclosure.

The technical solution provided by the disclosed embodiment has the following advantages over the prior art: the model testing solution provided by the disclosed embodiment constructs test questions based on the standard answers in the basic data set; constructs a test data set based on the test questions and the prompt word template; obtains the target model, and inputs the prompt words in the test data set into the target model to obtain the corresponding answer to be tested; performs unit testing on the answer to be tested to determine the unit test results; and determines the model test results of the target model based on the unit test results. By adopting the above technical solution, a test data set is constructed through a basic data set, and a unit test is performed on the answer to be tested outputted by the target model based on the test data set, and then the model test results of the target model are determined according to the unit test results. The unit test results are used as the test indicators of the target model. Compared with the prior art, considering the operability characteristics of special scenarios such as code processing, it can more objectively and effectively reflect the real capabilities of the model and greatly improve the accuracy of model testing.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

FIG1 is a schematic diagram of a flow chart of a model testing method provided by some embodiments of the present disclosure;

FIG2 is a schematic flow chart of another model testing method provided by some embodiments of the present disclosure;

FIG3 is a schematic diagram of a model testing process provided by some embodiments of the present disclosure;

FIG4 is a schematic diagram of another model testing process provided by some embodiments of the present disclosure;

FIG5 is a schematic diagram of the structure of a model testing device provided in some embodiments of the present disclosure;

FIG6 is a schematic diagram of the structure of an electronic device provided in some embodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. In addition, the method embodiments may include additional steps and/or omit the steps shown. The scope of the present disclosure is not limited in this respect.

The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

In the related art, one way to test code-related models is to use a dataset based on a non-real scenario. For example, a dataset including basic algorithm questions can be used to test the pass rate of the model in the final generated code under the premise of a complete context of a given function signature and a document string (docstring). In essence, it can be considered a reduced version of the programming scenario, which is quite different from the real development scenario. Another way is to use real development warehouse data, but the test method is based on the text similarity score between the model reasoning result and the correct answer to determine whether the model result is correct. This evaluation method does not take into account the characteristics of the code itself and cannot effectively and objectively evaluate the model's capabilities. Therefore, the testing of code-related models in the related art is difficult to ensure authenticity while taking into account effectiveness, and needs to be improved.

In order to solve the above problems, the embodiments of the present disclosure provide a model testing method, which is introduced below in conjunction with specific embodiments.

FIG1 is a flow chart of a model testing method provided by some embodiments of the present disclosure, which can be performed by a model testing device, wherein the device can be implemented by software and/or hardware, and can generally be integrated in an electronic device. As shown in FIG1 , the method includes:

Step 101: Construct test questions based on standard answers in the basic data set.

The model testing method of the embodiment of the present disclosure tests the target model to obtain the processing capability or processing effect of the target model. The target model can be any model that needs to be tested for capability and needs to consider the specificity of running. The target model of the embodiment of the present disclosure can be a code processing model as an example. The code processing model is not limited to a specific one. For example, the code processing model can include a code completion model, a unit test generation model, and a code optimization model.

Among them, the basic data set can be a data set obtained by filtering based on an existing model training data warehouse. When filtering, it can be based on a specific creation time, number of collections, and single test files. When filtering based on the creation time, the model training data warehouse with a creation time after a preset time is filtered. The preset time can be set according to actual conditions, for example, it can be set to a historical time that is closer to the current time; in order to ensure the quality of the test data, the model training data warehouse with a collection number greater than a preset number can be extracted, and the preset number can be set according to actual needs, for example, the preset number can be 100; when filtering based on single test files, the model training data warehouse including single test files can be filtered to ensure that the filtered data has a probability of passing the single test. The single test files can include workflow files and test files, etc.

The basic data set may include multiple standard answers, each of which may be a separate complete data in the basic data set. For example, when the target model is a code processing model, each standard answer may be a complete code. The test topic may be a topic that can test the function of the target model. For example, when the target model is a code processing model, the test topic may be a code context.

Specifically, the model testing device can obtain a basic data set, and construct test questions based on the standard answers in the basic data set and the functions of the target model. The number of test questions can be multiple.

Exemplarily, FIG2 is a flow chart of another model testing method provided by some embodiments of the present disclosure. As shown in FIG2 , in a feasible implementation manner, constructing a test question based on the standard answer in the basic data set may include:

Step 201: construct a single test environment based on the single test environment data in the basic data set.

Among them, the unit test environment data may include environment configuration data and environment configuration instructions for executing the unit test extracted from the configuration file of the unit test execution, and the configuration file of the unit test execution may be located in the workflow file. The unit test environment may be an environment for unit testing. Specifically, an independent image may be built based on the instructions in the combined image command text using a container engine. The independent image provides the basic operating environment and running code for the unit test. Unit testing, or unit testing, refers to the process of checking and verifying the smallest testable units in an application. These units may be independent operations or functions that can be executed independently without relying on other parts of the code.

Specifically, the model testing device can query the workflow file in the basic data set, extract the single test environment data therein, and then build a single test environment based on the single test environment data for subsequent use.

Step 202: Perform unit testing on the standard answers in the basic data set based on the unit test environment to obtain unit test coverage.

The unit test coverage rate can indicate the proportion or degree of the code in the application being tested. For example, the unit test coverage rate can include line coverage rate, function coverage rate, statement coverage rate, etc. After constructing the unit test environment, the model test device can perform unit tests on each standard answer in the basic data set in the unit test environment, and then obtain the unit test coverage rate.

Step 203: Construct test questions based on the processable answers and processable positions in the processable answers determined by the unit test coverage.

Among them, the processable answer may be a standard answer to a question covered by a single test selected from the standard answer and capable of being constructed through modifications such as digging holes and injections to match the functions of the target model. The processable position may include the code position covered by the single test in the processable answer. For example, when the target model is a code completion model in a code processing model, the processable answer is the standard answer covered by the single test, and the processable position is any code line position covered by the single test; when the target model is a single test generation model or a code optimization model in a code processing model, the processable answer is the standard answer covered by the single test, and the processable position is the location of the function.

The model testing device performs parsing based on the unit test coverage, extracts one or more of the single test coverage from the standard answer as processable answers, and determines one or more positions covered by the single test in the processable answer as processable positions, and then constructs test questions based on the processable answers and processable positions. The number of test questions can be multiple, and there is no specific limit. Exemplarily, when the target model is a code completion model in a code processing model, any processable position in the processable answer can be drilled to obtain a test question; when the target model is a single test generation model, the function in the processable answer can be drilled to obtain a test question; when the target model is a code optimization model, a vulnerability (bug) can be injected into any processable position in the processable answer to obtain a test question.

In the above solution, for the code processing model, you can build a unit test environment, perform unit tests on the standard answers in the basic data set, and build test questions based on the unit test execution results.

Step 102: construct a test data set based on the test questions and prompt word templates.

The prompt word template may be a template for configuring the content and position included in the prompt word, and is used to generate the prompt word (Prompt) of the target model. The prompt word may be context information generated by accurately converting user requirements into a form that the model can effectively understand and process. The test data set may be a data set for testing the capabilities of the target model. The test data set may include multiple prompt words, and each prompt word is generated based on a test question and a prompt word template.

In some embodiments, constructing a test data set based on a test topic and a prompt word template may include: generating corresponding prompt words based on the test topic according to at least one prompt word template; and combining at least one prompt word to obtain the test data set.

The number of prompt word templates may be at least one, and different prompt word templates may include different contents. For example, one prompt word template may include only a test question, and another prompt word template may include a test question and associated information of the test question. In the disclosed embodiment, the target model includes a code processing model, the test question includes a code context, the associated information of the test question includes similar code snippets of the code context, the code context may include the code context and/or the code context, and the similar code snippet of the code context may be a code snippet with the highest similarity to the code context that is selected from a model training data warehouse where the code context is located.

After the model testing device constructs the test questions, it can obtain at least one prompt word template, combine the test questions and other related information according to the configuration of each prompt word template, generate corresponding prompt words, and then generate corresponding prompt words according to each prompt word template for multiple test questions, and combine multiple prompt words of at least one prompt word template to determine the test data set. Compared with the related art, the test data set of the embodiment of the present disclosure does not require standard answers, but only prompt words.

Step 103: Obtain a target model, and input the prompt words in the test data set into the target model to obtain the corresponding answer to be tested.

Among them, the target model can be any model that needs to be tested for capabilities and needs to consider the specificity of running through. The target model of the embodiment of the present disclosure can be a code processing model as an example. The code processing model is not limited to a specific one. For example, the code processing model can include a code completion model, a unit test generation model, and a code optimization model.

The answer to be tested can be the answer generated by calculation of the target model for the test questions of the prompt words in the test data set. The correctness of the answer to be tested needs to be verified. The number of answers to be tested is the same as the number of prompt words, which can be multiple.

When the tester needs to test the capabilities of the target model, the target model can be sent to the model testing device. After the model testing device obtains the target model, it can use the test data set that has been created to input multiple prompt words in the test data set into the target model to obtain corresponding multiple answers to be tested. It can be understood that when the same test question generates two prompt words according to two different prompt word templates, the two answers to be tested can be different. For example, when the target model is a code completion model and the test question is a code context, one prompt word template only includes the code context, then the answer to be tested, that is, the completion result of the code context, can be a defined function, while the other prompt word template includes the code context and similar code fragments of the code context, and the answer to be tested, that is, the completion result of the code context, can be a cross-file defined function.

Step 104: Perform unit testing on the answers to be tested and determine the unit test results.

Unit testing refers to the process of checking and verifying the smallest testable units in an application. These units can be executed independently without relying on other parts of the code. The purpose of unit testing is to ensure that each unit works as expected and can correctly handle input and output when interacting with other units. Unit test results can include unit test pass rate, unit test coverage, compilation and unit test run logs, etc.

Specifically, the model testing device can perform unit testing on each answer to be tested based on the cloud computing service for the above-mentioned answer to be tested, and determine the unit testing result.

In some embodiments, unit testing is performed on the answer to be tested to determine the unit test result, including: constructing multiple unit test environments using multiple services; and unit testing is performed on the answer to be tested based on the multiple unit test environments to determine the unit test result.

Among them, the service can be a cloud computing service, and can be executed without managing server response events. The embodiments of the present disclosure can perform unit testing on the answers to be tested through the service and report the results. The number of services can include multiple, each service performs a single test on an answer to be tested, or each service performs a single test on the answer to be tested corresponding to a model training data warehouse. When the model testing device performs unit testing on the answers to be tested, the answers to be tested and tasks can be distributed to multiple services. Each service uses a container engine to build a container image as a single test environment. The single test environment has an independent operating environment and a longer operating time; multiple single test environments are used to asynchronously perform unit testing on the answers to be tested to obtain complete unit test results. The unit testing of the answers to be tested is performed asynchronously by the service, which effectively improves the efficiency of unit testing.

Step 105: Determine the model test result of the target model based on the unit test result.

Among them, the model test result can be a test result that reflects the ability or accuracy of the target model. The test indicator for the target model test in the embodiment of the present disclosure can be set as a single test pass rate, and the model test result can be represented by the single test pass rate.

In an embodiment of the present disclosure, when the model testing device determines the model testing results of the target model based on the unit testing results, it can extract the single test pass rate in the unit testing results; divide the single test pass rate in the unit testing results according to at least one test dimension, and determine the single test pass rate of each test dimension as the model testing result of the target model.

Among them, the test dimension can be a specific classification that reflects the ability of the target model in a certain aspect. In the embodiment of the present disclosure, the test dimension can include at least one of the following: data set, code language, prompt word template, and scenario knowledge point, etc. The data set can be the entire test data set as a test dimension, which can determine the overall single test pass rate. The scenario knowledge point can be a knowledge point determined according to the classification corresponding to the application scenario of the target model. The single test pass rate can be the proportion of units that successfully pass the test in the process of unit testing to the total number of units. The higher the single test pass rate of a test answer, the higher the accuracy of the test answer. For all the test answers corresponding to the test data set, if the overall single test pass rate is higher, it means that more test answers have passed the verification, and the ability of the target model is higher.

The model testing device can extract the single test pass rate in the unit test results, and calculate the corresponding single test pass rate according to each test dimension in at least one test dimension. Specifically, when the test dimension is the whole, all the answers to be tested can be taken as a whole to calculate the overall single test pass rate; when the test dimension is language, the test pass rates corresponding to different code languages can be calculated respectively; when the test dimension is a prompt word template, the single test pass rates corresponding to different prompt word templates can be calculated respectively; when the test dimension is a scene knowledge point, the single test pass rates corresponding to different scene knowledge points can be calculated respectively. Afterwards, the model testing device can determine the single test pass rate of each test dimension as the model test result of the target model.

In the above scheme, when testing the target model, the evaluation indicator of the single test pass rate based on the unit test method can not only more accurately and effectively reflect the true capabilities of the target model, but also support multi-dimensional model testing and improve the diversity and flexibility of model testing.

The model testing scheme provided by the disclosed embodiment constructs test questions based on the standard answers in the basic data set; constructs a test data set based on the test questions and the prompt word template; obtains the target model, and inputs the prompt words in the test data set into the target model to obtain the corresponding answers to be tested; performs unit tests on the answers to be tested and determines the unit test results; and determines the model test results of the target model based on the unit test results. The above technical scheme is adopted to construct a test data set through the basic data set, and the answers to be tested obtained by the output of the target model are unit tested based on the test data set. Then, the model test results of the target model are determined according to the unit test results, and the unit test results are used as the test indicators of the target model. Compared with the existing technology, considering the operability characteristics of special scenarios such as code processing, it can more objectively and effectively reflect the real capabilities of the model and greatly improve the accuracy of model testing.

In some embodiments, after constructing the test questions based on the standard answers in the basic data set, the model testing method of the disclosed embodiment may further include: establishing a correspondence between the test questions and the scene knowledge points. Optionally, establishing a correspondence between the test questions and the scene knowledge points may include: classifying the background knowledge of the target model as initial knowledge points; based on the initial knowledge points, the test questions and the annotation model, determining the scene knowledge points corresponding to the test questions, and establishing a correspondence between the test questions and the scene knowledge points.

The scenario knowledge points may be knowledge points related to the application scenario of the target model, which may be determined according to the classification corresponding to the application scenario of the target model. For example, when the target model is a code processing model, the scenario knowledge points may include memory management and buffer processing in programming basics, configuration management and serialization in common tools, etc. These are just examples. The annotation model may be a model that annotates which initial knowledge point the test question belongs to, which may be implemented through a large model.

In the disclosed embodiment, after constructing the test questions, the model testing device can establish a correspondence between the test questions and the scene knowledge points. The specific establishment method can be to first obtain the background knowledge classification of the target model, and use the background knowledge classification as the initial knowledge point, and then construct a prompt word based on the initial knowledge point and the test question. Here, the prompt word is used to guide the annotation model to give the classification of the test question (that is, which initial knowledge point it belongs to) and the reason. For example, the prompt word can be expressed as "Judge the current code snippet: \n{code_snippet}\nWhich of the following knowledge points does it belong to (if it is not in the list, answer other-category name and give reasons): \n{categories}\nAnswer in the following format: \nCategory: [ ] \nReason: [ ]", which is only an example, and the prompt word is input into the annotation model, and the initial knowledge point to which the test question belongs is output as the scene knowledge point, and the correspondence between the test question and the scene knowledge point is established. The comprehensiveness of the above-mentioned test data set can be measured by the scene knowledge point. The more scene knowledge points covered by the test questions, the more comprehensive the test data set is. Later, when determining the model test results of the target model, the single test pass rates corresponding to knowledge points in different scenarios can be obtained to implement model testing from the knowledge point dimension.

Next, the model testing scheme of the embodiment of the present disclosure is further explained through a specific example. Exemplarily, Figure 3 is a schematic diagram of the model testing process provided by some embodiments of the present disclosure. As shown in Figure 3, the target model is taken as a code processing model as an example in the figure, and the model testing process is shown, which can include the three stages of data set construction, model reasoning and model testing in the figure. The data set construction stage can include warehouse screening, operating environment configuration, coverage analysis, and construction of test questions. Finally, a test data set is obtained based on the test questions and at least one prompt word template; the model reasoning stage can be implemented by a model reasoning service. After the user initiates a test on the target model, the model reasoning service can input the test data set into the target model for processing, and obtain the answer to be tested corresponding to each prompt word in the test data set, which is the model reasoning result in the figure; the model testing stage can include model reasoning result acquisition, execution of unit test, analysis of unit test results and indicator calculation, and finally obtain the single test pass rate of at least one test dimension based on the single test pass rate indicator as the model test result, and the user can obtain the model test result.

Exemplarily, FIG4 is a schematic diagram of another model testing process provided by some embodiments of the present disclosure. As shown in FIG4, the figure takes the target model including a single test generation model and a single test optimization model as an example, and shows the process of executing the single test in the model testing phase in FIG3. Specifically, it can be executed by the test processor in the figure. The test processor can distribute the answers to be tested and the test tasks to multiple cloud computing services, and can handle the current limiting errors of the cloud computing services and retry, query the task execution status of the cloud computing services, and obtain the operation results of the cloud computing services; each cloud computing service uses a container engine to build a container image as a single test environment; the warehouse in the figure represents the model training data warehouse, and the multiple test questions corresponding to the multiple prompt words included in the test data set belong to multiple warehouses. When testing multiple target models, a corresponding warehouse can be set for each target model. One warehouse can test one or more target models. The answers to be tested of different target models in the same warehouse are unit tested through single test environments constructed by different cloud computing services, and the answers to be tested are asynchronously unit tested using multiple single test environments to obtain unit test results.

This solution provides a model testing solution based on the single test pass rate as the evaluation indicator. The evaluation indicator can accurately reflect the real capabilities of the model. It selects real development warehouses and coding scenarios to improve the authenticity of the evaluation set. It supports model testing from multiple dimensions such as language, knowledge points, and prompt word templates. It also supports model testing in multiple application scenarios to improve the diversity of model testing.

FIG5 is a schematic diagram of the structure of a model testing device provided in some embodiments of the present disclosure. The device can be implemented by software and/or hardware and can generally be integrated in an electronic device. As shown in FIG5 , the device includes:

A question module 501 is used to construct a test question based on the standard answer in the basic data set;

A data set module 502, used to construct a test data set based on the test questions and prompt word templates;

An input module 503 is used to obtain a target model and input the prompt words in the test data set into the target model to obtain a corresponding answer to be tested;

The unit test module 504 is used to perform unit test on the answer to be tested and determine the unit test result;

The result module 505 is used to determine the model test result of the target model based on the unit test result.

Optionally, the topic module 501 is used to:

Building a single test environment based on the single test environment data in the basic data set;

Performing unit testing on the standard answers in the basic data set based on the unit test environment to obtain unit test coverage;

The test questions are constructed based on processable answers determined by the unit test coverage and processable positions in the processable answers.

Optionally, the data set module 502 is used to:

Generate a corresponding prompt word based on the test question according to at least one prompt word template;

The test data set is obtained by combining at least one prompt word.

Optionally, the result module 505 is used to:

Extract the unit test pass rate from the unit test results;

The unit test pass rates in the unit test results are divided according to at least one test dimension, and the unit test pass rates of each test dimension are determined as the model test results of the target model.

Optionally, the device further comprises a knowledge point module, which is used to: after constructing the test questions based on the standard answers in the basic data set,

Establish a correspondence between the test questions and scenario knowledge points.

Optionally, the knowledge point module is used to:

Classifying the background knowledge of the target model into initial knowledge points;

Based on the initial knowledge points, the test questions and the annotation model, the scenario knowledge points corresponding to the test questions are determined, and a corresponding relationship between the test questions and the scenario knowledge points is established.

Optionally, the test dimension includes at least one of the following: a data set, a code language, a prompt word template, and a scenario knowledge point.

Optionally, the prompt word template includes the test topic and associated information of the test topic;

The target model includes a code processing model, the test topic includes a code context, and the associated information of the test topic includes similar code fragments of the code context.

The model testing device provided in the embodiments of the present disclosure can execute the model testing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

The embodiments of the present disclosure also provide a computer program product, including a computer program/instruction, which implements the model testing method provided by any embodiment of the present disclosure when executed by a processor.

FIG6 is a schematic diagram of the structure of an electronic device provided in some embodiments of the present disclosure.

6, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The electronic device 600 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG6 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present disclosure.

As shown in FIG6 , the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to the bus 604.

Typically, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 6 shows an electronic device 600 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the model testing method of the embodiment of the present disclosure are executed.

It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: constructs a test question based on the standard answer in the basic data set; constructs a test data set based on the test question and the prompt word template; obtains a target model, and inputs the prompt words in the test data set into the target model to obtain the corresponding answer to be tested; performs unit testing on the answer to be tested to determine the unit test result; and determines the model test result of the target model based on the unit test result.

Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Although the subject matter has been described in language specific to structural features and/or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims (11)

1. A method of model testing, comprising:

Constructing a test question based on standard answers in the basic data set;

constructing a test data set based on the test questions and the prompt word templates;

acquiring a target model, and inputting a prompt word in the test data set into the target model to obtain a corresponding answer to be tested;

performing unit test on the answer to be tested, and determining a unit test result;

and determining a model test result of the target model based on the unit test result.

2. The method of claim 1, wherein constructing a test question based on standard answers in a base dataset comprises:

constructing a single-test environment based on the single-test environment data in the basic data set;

Performing unit test on the standard answers in the basic data set based on the single-test environment to obtain unit test coverage rate;

And constructing the test questions based on the processable answers determined by the unit test coverage rate and the processable positions in the processable answers.

3. The method of claim 1, wherein constructing a test dataset based on the test topics and a hint word template comprises:

Generating corresponding prompting words based on the test questions according to at least one prompting word template;

and combining at least one prompt word to obtain the test data set.

4. The method of claim 1, wherein determining the model test result of the target model based on the unit test result comprises:

Extracting single test passing rate in the unit test result;

Dividing the single-test passing rate in the unit test results according to at least one test dimension, and determining the single-test passing rate of each test dimension as a model test result of the target model.

5. The method of claim 1, wherein after constructing a test question based on standard answers in the base dataset, the method further comprises:

and establishing the corresponding relation between the test questions and the scene knowledge points.

6. The method of claim 5, wherein establishing the correspondence of the test topic to a scene knowledge point comprises:

Determining background knowledge classification of the target model as an initial knowledge point;

and determining scene knowledge points corresponding to the test questions based on the initial knowledge points, the test questions and the labeling model, and establishing corresponding relations between the test questions and the scene knowledge points.

7. The method of claim 4 or 5, wherein the test dimension comprises at least one of: data sets, code languages, hint word templates, and scene knowledge points.

8. The method of claim 1, wherein the alert word template includes the test question and associated information of the test question;

The object model includes a code processing model, the test question includes a code context, and the associated information of the test question includes similar code segments of the code context.

9. A model test apparatus, comprising:

the question module is used for constructing a test question based on standard answers in the basic data set;

The data set module is used for constructing a test data set based on the test questions and the prompt word templates;

The input module is used for acquiring a target model, inputting the prompt words in the test data set into the target model and obtaining corresponding answers to be tested;

the single test module is used for carrying out unit test on the answer to be tested and determining a unit test result;

and the result model is used for determining the model test result of the target model based on the unit test result.

10. An electronic device, the electronic device comprising:

A processor;

a memory for storing the processor-executable instructions;

the processor is configured to read the executable instructions from the memory and execute the instructions to implement the model test method according to any one of the preceding claims 1-8.

11. A computer readable storage medium, characterized in that the storage medium stores a computer program for executing the model test method according to any of the preceding claims 1-8.