CN110928537B - Model evaluation method, device, equipment and computer readable medium - Google Patents
Model evaluation method, device, equipment and computer readable medium Download PDFInfo
- Publication number
- CN110928537B CN110928537B CN201811096522.0A CN201811096522A CN110928537B CN 110928537 B CN110928537 B CN 110928537B CN 201811096522 A CN201811096522 A CN 201811096522A CN 110928537 B CN110928537 B CN 110928537B
- Authority
- CN
- China
- Prior art keywords
- model
- tested
- check point
- construction process
- cheating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/35—Creation or generation of source code model driven
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a model evaluation method, a device, equipment and a computer readable medium, wherein the model evaluation method comprises the following steps: determining a development template for constructing a model, the development template comprising a plurality of checkpoints; acquiring construction process content of the model to be tested corresponding to each check point; and judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point. The technical scheme of the invention can reduce the cheating probability and improve the fairness and effectiveness of model evaluation.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a method, an apparatus, a device, and a computer readable medium for model evaluation.
Background
This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the development of big data and artificial intelligence (AI, artificial Intelligence), more and more enterprises require third parties to model the data and application scenarios of the enterprises. In order to obtain better model effects, the way to select a good model by a model design game is increasingly selected. In the evaluation process of the models designed by the competitors, the evaluation party can disclose the test set because the environment and the language of each model are different, the competitors input the test set into the designed models, the output result of the models is sent to the evaluation party, and the evaluation party performs scoring of the model effect based on the output result. The disclosure of the test set can cause cheating, and if some competition teams adopt manual labeling, brushing and other actions to cheat, unfairness is caused.
In order to avoid cheating, the prior art mainly analyzes the solving ideas and checks the solving codes in a mode of inviting professional specialists to answer on site, and eliminates the team of cheating. This requires a series of offline preparation and spending budgets, such as planning an arrangement debate scene, organizing debate personnel, inviting specialists, etc., which is time-efficient and costly.
Disclosure of Invention
The embodiment of the invention provides a model evaluation method, device, equipment and computer readable medium, which are used for at least solving one or more technical problems in the prior art.
In a first aspect, an embodiment of the present invention provides a method for evaluating a model, including:
determining a development template for constructing a model, the development template comprising a plurality of checkpoints;
acquiring construction process content of the model to be tested corresponding to each check point;
and judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point.
With reference to the first aspect, in a first implementation manner of the first aspect, the embodiment of the present invention, the model evaluation method further includes:
setting a plurality of construction step sub-templates included in the development template;
at least one checkpoint is set in each of the build step sub-templates.
With reference to the first aspect, in a second implementation manner of the first aspect, the determining whether the model to be tested has cheating based on the construction process content corresponding to each check point includes:
judging whether cheating exists in the construction process content corresponding to the current check point;
if the content of the construction process corresponding to the current check point is cheated, judging that the model to be tested is cheated.
With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the method according to the embodiment of the present invention determines whether the model to be tested has cheating based on the construction process content corresponding to each check point, and further includes:
if no cheating exists in the construction process content corresponding to the current check point, judging whether the model to be tested has cheating or not based on the construction process content corresponding to the next check point.
With reference to the first aspect or the first implementation manner of the first aspect or the second implementation manner of the first aspect or the third implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the embodiment of the present invention further includes:
acquiring a service interface of the model to be tested;
inputting a test set to the model to be tested based on the service interface;
and evaluating the model to be tested according to the output result of the model to be tested.
In a second aspect, an embodiment of the present invention provides a model evaluation device, including:
a determination module for determining a development template for building a model, the development template comprising a plurality of checkpoints;
the first acquisition module is used for acquiring the construction process content of the model to be tested corresponding to each check point;
and the judging module is used for judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point.
With reference to the second aspect, in a first implementation manner of the second aspect, the embodiment of the present invention further includes:
the setting module is used for setting a plurality of construction step sub-templates included in the development template;
and the setting module is used for setting at least one check point in each construction step sub-template.
With reference to the second aspect, in a second implementation manner of the embodiment of the present invention, the determining module includes:
the first judging submodule is used for judging whether cheating exists in the construction process content corresponding to the current check point;
and the judging sub-module is used for judging that the model to be tested has cheating if the content of the construction process corresponding to the current check point has cheating.
The first judging submodule is used for judging whether cheating exists in the construction process content corresponding to the current check point;
with reference to the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the embodiment of the present invention further includes:
and the second judging sub-module is used for judging whether the model to be tested has cheating or not based on the construction process content corresponding to the next check point if the construction process content corresponding to the current check point does not have cheating.
With reference to the second aspect or the first implementation manner of the second aspect or the second implementation manner of the second aspect or the third implementation manner of the second aspect, in a fourth implementation manner of the second aspect, the embodiment of the present invention further includes:
the second acquisition module is used for acquiring a service interface of the model to be tested;
the input module is used for inputting a test set to the model to be tested based on the service interface;
and the evaluation module is used for evaluating the model to be tested according to the output result of the model to be tested.
In a third aspect, an embodiment of the present invention provides a model evaluation device, including a processor and a memory, where the memory is configured to store a program for supporting the model evaluation device to execute the model evaluation method of the first aspect, and the processor is configured to execute the program stored in the memory. The modeling evaluation device may further comprise a communication interface for the modeling evaluation device to communicate with other devices or a communication network.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium storing computer software instructions for use by a model evaluation device, which includes a program for executing the model evaluation method according to the first aspect.
The embodiment of the invention can reduce the cheating probability and improve the fairness and the effectiveness of model evaluation.
The foregoing summary is for the purpose of the specification only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present invention will become apparent by reference to the drawings and the following detailed description.
Drawings
In the drawings, the same reference numerals refer to the same or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily drawn to scale. It is appreciated that these drawings depict only some embodiments according to the disclosure and are not therefore to be considered limiting of its scope.
FIG. 1 is a flow chart of a model evaluation method according to an embodiment of the present invention.
FIG. 2 is a flow chart of a method for model evaluation according to an embodiment of the present invention.
FIG. 3 is a flow chart of a method for model evaluation according to another embodiment of the present invention.
FIG. 4 is a flow chart of a method of model evaluation according to yet another embodiment of the present invention.
FIG. 5 is a flow chart of a method for model evaluation according to still another embodiment of the present invention.
Fig. 6 is a schematic structural diagram of a model evaluation device according to an embodiment of the present invention.
Fig. 7 is a schematic structural diagram of a model evaluation device according to an embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a model evaluation device according to another embodiment of the present invention.
Fig. 9 is a schematic structural diagram of a model evaluation device according to still another embodiment of the present invention.
Fig. 10 is a schematic structural diagram of a model evaluation device according to still another embodiment of the present invention.
Fig. 11 is a schematic structural diagram of a model evaluation device according to an embodiment of the present invention.
Detailed Description
Hereinafter, only certain exemplary embodiments are briefly described. As will be recognized by those of skill in the pertinent art, the described embodiments may be modified in various different ways without departing from the spirit or scope of the present invention. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.
The invention aims to provide a model evaluation method and a device, which judge whether the model to be tested is cheated or not by setting check points on a development template and acquiring the construction process content of the model to be tested corresponding to the check points, so that fairness and effectiveness of model evaluation can be ensured.
Various non-limiting embodiments of the invention are described in detail below.
As shown in fig. 1, the model evaluation method of the embodiment of the present invention may include:
s110, determining a development template for constructing a model, wherein the development template comprises a plurality of check points.
Specifically, a unified standardized development template can be set for the competitor at the beginning of the competition, and the competitor builds a model to be tested based on the development model. In addition, a plurality of checkpoints can be set on the development template by piling. For example, piling is performed where checkpointing is required to form an output interface.
S120, acquiring construction process contents of the model to be tested corresponding to each check point.
Specifically, the development template sets necessary flows for the competitor to construct the model to be tested, for example: data preprocessing, feature extraction, model creation based on algorithms, model training, and the like. If the competitor builds the model to be tested based on the development template, the content of the building process is formed. For example: result data of data preprocessing is carried out; or a plurality of extracted features; or an algorithm code based thereon; or process data for training a model.
S130, judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point.
In one possible implementation, as shown in fig. 2, in step 130, it may include:
s210, judging whether cheating exists in the construction process content corresponding to the current check point.
S220, if cheating exists in the construction process content corresponding to the current check point, judging that cheating exists in the model to be tested. After a certain check point obtains the cheating result, the cheating result can be output, the process is ended, and the subsequent check point is not judged any more.
S230, if no cheating exists in the construction process content corresponding to the current check point, judging whether the model to be tested has the cheating or not based on the construction process content corresponding to the next check point.
Specifically, the competitor may be required to first output the build process content of the current checkpoint. If the construction process content corresponding to the current check point has cheating, judging that the model to be tested has cheating, and no longer requiring competitors to output the construction process content of other check points. If no cheating exists in the construction process corresponding to the current check point, the competitor can be required to output the construction process content corresponding to the next check point, and whether the model to be tested has the cheating or not can be judged based on the construction process content corresponding to the next check point.
In addition, competitors may also be required to output the build process content of multiple or all checkpoints. If the content of the construction process corresponding to the current check point is cheated, judging that the model to be tested is cheated. If no cheating exists in the construction process content corresponding to the current check point, judging based on the construction process content corresponding to the next check point until all construction process contents corresponding to the check points are traversed.
In a possible implementation manner, as shown in fig. 3, the model evaluation method of the present embodiment may further include:
s310, setting a plurality of construction step sub-templates included in the development template.
S320, setting at least one check point in each sub-template of the construction step.
In particular, in the process of building a model, a plurality of necessary steps are usually required, and a corresponding sub-template of the building step can be built for each necessary step in the development template; one or more checkpoints are set in each build step sub-template.
The model evaluation method of the embodiment can judge whether the model to be tested has cheating in the construction process by checking whether the construction process content output at the check point comprises the content of the artificial cheating.
In a possible implementation manner, as shown in fig. 4, the model evaluation method of the present embodiment may further include:
s410, acquiring a service interface of the model to be tested.
In the process of evaluating a model to be tested, a test set is generally required to be disclosed, a competitor inputs the test set into the model to be tested, an output result of the model to be tested is returned to an evaluating party, and the evaluating party evaluates the model to be tested based on the output result.
The disclosure of the test set increases the possibility of artificial cheating of competitors in the evaluation process, and in order to avoid the artificial cheating in the model evaluation process, the competitors may be required to design a uniform service interface for the model to be tested and send the model to be tested to the evaluation party.
S420, inputting a test set to the model to be tested based on the service interface.
Specifically, through a unified service interface, the evaluating party can directly operate the model to be tested, input an internal test set into the model to be tested, and obtain an output result of the model to be tested.
S430, evaluating the model to be tested according to the output result of the model to be tested.
Specifically, the to-be-measured model may be scored based on an output result of the to-be-measured model, and an effect of the to-be-measured model may be evaluated based on the scoring result.
In a possible implementation manner, as shown in fig. 5, the model evaluation method of the present embodiment may further include:
s510, sending a test set to the model to be tested, wherein the test set comprises interference data and effective data.
Specifically, the evaluating party may disclose the test set, which is entered into the model under test by the competitor. The model to be tested can be built by competitors based on different design environments and design languages, so long as the requirements of model users are met.
The test set of the present embodiment includes interference data and valid data, wherein the interference data is typically invalid data. The invalid data can be of the same or similar type, long and short forms, numerical intervals and the like as the valid data, so that the situation that the competitors directly see which are the interference data to perform artificial cheating can be avoided.
S520, receiving an output result of the model to be tested.
Specifically, the competitor can input the disclosed test set into the model to be tested designed by himself, obtain the output result of the model to be tested, and send the output result to the evaluating party.
S530, evaluating the output result based on the valid data.
Specifically, after the output result of the model to be tested is obtained, the evaluating party can score the model to be tested based on the output result. In the scoring process, only valid data can be calculated, so that the effectiveness of scoring can be ensured.
The model evaluation method of the embodiment can judge whether the model to be tested has cheating in the construction process by checking whether the construction process content output at the check point comprises the content of the artificial cheating. In addition, the model evaluation method of the embodiment does not disclose a test set, and performs model evaluation based on a service interface of the model to be tested, so that cheating in the evaluation process is avoided. Or, the model evaluating method of the embodiment can further avoid cheating in the evaluating process by inserting interference data in the test set. In summary, the model evaluation method of the embodiment can reduce the cheating probability of the model to be tested and improve the fairness and effectiveness of model evaluation.
The embodiment of the invention also provides a model evaluation device, as shown in fig. 6, where the model evaluation device of the embodiment may include:
a determination module 110 for determining a development template for building a model, the development template comprising a plurality of checkpoints;
the first obtaining module 120 is configured to obtain content of a construction process corresponding to each check point of the model to be tested;
and the judging module 130 is configured to judge whether the model to be tested has cheating based on the content of the construction process corresponding to each check point.
In one possible implementation, as shown in fig. 7, the determining module 130 may include:
a first judging sub-module 210, configured to judge whether the content of the construction process corresponding to the current checkpoint has cheating;
a judging submodule 220, configured to judge that the model to be tested has cheating if the content of the construction process corresponding to the current check point has cheating;
and the second judging sub-module 230 is configured to judge whether the model to be tested has cheating based on the construction process content corresponding to the next check point if the construction process content corresponding to the current check point does not have cheating.
In a possible implementation manner, as shown in fig. 8, the model evaluation device of this embodiment may further include:
a setting module 310, configured to set a plurality of construction step sub-templates included in the development template;
a setting module 320 is configured to set at least one checkpoint in each of the build step sub-templates.
In a possible implementation manner, as shown in fig. 9, the model evaluation device of this embodiment may further include:
a second obtaining module 410, configured to obtain a service interface of the model to be tested;
an input module 420, configured to input a test set to the model to be tested based on the service interface;
and the first evaluation module 430 is configured to evaluate the model to be tested according to an output result of the model to be tested.
In a possible implementation manner, as shown in fig. 10, the model evaluation device of this embodiment may further include:
a sending module 510, configured to send the test set to the model to be tested;
and the receiving module 520 is configured to receive an output result of the model to be tested.
And a second evaluation module 530, configured to evaluate the output result based on the valid data.
The embodiment also provides a model evaluating device, as shown in fig. 11, which includes: memory 21 and processor 22, and memory 21 stores a computer program executable on processor 22. The processor 22 implements the model evaluation method in the above-described embodiment when executing the computer program. The number of the memories 21 and the processors 22 may be one or more.
The apparatus further comprises:
and the communication interface 23 is used for communicating with external equipment and carrying out data interaction transmission.
The memory 21 may comprise a high-speed RAM memory or may further comprise a non-volatile memory (non-volatile memory), such as at least one disk memory.
If the memory 21, the processor 22 and the communication interface 23 are implemented independently, the memory 21, the processor 22 and the communication interface 23 may be connected to each other and perform communication with each other through a bus. The bus may be an industry standard architecture (ISA, industry Standard Architecture) bus, a peripheral component interconnect (PCI, peripheral Component) bus, or an extended industry standard architecture (EISA, extended Industry Standard Component) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is shown in FIG. 11, but not only one bus or one type of bus.
Alternatively, in a specific implementation, if the memory 21, the processor 22 and the communication interface 23 are integrated on a chip, the memory 21, the processor 22 and the communication interface 23 may communicate with each other through internal interfaces.
In summary, according to the method and the device for evaluating the model in the embodiment, by setting the check point in the development template and based on the construction process content of the model to be tested corresponding to the check point, whether the model to be tested has cheating in the design process can be judged, and the effectiveness and fairness of model evaluation are improved.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
Those of ordinary skill in the art will appreciate that all or a portion of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, where the program may be stored in a computer readable storage medium, and where the program, when executed, includes one or a combination of the steps of the method embodiments.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that various changes and substitutions are possible within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (12)
1. A model evaluation method, comprising:
determining a development template for constructing a model, the development template comprising a plurality of checkpoints, wherein the development template sets a necessary flow for constructing a model to be tested for a competitor, the necessary flow comprising: data preprocessing, feature extraction, model creation based on an algorithm and model training;
acquiring construction process contents of the model to be tested corresponding to each check point, wherein the construction process contents corresponding to each check point comprise: result data of data preprocessing is carried out; or a plurality of extracted features; or an algorithm code based thereon; or process data of the training model;
judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point;
transmitting a test set to a model to be tested, wherein the test set comprises interference data and effective data;
receiving an output result of the model to be tested;
and evaluating the output result based on the valid data.
2. The model evaluation method according to claim 1, further comprising:
setting a plurality of construction step sub-templates included in the development template;
at least one checkpoint is set in each of the build step sub-templates.
3. The model evaluation method according to claim 1, wherein determining whether the model to be tested is cheating based on the construction process content corresponding to each check point comprises:
judging whether cheating exists in the construction process content corresponding to the current check point;
if the content of the construction process corresponding to the current check point is cheated, judging that the model to be tested is cheated.
4. The model evaluation method according to claim 3, wherein determining whether the model to be tested is cheating based on the construction process content corresponding to each of the checkpoints, further comprises:
if no cheating exists in the construction process content corresponding to the current check point, judging whether the model to be tested has cheating or not based on the construction process content corresponding to the next check point.
5. The model evaluation method according to any one of claims 1 to 4, further comprising:
acquiring a service interface of the model to be tested;
inputting a test set to the model to be tested based on the service interface;
and evaluating the model to be tested according to the output result of the model to be tested.
6. A model evaluation device, characterized by comprising:
the system comprises a determining module, a judging module and a judging module, wherein the determining module is used for determining a development template for constructing a model, the development template comprises a plurality of check points, the development template sets a necessary flow for constructing a model to be tested for a competitor, and the necessary flow comprises: data preprocessing, feature extraction, model creation based on an algorithm and model training;
the first obtaining module is configured to obtain content of a construction process of the model to be tested corresponding to each check point, where the content of the construction process corresponding to each check point includes: result data of data preprocessing is carried out; or a plurality of extracted features; or an algorithm code based thereon; or process data of the training model;
the judging module is used for judging whether the model to be tested has cheating or not based on the construction process content corresponding to each check point;
the transmission module is used for transmitting a test set to the model to be tested, wherein the test set comprises interference data and effective data;
the receiving module is used for receiving the output result of the model to be tested;
and the second evaluation module is used for evaluating the output result based on the valid data.
7. The model evaluation device according to claim 6, further comprising:
the setting module is used for setting a plurality of construction step sub-templates included in the development template;
and the setting module is used for setting at least one check point in each construction step sub-template.
8. The model evaluation device according to claim 6, wherein the judging module includes:
the first judging submodule is used for judging whether cheating exists in the construction process content corresponding to the current check point;
and the judging sub-module is used for judging that the model to be tested has cheating if the content of the construction process corresponding to the current check point has cheating.
9. The model evaluation device according to claim 8, wherein the judging module further comprises:
and the second judging sub-module is used for judging whether the model to be tested has cheating or not based on the construction process content corresponding to the next check point if the construction process content corresponding to the current check point does not have cheating.
10. The model evaluation device according to any one of claims 6 to 9, further comprising:
the second acquisition module is used for acquiring a service interface of the model to be tested;
the input module is used for inputting a test set to the model to be tested based on the service interface;
and the evaluation module is used for evaluating the model to be tested according to the output result of the model to be tested.
11. A model evaluation apparatus, characterized in that the apparatus comprises:
one or more processors;
a storage means for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-5.
12. A computer readable storage medium storing a computer program, which when executed by a processor implements the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811096522.0A CN110928537B (en) | 2018-09-19 | 2018-09-19 | Model evaluation method, device, equipment and computer readable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811096522.0A CN110928537B (en) | 2018-09-19 | 2018-09-19 | Model evaluation method, device, equipment and computer readable medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110928537A CN110928537A (en) | 2020-03-27 |
| CN110928537B true CN110928537B (en) | 2023-08-11 |
Family
ID=69855282
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811096522.0A Active CN110928537B (en) | 2018-09-19 | 2018-09-19 | Model evaluation method, device, equipment and computer readable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928537B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113902296B (en) * | 2021-10-09 | 2022-06-07 | 鹤山市民强五金机电有限公司 | Intelligent test method and system for single-phase asynchronous motor |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20090008044A (en) * | 2007-07-16 | 2009-01-21 | (주)엔인포메이션시스템즈 | Data Mining Method |
| CN102549576A (en) * | 2009-08-17 | 2012-07-04 | 费兹孔克公司 | Audit device |
| CN103729351A (en) * | 2012-10-10 | 2014-04-16 | 阿里巴巴集团控股有限公司 | Search term recommendation method and device |
| CN104598709A (en) * | 2013-10-31 | 2015-05-06 | 大连智友软件科技有限公司 | Detection data fusion method based on extended OODA model |
| CN104615680A (en) * | 2015-01-21 | 2015-05-13 | 广州神马移动信息科技有限公司 | Method and device for establishing web page quality model |
| CN105931632A (en) * | 2016-06-13 | 2016-09-07 | 科大讯飞股份有限公司 | Oral exam cheating detection method and device |
| CN106815452A (en) * | 2015-11-27 | 2017-06-09 | 苏宁云商集团股份有限公司 | A kind of cheat detection method and device |
| CN106874741A (en) * | 2017-01-04 | 2017-06-20 | 北京三快在线科技有限公司 | A kind of same device identification method of business, device, computing device and storage medium |
| CN106971044A (en) * | 2017-03-31 | 2017-07-21 | 青岛理工大学 | Tailing dam piping damage scene construction system and method |
| CN107153656A (en) * | 2016-03-03 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of information search method and device |
| CN107766500A (en) * | 2017-10-19 | 2018-03-06 | 国网冀北电力有限公司 | Audit method of fixed asset card |
| CN108306864A (en) * | 2018-01-12 | 2018-07-20 | 深圳壹账通智能科技有限公司 | Network data detection method, device, computer equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080109272A1 (en) * | 2006-11-08 | 2008-05-08 | Anshul Sheopuri | Apparatus, System, Method and Computer Program Product for Analysis of Fraud in Transaction Data |
| US9076342B2 (en) * | 2008-02-19 | 2015-07-07 | Architecture Technology Corporation | Automated execution and evaluation of network-based training exercises |
-
2018
- 2018-09-19 CN CN201811096522.0A patent/CN110928537B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20090008044A (en) * | 2007-07-16 | 2009-01-21 | (주)엔인포메이션시스템즈 | Data Mining Method |
| CN102549576A (en) * | 2009-08-17 | 2012-07-04 | 费兹孔克公司 | Audit device |
| CN103729351A (en) * | 2012-10-10 | 2014-04-16 | 阿里巴巴集团控股有限公司 | Search term recommendation method and device |
| CN104598709A (en) * | 2013-10-31 | 2015-05-06 | 大连智友软件科技有限公司 | Detection data fusion method based on extended OODA model |
| CN104615680A (en) * | 2015-01-21 | 2015-05-13 | 广州神马移动信息科技有限公司 | Method and device for establishing web page quality model |
| CN106815452A (en) * | 2015-11-27 | 2017-06-09 | 苏宁云商集团股份有限公司 | A kind of cheat detection method and device |
| CN107153656A (en) * | 2016-03-03 | 2017-09-12 | 阿里巴巴集团控股有限公司 | A kind of information search method and device |
| CN105931632A (en) * | 2016-06-13 | 2016-09-07 | 科大讯飞股份有限公司 | Oral exam cheating detection method and device |
| CN106874741A (en) * | 2017-01-04 | 2017-06-20 | 北京三快在线科技有限公司 | A kind of same device identification method of business, device, computing device and storage medium |
| CN106971044A (en) * | 2017-03-31 | 2017-07-21 | 青岛理工大学 | Tailing dam piping damage scene construction system and method |
| CN107766500A (en) * | 2017-10-19 | 2018-03-06 | 国网冀北电力有限公司 | Audit method of fixed asset card |
| CN108306864A (en) * | 2018-01-12 | 2018-07-20 | 深圳壹账通智能科技有限公司 | Network data detection method, device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于云模型的软件评测模型的研究;刘洋;中国优秀硕士学位论文全文数据库 工程科技I辑(第2期);第B019-459页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110928537A (en) | 2020-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110990297B (en) | Test method and device of interlocking system and electronic equipment | |
| CN108121795A (en) | User's behavior prediction method and device | |
| CN107682368B (en) | Verification method, client, server and system based on interactive operation | |
| CN110134375B (en) | Game character behavior control method and device and readable storage medium | |
| CN107315682A (en) | Test browser-safe method, device, storage medium and electronic equipment | |
| CN112370772B (en) | Game task processing method and device and electronic equipment | |
| CN116991711A (en) | Test case generation method and device, terminal equipment and storage medium | |
| CN112015634A (en) | Page structure information generation method and device and electronic equipment | |
| CN109657038A (en) | The method for digging, device and electronic equipment of a kind of question and answer to data | |
| CN109548691A (en) | A kind of pet recognition methods, device, medium and electronic equipment | |
| CN113672389A (en) | Server compatibility method, system, equipment and computer readable storage medium | |
| CN117763128A (en) | Man-machine interaction data processing method, server, storage medium and program product | |
| CN117951275A (en) | A method and device for evaluating a multimodal large language model | |
| CN118861218A (en) | Sample data generation method, device, electronic device and storage medium | |
| CN116629988A (en) | Bid data processing method and device based on blockchain | |
| CN110928537B (en) | Model evaluation method, device, equipment and computer readable medium | |
| CN110737900B (en) | Webpage function testing method and device, server and computer readable storage medium | |
| CN116343007A (en) | Target detection method, device, equipment and storage medium | |
| CN115147127A (en) | Question answering data mining method, device and equipment | |
| CN110659204A (en) | Code detection method, device and medium | |
| CN111175055B (en) | Automatic driving distributed collaborative simulation method and device and terminal | |
| CN110837562B (en) | Case processing method, device and system | |
| CN114493360A (en) | Process creative evaluation method, device, equipment and medium based on RPA and AI | |
| CN114065005A (en) | A system configuration optimization parameter method, device, electronic device and storage medium | |
| CN110929967A (en) | Model evaluation method, device, equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |