JP2023015768A - Model evaluation method and model evaluation system - Google Patents

Abstract

To improve the accuracy of model evaluation performed by a detection AI.SOLUTION: There is provided a model evaluation method for an object model outputting output data according to input data, and includes the steps of: acquiring a plurality of properties related to the input data of the object model; acquiring an evaluation index for evaluating the object model and acquiring an evaluation index related to the plurality of properties; and outputting index specification data for specifying an evaluation index corresponding to a predetermined condition based on the plurality of properties.SELECTED DRAWING: Figure 1

Description

The present invention relates to a model evaluation method and a model evaluation system.

It is known to "apply sensor data acquired in the manufacturing process of various electronic devices such as display devices to a machine learning model to detect or predict a defective state or defect" (see, for example, Patent Document 1).
Patent Document 1 Japanese Patent Application Laid-Open No. 2020-126601

It is preferable to improve the model evaluation accuracy of detection AI.

According to a first aspect of the present invention, there is provided a model evaluation method for a target model that outputs output data according to input data, comprising: obtaining a plurality of properties relating to the input data of the target model; and evaluating the target model. a step of acquiring evaluation indicators for a property, the step of acquiring evaluation indicators related to a plurality of properties; and outputting data.

The index identification data may include sample data of input data corresponding to evaluation indices that meet predetermined conditions.

The index identification data may include a property value range corresponding to the evaluation index that satisfies a predetermined condition.

The model evaluation method may comprise obtaining distributions of evaluation indices according to changes in a plurality of properties.

An evaluation index that satisfies a predetermined condition may be an extreme value of the evaluation index distribution or an evaluation index within a predetermined range from the extreme value.

The index identification data may be information for identifying evaluation indexes to be improved that do not satisfy predetermined criteria.

The model evaluation method may comprise outputting an evaluation metric for each arbitrary property of the plurality of properties.

The plurality of properties may include continuously changing properties whose values change continuously.

The continuously changing property may indicate at least one of the position of the sensing target, the shape of the sensing target, the color of the sensing target, the light source position or the light source intensity in the input data of the target model.

The multiple properties may include non-continuously varying properties whose values do not vary continuously.

A non-continuously varying property may indicate at least one of a texture, an input image, a category, or a class to be detected in the input data of the target model.

The model evaluation method may comprise generating imputed data interpolated between non-continuously varying properties using a predetermined extended model. The model evaluation method may comprise subjecting the target model to a learning process using the imputed data.

The model evaluation method may comprise specifying non-evaluation properties that are not to be evaluated for the target model from a plurality of properties. The model evaluation method may comprise obtaining an evaluation index according to a combination of multiple properties based on the properties to be evaluated excluding the properties not to be evaluated from the multiple properties.

The target model may be a model learned based on original data and extended data extended from the original data.

The model evaluation method may comprise generating additional augmented data based on evaluation metrics that meet predetermined conditions. The model evaluation method may comprise performing a training process for the target model based on the additional augmented data.

A second aspect of the present invention provides a program for causing a computer to execute the model evaluation method according to the first aspect of the present invention.

In a third aspect of the present invention, there is provided a model evaluation system for a target model that outputs output data according to input data, comprising: An index acquisition unit that acquires an evaluation index that is an evaluation index for evaluation and is related to a plurality of properties; and an output for outputting data.

It should be noted that the above summary of the invention does not list all the features of the invention. Subcombinations of these feature groups can also be inventions.

An overview of the configuration of the model evaluation system 100 is shown. It is an example of an evaluation method using the model evaluation system 100 . It is an example of an evaluation result Er using the model evaluation system 100 . It is a modified example of the evaluation result Er using the model evaluation system 100. FIG. A modification of the model evaluation system 100 is shown. It is an example of an evaluation method using the model evaluation system 100 . An example of a method for generating extended data De will be shown. An example of a method for generating extended data De will be shown. An example of a method for generating extended data De will be shown. An example of an evaluation result Er including a discontinuous change property Pd is shown. A modification of the model evaluation system 100 is shown. It is an example of an evaluation method using the model evaluation system 100 . An example computer 2200 is shown in which aspects of the present invention may be embodied in whole or in part.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution of the invention.

FIG. 1 shows an overview of the configuration of a model evaluation system 100. As shown in FIG. The model evaluation system 100 includes an input unit 10 , an index acquisition unit 20 and an output unit 30 . The input unit 10 of this example has a target model acquisition unit 12 , a property acquisition unit 14 , and an original data acquisition unit 16 . The model evaluation system 100 evaluates a predetermined target model Mt and outputs an evaluation result Er.

The target model acquisition unit 12 acquires output data of the target model Mt. The target model acquisition unit 12 of this example has a learned target model Mt and acquires output data. The target model acquisition unit 12 may execute learning processing for the target model Mt. The target model acquisition unit 12 may acquire output data from an externally provided target model Mt without having the target model Mt.

The target model Mt outputs a detection result of detecting a detection target in accordance with predetermined input data. The target model Mt may be a detection AI that can be generated by learning processing. The type of detection model of the target model Mt is not particularly limited. The target model Mt may be an object detection model, a segmentation model, or a label estimation model.

The detection model of the target model Mt may be any one of an image detection model, an audio detection model, an action detection model, a sentence judgment model, a movie analysis model, or a plant anomaly detection model. In the case of the face detection model, the target model Mt may obtain the coordinates of the rectangular area containing the face in the image from the input image. In the case of the defect detection model, the target model Mt may detect the position or shape of the defect location to be detected.

The property acquisition unit 14 acquires property information Ip regarding the input data of the target model Mt. The property information Ip includes information about the specified property P type and property P value. The model evaluation system 100 may evaluate the target model Mt based on the output data of the property P designated by the property acquisition unit 14 .

The property information Ip may include an evaluation target property specified as an evaluation target from a plurality of properties P, or may include an evaluation non-target property that is not an evaluation target of the target model Mt. The property acquisition unit 14 can simplify the calculation and reduce the cost by specifying the non-evaluation target property. The property acquisition unit 14 may specify the property P to be evaluated according to input from the user.

By specifying the type or value of the property P, it is possible to specify a specific range and evaluate the target model Mt. The property information Ip may include any information such as sensing target properties, background properties, or environment properties. Also, the property information Ip may include information about the texture to be detected, and may include information about the mode (for example, the type of defect).

The sensed object properties may include information such as the sensed person's identity, gender, race, age or face angle. The background property may include information indicating where the sensing target is located. Environmental properties may include information about lighting conditions, presence or absence of obstructions in the face region (eg, hands, arms, glasses, masks, hair or hats, etc.), face resolution or noise application. The face resolution may be calculated according to the number of pixels between the eyes.

The property acquisition unit 14 acquires properties P for evaluating the target model Mt. The property acquisition unit 14 may specify sample points for evaluating the target model Mt from the acquired property P. FIG. A method of specifying sample points may be grid search or Gaussian process regression. When using the grid search, the property acquisition unit 14 may specify sample points at specified intervals in a predetermined range of each property P. FIG. Also, when using Gaussian process regression, sample points can be reduced compared to grid search. This makes it possible to improve comprehensiveness and continuity more efficiently. Also, a method of specifying sample points using a trained model may be used without being limited to grid search or Gaussian process regression.

The original data acquisition unit 16 acquires original data Do to be input to the target model Mt. The original data Do may be used for learning the target model Mt. The original data Do may be an image including the detection target. The original data acquisition unit 16 inputs the acquired original data Do to the index acquisition unit 20 .

The index acquisition unit 20 acquires an evaluation index Ei for evaluating the target model Mt. For example, the index acquisition unit 20 acquires the detection rate, false positive rate, or missed rate of detection AI as the evaluation index Ei. The index acquisition unit 20 in this example acquires evaluation indexes Ei related to multiple properties P. FIG. The index acquisition unit 20 may calculate the evaluation index Ei within a range corresponding to the sample points of the property P specified by the property acquisition unit 14 .

The index acquisition unit 20 may acquire the evaluation index Ei corresponding to the plurality of properties P based on the property to be evaluated from the plurality of properties P. FIG. In this case, the index acquisition unit 20 does not need to evaluate the target model Mt within a range according to the non-evaluation target property. This eliminates the need to acquire the unnecessary evaluation index Ei, and simplifies the calculation.

The output unit 30 outputs the evaluation result Er of the target model Mt. The evaluation result Er may include the relationship between the specified property P and the evaluation index Ei. The output unit 30 may have a display unit such as a display for displaying the evaluation result Er.

Based on a plurality of properties P, the output unit 30 of this example outputs index identification data Di for identifying an evaluation index Ei that satisfies a predetermined condition. For example, if the evaluation index Ei is the detection rate, the evaluation index Ei that satisfies a predetermined condition may be an evaluation index Ei with a lower detection rate than the standard required for the target model Mt.

The index specifying data Di may be input data corresponding to the evaluation index Ei that satisfies a predetermined condition. In one example, the index identification data Di is information for identifying an evaluation index Ei with a low detection rate. For example, the index specifying data Di includes sample data such as input image data of the target model Mt. By outputting the input image data as sample data, the output unit 30 can intuitively notify the user of the index corresponding to the predetermined condition.

Moreover, the index specifying data Di may include the range of the property P corresponding to the evaluation index Ei that satisfies a predetermined condition. The index specifying data Di may be a numerical range of the property P corresponding to the evaluation index Ei equal to or higher than a predetermined false positive rate. For example, the output unit 30 can visualize the range in which the target model Mt is weak by outputting the range of the property P with a low detection rate. Note that the index specifying data Di may be an image sample of the evaluation index Ei with a high detection rate, a contribution rate of the property P to the evaluation index Ei, or the like.

FIG. 2A is an example of an evaluation method using the model evaluation system 100. FIG. The model evaluation system 100 of this example evaluates the target model Mt, which is a person detection model, and displays the relationship between the evaluation index Ei and the property P value. The original data Do in this example is an image of a person to be detected, and is input to the target model Mt. The output unit 30 of this example outputs the evaluation result Er for evaluating the target model Mt based on the evaluation index Ei of the target model Mt.

The display format of the evaluation result Er is not limited as long as it is the result of evaluation of the target model Mt by the model evaluation system 100 . The evaluation result Er in this example is the distribution of the evaluation index Ei according to the property P change. The vertical axis of the evaluation result Er indicates the evaluation index Ei, and the horizontal axis indicates the property P value. For example, the evaluation result Er is the distribution of the false positive rate according to age. In the evaluation result Er of this example, the areas where the evaluation index Ei should be improved are marked with dashed circles.

FIG. 2B is an example of an evaluation result Er using the model evaluation system 100. FIG. The model evaluation system 100 uses the extreme value Ve of the distribution of the evaluation index Ei to output index specific data Di that meets a predetermined condition.

The range Rve is the value of the property P within a predetermined range from the extreme value Ve. For example, the range Rve is the range of the property P in which the evaluation index Ei is half the value of the extreme value Ve. The evaluation index Ei corresponding to the predetermined condition may be the evaluation index Ei of the extreme value Ve or the range Rve. The model evaluation system 100 of this example can visualize a range with a low detection rate using the extreme value Ve as a reference.

The model evaluation system 100 may output index specific data Di corresponding to the extreme value Ve or range Rve. The model evaluation system 100 of this example visualizes the original data Do with a low detection rate by displaying the input image of the extreme value Ve as the index specifying data Di. The model evaluation system 100 can intuitively inform the user of the original data Do with a low detection rate by displaying the input image as the index specifying data Di to the user.

FIG. 2C is a modification of the evaluation result Er using the model evaluation system 100. FIG. The model evaluation system 100 indicates the relationship between each of the multiple properties P and the evaluation index Ei.

The output unit 30 outputs an evaluation index Ei for each property P of multiple properties P. FIG. In this example, as a plurality of properties P, the age and face angle of the detection target are specified. The output unit 30 outputs the evaluation index Ei independently for each specified property P.

For example, there is a case where there is a large amount of original data Do for people aged 20 to 30, and the detection rate for people in their teens and those in their 40s or older is low. In this case, the model evaluation system 100 may increase data for age groups with a low detection rate and cause the target model Mt to perform the learning process. Thereby, the detection accuracy of the target model Mt can be improved.

The output unit 30 in this example outputs the distribution of the evaluation index Ei for one property P, but may output the distribution of the evaluation index Ei for each property P of a plurality of properties P. The output unit 30 may output the distribution of the evaluation index Ei according to the combination of multiple properties P. FIG. For example, the output unit 30 displays the distribution of the evaluation index Ei in a three-dimensional space in which multiple properties P are combined. In addition, the output unit 30 may divide the specified property P into an arbitrary number and display the average index for all data included in the divided range as the value of the range. The output unit 30 may also display the confidence interval of the evaluation index Ei.

The model evaluation system 100 can improve the continuity and comprehensiveness of the property P by visualizing data with a poor evaluation index Ei, and can efficiently learn the target model Mt. By improving comprehensiveness, the model evaluation system 100 can detect the evaluation index Ei corresponding to various properties P, and can improve the accuracy of model evaluation compared to the case of evaluation with a limited data range. By improving the continuity of the property P, the model evaluation system 100 can improve the evaluation accuracy more than discretely acquiring the evaluation index Ei.

FIG. 3 shows a modification of the model evaluation system 100. As shown in FIG. The model evaluation system 100 differs from the embodiment shown in FIG. 1 in that it includes a data extension section 22 and a storage section 40 . Other points may be the same as the embodiment of FIG.

The data extension unit 22 has a data extension model Me, and generates extended data De by extending the original data Do. The data expansion unit 22 of this example expands the original data Do based on the evaluation result Er or the evaluation index Ei output by the output unit 30 . The data extender 22 may generate the extended data De so as to complement the poor range of the evaluation index Ei. The data extension unit 22 outputs the generated extension data De to the index acquisition unit 20 .

The index acquisition unit 20 uses the original data Do and the extended data De to perform learning processing for the target model Mt. By extending and learning the original data Do in this way, it is possible to avoid the relationship between the property P and the evaluation index Ei from becoming discrete, thereby improving comprehensiveness and continuity.

The storage unit 40 may store input data such as the property P or the original data Do acquired by the input unit 10 . The storage unit 40 may store the extension data De generated by the data extension unit 22 . The data extension unit 22 may read the information stored in the storage unit 40 when extending the data.

Data extension means by the data extension unit 22 is not particularly limited. The data extension means by the data extension unit 22 may be data extension using an image transformation model or data extension by image processing. In the data enhancement by the image conversion model, data enhancement may be performed using methods such as age conversion, face angle conversion, or lighting change. In the data extension by image processing, the data may be extended using methods such as changing the resolution by resizing, applying blur, applying noise, pasting obstacles, and the like. The data extension unit 22 may extend the personality of the original data Do by using the data extension model Me. Character extension may create a person between any person A and another person B.

FIG. 4A is an example of an evaluation method using the model evaluation system 100. FIG. The model evaluation system 100 of this example generates extended data De extended from the original data Do, and evaluates the target model Mt learned with the original data Do and the extended data De. The vertical axis of the evaluation result Er indicates the false positive rate as an example of the evaluation index Ei.

The original data Do includes a non-defective product image Dog and a defective product image Dob. The data extension model Me generates an extended defective product extended image Deb according to the non-defective product actual image Dog and the defective product actual image Dob. The defective product extended image Deb is an example of the extended data De. As a result, even when the number of actual defective product images Dob is small, the defective product images can be complemented with the defective product extended images Deb, and the target model Mt can be learned.

For the target model Mt, a learning process is executed using the non-defective actual image Dog, the defective actual image Dob, and the defective extended image Deb as learning data. The evaluation index Ei is generated by evaluating the target model Mt, which has been trained using the defective product extended image Deb, with an arbitrary evaluation model. The evaluation result Er indicates the relationship between the evaluation index Ei of the target model Mt learned with the defective product extended image Deb and the property P value.

The data extension unit 22 may extend the data of the property P which has less original data Do for learning and whose evaluation index Ei is worse than a predetermined standard. The lower limit of the data amount of the extended data De may be determined such that the evaluation index Ei is improved over a predetermined standard. The upper limit of the extended data De may be determined so as to avoid an increase in data generation cost for generating the evaluation result Er. In one example, the data expansion unit 22 determines the amount of expansion data De to be generated in advance, and preferentially expands data with high uncertainty in detection accuracy.

A dashed line of the evaluation result Er indicates the evaluation result Er before data expansion. A solid line of the evaluation result Er indicates the evaluation result Er after data expansion. In the evaluation index Ei after data extension, the false detection rate is reduced as a whole by using the extension data De. In addition, the data range that was not evaluated before the data extension is evaluated in the evaluation index Ei after the data extension.

In this way, the model evaluation system 100 of this example can extend the data of an unknown image that is not included in the original data Do of the actual image. As a result, it is possible to improve the detection accuracy of the target model Mt and shorten the preparation period until introduction to the actual line.

FIG. 4B shows an example of a method for generating extension data De. The object to be detected in this example is the cap of a PET bottle. In this example, the label information L1 is used to designate the defective position of the lid of the PET bottle, and the extended data De of the defective product is generated. A non-defective actual image Dog is an image of a PET bottle cap that is free of defects.

The label information L1 includes information designating the data application area R1 at an arbitrary position of the non-defective product image Dog. The label information L1 may include information on at least one of the position, shape, and texture of the detection target. The label information L1 may be any information that can specify the position of the detection target, and may be image data, numerical data such as coordinates, or a combination thereof.

The data application region R1 designates an arbitrary position and shape on the cap of the PET bottle. The data extension model Me applies data of an arbitrary defective part to the data application area R1 to generate extended data De of the defective product. The defect in this example is a scratch on the side surface of the lid, but the type of defect mode is not limited to this. The data extension unit 22 may extend the data by applying multiple types of flaws to the data application region R1.

FIG. 4C shows an example of a method for generating extension data De. In this example, extended data De is generated by extending a defective portion of a type different from that in the example of FIG. 4B. The data application region R1 in this example indicates a missing lid. The data application area R1 is designated as the triangular area at the bottom of the lid. The shape of the data application region R1 may be another shape, or a plurality of shapes may be provided. The position, size, etc. of the data application region R1 may be appropriately changed according to the distribution of the evaluation index Ei output by designating the property P as the position and size of the defective portion.

FIG. 4D shows an example of a method for generating extension data De. In this example, the image of the lid having the defective portion is expanded to change the position of the defective portion. The model evaluation system 100 of this example changes the position of the defective portion to generate n extension data De of extension data De_1 to extension data De_n.

Here, the detection accuracy of the target model Mt also varies depending on which position in the horizontal axis direction of the image the defective portion belongs to. For example, the detection accuracy may be worse when the defective portion is located at the edge of the lid than when the defective portion is located at the center of the lid. In such a case, the model evaluation system 100 can improve the detection accuracy of the target model Mt by specifying the data application region R1 at the edge of the lid and generating the extended data De.

FIG. 4E shows an example of an evaluation result Er including non-continuously changing properties Pd. The plurality of properties P in this example includes continuously changing properties Pc and non-continuously changing properties Pd. The vertical axis indicates the false detection rate as the evaluation index Ei. The horizontal axis indicates the coordinates of the defect position as an example of the continuously changing property Pc, and the texture as an example of the non-continuously changing property Pd.

A continuously changing property Pc is a property P whose value changes continuously. For example, the continuously changing property Pc is the position, shape or color of the sensing target. The continuously changing property Pc may include the light source position or light source intensity in the space where the detection target exists. The continuous-varying property Pc may be used by normalizing the value of the property P to a range from 0 to 1, for example. When the continuously changing property Pc is age, the age value may be used as it is, or the age value may be set by dividing it into a predetermined number (for example, 20) of sample points.

A discontinuous changing property Pd is a property P whose value does not change continuously. Texture is an example of a non-continuously varying property Pd, and may include textures such as flaws or blemishes at imperfections. The non-continuous change property Pd may be at least one of a texture to be detected, an input image, a category or a class in the input data of the target model Mt.

Here, the property acquisition unit 14 may use a predetermined extension model Me to generate complementary data that complements the discontinuous changing properties Pd. When the non-continuously changing property Pd is a texture, the value of the property P may be generated by specifying the values of the property P in the number corresponding to the type of texture and interpolating each property P value. The target model acquisition unit 12 may cause the target model Mt to perform learning processing using the complementary data.

For example, the model evaluation system 100 generates a texture that looks like something between a stain and a scratch on the cap of a PET bottle, so that the distribution of the evaluation index Ei can be calculated even for a property P that has discontinuous values like the texture. can be obtained. Therefore, the model evaluation system 100 can improve the continuity and comprehensiveness of the discontinuous changing property Pd by complementing the property P value.

FIG. 5A shows a variation of model evaluation system 100 . The data extension unit 22 of this example differs from the embodiment of FIG. 3 in that it generates additional extension data Da based on the evaluation result Er. Other points may be the same as the embodiment of FIG.

The data extender 22 generates additional extended data Da based on the evaluation index Ei that satisfies a predetermined condition. The data extension unit 22 can complement the portion where the number of sample data is insufficient in the original data Do or the extension data De with the additional extension data Da. The data extension unit 22 may acquire the original data Do and the extension data De stored in the storage unit 40 when the additional extension data Da is generated. The additional extension data Da may be stored in the storage unit 40 .

The data amount of the additional extension data Da may be determined according to the data amounts of the original data Do and the extension data De. If the data amount of the additional extension data Da is smaller than the data amount of the original data Do and the extension data De, the effect of the additional extension data Da cannot be sufficiently obtained, and the data amount of the additional extension data Da is less than the data amount of the original data Do and the extension data De. Even if the amount of data is larger than that of De, the detection accuracy of the target model Mt may deteriorate.

The index acquisition unit 20 executes learning processing of the target model Mt based on the additional augmented data Da. As a result, coverage and continuity of the property P can be improved according to the evaluation result Er, and detection accuracy of the target model Mt can be improved.

FIG. 5B is an example of an evaluation method using the model evaluation system 100. FIG. The model evaluation system 100 of this example generates a defective product added extended image Dab and evaluates the target model Mt learned with the defective product added extended image Dab. The defective product additional extended image Dab is an example of additional extended data Da. The vertical axis of the evaluation result Er indicates the false positive rate as an example of the evaluation index Ei. The data expansion unit 22 of this example additionally expands the data range with a large false positive rate in the evaluation result Er. As a result, the evaluation index Ei after data expansion is improved mainly in the data range with a high false positive rate, and the bias in the distribution of the evaluation index Ei can be corrected.

Since the model evaluation system 100 of this example generates the additional extension data Da based on the evaluation result Er, it is possible to extend the data more efficiently. Moreover, the model evaluation system 100 can consistently realize both the evaluation of the target model Mt and the improvement of the detection accuracy of the target model Mt.

The model evaluation system 100 can evaluate and improve any target model Mt without being limited to the type of detection AI model. For example, the model evaluation system 100 is used to evaluate an object detection model for autonomous driving. In this case, model evaluation system 100 may specify, as property P, weather, category of object, size, color, texture, angle, pose or clothing of a person. The model evaluation system 100 may also be used to evaluate segmentation models. The model evaluation system 100 can prevent accidents in advance by visualizing evaluation results Er such as missing a pedestrian in a specific situation. The model evaluation system 100 may also be used for human behavior detection using surveillance cameras.

FIG. 6 illustrates an example computer 2200 in which aspects of the invention may be implemented in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as one or more sections of an operation or apparatus associated with an apparatus according to embodiments of the invention, or to Sections may be executed and/or computer 2200 may be caused to execute processes or steps of such processes according to embodiments of the present invention. Such programs may be executed by CPU 2212 to cause computer 2200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

Computer 2200 according to this embodiment includes CPU 2212 , RAM 2214 , graphics controller 2216 , and display device 2218 , which are interconnected by host controller 2210 . Computer 2200 also includes input/output units such as communication interface 2222, hard disk drive 2224, DVD-ROM drive 2226, and IC card drive, which are connected to host controller 2210 via input/output controller 2220. there is The computer also includes legacy input/output units such as ROM 2230 and keyboard 2242 , which are connected to input/output controller 2220 through input/output chip 2240 .

The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. Graphics controller 2216 retrieves image data generated by CPU 2212 into itself, such as a frame buffer provided in RAM 2214 , and causes the image data to be displayed on display device 2218 .

Communication interface 2222 communicates with other electronic devices over a network. Hard disk drive 2224 stores programs and data used by CPU 2212 within computer 2200 . DVD-ROM drive 2226 reads programs or data from DVD-ROM 2201 and provides programs or data to hard disk drive 2224 via RAM 2214 . The IC card drive reads programs and data from IC cards and/or writes programs and data to IC cards.

ROM 2230 stores therein programs that depend on the hardware of computer 2200, such as a boot program that is executed by computer 2200 upon activation. Input/output chip 2240 may also connect various input/output units to input/output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, and the like.

A program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from a computer-readable medium, installed in hard disk drive 2224 , RAM 2214 , or ROM 2230 , which are also examples of computer-readable medium, and executed by CPU 2212 . The information processing described within these programs is read by computer 2200 to provide coordination between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing manipulation or processing of information in accordance with the use of computer 2200 .

For example, when communication is performed between the computer 2200 and an external device, the CPU 2212 executes a communication program loaded in the RAM 2214 and sends communication processing to the communication interface 2222 based on the processing described in the communication program. you can command. The communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or an IC card under the control of the CPU 2212, and transmits the read transmission data. It sends data to the network or writes received data received from the network to a receive buffer processing area or the like provided on the recording medium.

In addition, the CPU 2212 causes the RAM 2214 to read all or necessary portions of files or databases stored in external recording media such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. Various types of processing may be performed on the data in RAM 2214 . CPU 2212 then writes back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on recording media and subjected to information processing. CPU 2212 performs various types of operations on data read from RAM 2214, information processing, conditional decision making, conditional branching, unconditional branching, and information retrieval, as specified throughout this disclosure and by instruction sequences of programs. Various types of processing may be performed, including /replace, etc., and the results written back to RAM 2214 . In addition, the CPU 2212 may search for information in a file in a recording medium, a database, or the like. For example, if a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2212 determines that the attribute value of the first attribute is specified. search the plurality of entries for an entry that matches the condition, read the attribute value of the second attribute stored in the entry, and thereby associate it with the first attribute that satisfies the predetermined condition. an attribute value of the second attribute obtained.

The programs or software modules described above may be stored on computer readable media on or near computer 2200 . Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network. do.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the scope of claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The execution order of each process such as actions, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is particularly "before", "before etc., and it should be noted that they can be implemented in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if the description is made using "first," "next," etc. for the sake of convenience, it means that it is essential to carry out in this order. not a thing

Reference Signs List 10 Input unit 12 Target model acquisition unit 14 Property acquisition unit 16 Original data acquisition unit 20 Index acquisition unit 22 Data extension unit 30 ... output section, 40 ... storage section, 100 ... model evaluation system, 2200 ... computer, 2201 ... DVD-ROM, 2210 ... host controller, 2212 ... CPU, 2214 ... RAM 2216 graphic controller 2218 display device 2220 input/output controller 2222 communication interface 2224 hard disk drive 2226 DVD-ROM drive 2230...ROM, 2240...input/output chip, 2242...keyboard

Claims (17)

A model evaluation method for a target model that outputs output data according to input data,
obtaining a plurality of properties about the input data of the target model;
obtaining a metric for evaluating the target model, the metric associated with the plurality of properties;
and outputting index identification data for identifying the evaluation index that satisfies a predetermined condition based on the plurality of properties. 2. The model evaluation method according to claim 1, wherein said index specifying data includes sample data of said input data corresponding to said evaluation index corresponding to said predetermined condition. 3. The model evaluation method according to claim 1, wherein the index specifying data includes a property value range corresponding to the evaluation index that satisfies the predetermined condition. The model evaluation method according to any one of claims 1 to 3, further comprising acquiring a distribution of said evaluation index according to changes in said plurality of properties. 5. The model evaluation method according to claim 4, wherein said evaluation index corresponding to said predetermined condition is an extreme value of a distribution of said evaluation index or an evaluation index within a predetermined range from said extreme value. The model evaluation method according to any one of claims 1 to 5, wherein the index identification data is information for identifying the evaluation index to be improved that does not satisfy a predetermined criterion. The model evaluation method according to any one of claims 1 to 6, further comprising outputting the evaluation index for each arbitrary property among the plurality of properties. 8. The model evaluation method according to any one of claims 1 to 7, wherein the plurality of properties includes continuously changing properties whose values change continuously. 9. The model of claim 8, wherein the continuously changing property indicates at least one of a position of a sensing target, a shape of the sensing target, a color of the sensing target, a light source position, or a light source intensity in the input data of the target model. Evaluation method. 10. The model evaluation method according to any one of claims 1 to 9, wherein the plurality of properties include discontinuous changing properties whose values do not change continuously. 11. The model evaluation method of claim 10, wherein the non-continuously changing property indicates at least one of a texture to be detected, an input image, a category or a class in the input data of the target model. generating interpolated data that interpolates between the non-continuously changing properties using a predetermined extension model;
The model evaluation method according to claim 10 or 11, comprising the step of causing the target model to perform learning processing using the complementary data. Designating a non-evaluation target property that is not to be evaluated by the target model from the plurality of properties;
obtaining an evaluation index corresponding to a combination of the plurality of properties based on the property to be evaluated, which excludes the non-evaluation property from the plurality of properties;
The model evaluation method according to any one of claims 1 to 12, comprising: The model evaluation method according to any one of claims 1 to 13, wherein the target model is a model learned based on original data and extended data extended from the original data. generating additional augmented data based on the evaluation index corresponding to the predetermined condition;
15. The model evaluation method according to any one of claims 1 to 14, comprising: performing a learning process of the target model based on the additional augmented data. A program for causing a computer to execute the model evaluation method according to any one of claims 1 to 15. A model evaluation system for a target model that outputs output data according to input data,
a property acquisition unit that acquires multiple properties related to the input data of the target model;
an index acquisition unit that acquires the evaluation index for evaluating the target model, the evaluation index being related to the plurality of properties;
A model evaluation system comprising: an output unit that outputs index identification data for identifying the evaluation index that satisfies a predetermined condition based on the plurality of properties.

Images