JP2019061494A - INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a discrimination model with high discrimination accuracy.
According to the present invention, a plurality of feature quantities are selected while making a combination different among a plurality of feature quantities. Then, different combinations of feature amounts are used to determine whether the data is noise data, and the determination results are integrated to determine noise data.
[Selected figure] Figure 5

Description

  The present invention relates to a technique for determining learning data used in learning of a discrimination model.

  There is a method of performing pass / fail judgment (two-class judgment of non-defective product and defective product) based on an identification model generated in advance using various feature amount groups such as average and variance of pixel values from an image obtained by photographing an object. When generating a classification model, if the label of learning data for generating a classification model is set incorrectly, the classification model can not be generated properly, and the accuracy of the pass / fail judgment is lowered. there were.

  In the method disclosed in Non-Patent Document 1, after extracting a feature amount set in advance, a data set is divided into a plurality of data sets, and divided into learning data and verification data for each data set. Then, a classifier is learned using learning data for each divided data set, and a recognition score of each data is calculated. Furthermore, the recognition score for each data is integrated, and noise data and good data are determined by threshold processing. Here, the noise data is data that is labeled as abnormal data despite being normal data, and the good data is data that seems more normal among the normal data. Then, the determined noise data and good data are removed from the learning data. By repeating the above processing a plurality of times, noise data and good data determined from the learning data are continuously removed. Finally, the learning data is determined by restoring the removed good data.

Patent No. 5414416 gazette

X. Zhu, X. Wu, Q. Chen et al., "Eliminating class noise in large datasets," 20th Int. Conf. Machine Learning, Washington, DC, August 2003, pages 920-927.

  However, in Non-Patent Document 1, after fixing the feature amount, the recognition score is obtained to determine the noise data and the good data. Since the determination of the noise data and the good data here depends on the feature amount, when the feature amount is fixed, the noise data and the good data are determined by only one reference. In the method of determining learning data based on the noise data and the good data determined in this manner, there is a problem that the identification accuracy by the identification model generated based on the learning data is low.

  The present invention is an information processing apparatus for determining learning data used when the first generation means generates a first identification model for identifying input data, and acquires a plurality of learning data. Selection means for selecting one or more feature amounts from among a plurality of types of feature amounts extracted from each of the learning data, an extraction means for extracting a plurality of types of feature amounts from each of the plurality of acquired learning data Selecting means for executing processing, second generation means for executing generation processing for generating a second identification model based on the selected feature amount, and calculating a recognition score of the generated second identification model Calculation means for executing calculation processing, the selection processing, the generation processing, and the plurality of the plurality of recognition scores obtained by performing the calculation processing a plurality of times respectively; And having a determining means for determining a training data used in generating the first identification model from the training data.

  According to the present invention, a discrimination model with high discrimination accuracy can be generated.

BRIEF DESCRIPTION OF THE DRAWINGS The figure which shows the outline of the system which performs normal abnormality determination of the target object which concerns on 1st Embodiment. FIG. 1 is a schematic block diagram showing a functional configuration of an information processing apparatus according to a first embodiment. 6 is a flowchart showing details of identification model generation processing according to the first embodiment. BRIEF DESCRIPTION OF THE DRAWINGS The schematic for demonstrating a Haar wavelet transform in 1st Embodiment. 6 is a flowchart showing details of label noise cleansing according to the first embodiment. The flowchart which shows the detail of the label noise cleansing which concerns on 2nd Embodiment. The flowchart which shows the detail of the label noise cleansing which concerns on 3rd Embodiment. The flowchart which shows the process selection process which concerns on 2nd Embodiment. The flowchart which shows the detail of the label noise cleansing which concerns on 4th Embodiment. The flowchart which shows the detail of the label noise cleansing which concerns on 5th Embodiment. A flow chart which shows details of processing of discernment model generation concerning a 6th embodiment.

First Embodiment
Hereinafter, the details of the first embodiment of the present invention will be described with reference to the drawings. In the present embodiment, erroneously labeled data is detected and removed from learning data (learning image) used when generating an identification model used for object quality determination (normality / abnormality determination). The configuration will be described.

  FIG. 1 is a diagram schematically illustrating an information processing system that performs normal / abnormal determination of an object using the generated identification model (classifier). In the figure, reference numeral 101 denotes an object, and the present system performs normal / abnormal judgment on the object 101. An image capturing apparatus (camera) 102 captures an image of the object 101. An information processing apparatus 103 extracts a predetermined feature amount from an image captured by the image capturing device 102, and the target object is normal based on the extracted feature amount and a classifier generated in advance. It is judged whether it is abnormal or not. Reference numeral 104 denotes a display device, which displays the result determined by the information processing apparatus 103. Reference numeral 105 denotes a light source, which emits light from the light source 105 to the object 101 for visualizing defects. In this state, the image capturing apparatus 102 captures an image of the object.

  Next, generation processing for generating the above-described identification model in advance will be described. Here, the configuration for generating the identification model by the system shown in FIG. 1 that actually performs the normal / abnormal determination will be described. However, the generation of the identification model may be performed by a system other than the system that makes the actual normal / abnormal determination. The information processing apparatus 103 in the system of the present embodiment determines whether the label manually attached to the image captured by the image capturing apparatus 102 is correct. Then, when it is determined that the label is incorrect, the image is not used (removed) as a learning image at the time of generating the identification model. Using the learning image determined in this manner, a discrimination model for performing normal / abnormal determination is generated by learning.

  The information processing apparatus 103 includes a hardware configuration such as a CPU, a ROM, a RAM, and an HDD, and the CPU executes a program stored in the ROM, the HD, or the like to process, for example, each functional configuration and flowchart described later To be realized. The RAM has a storage area that functions as a work area in which the CPU develops and executes a program. The ROM has a storage area for storing programs to be executed by the CPU. The HDD has a storage area for storing various data including various programs required when the CPU executes a process, data on a threshold, and the like.

  FIG. 2 is a schematic block diagram showing a functional configuration of the information processing apparatus 103 according to the present embodiment. The learning data setting unit 201 acquires information of a label determined by a person (user) with respect to a photographed image of the object 101, and adds the information to the image. Here, it is binary information indicating whether the object 101 included in the image is normal (first information) or abnormal (second information). Also, the learning data setting unit 201 calculates the feature amount from the image (learning image) to which the label is attached. In addition to the configuration in which the learning data setting unit 201 acquires learning data by generating labels by adding labels to data, the learning data setting unit 201 may be configured to obtain learning data to which labels have already been added. It may be obtained from In any case, the learning data setting unit 201 functions as an acquiring unit that acquires learning data to which a label is attached. Also, as described above, it also functions as an extraction unit that extracts a feature amount from learning data.

  The cleansing unit 202 selects and determines a learning image to be used when generating the identification model by removing the learning data (image) determined that the label given by the learning data setting unit 201 is incorrect. The feature amount selection unit 203 performs feature amount selection by performing feature amount selection on the feature amounts extracted by the learning data setting unit 201.

  The identification model learning unit 204 learns the identification model using the learning data determined by the cleansing unit 202. The parameter setting unit 205 determines the number of selections from the feature quantities ranked by the feature quantity selection unit 203 and the parameters of the identification model learned by the identification model learning unit 204 using the cross-validation method. The normal / abnormality determination unit 206 extracts a feature amount from test data (test image) based on the feature amount selection number determined by the parameter setting unit 205. Further, based on the parameters determined by the parameter setting unit 205, the normal / abnormal determination is performed by the identification model learned in advance.

  FIG. 3 is a flowchart showing the details of the identification model generation process in the present embodiment.

(Step S301: Feature extraction for learning data)
In step S301, the learning data setting unit 201 extracts a feature amount from a target region of a learning image to which a label is attached. When a plurality of feature quantities are used, for example, Haar-Wavelet (Haar Wavelet) transformation is performed on a target region of a learning image to hierarchically generate an image. The Haar wavelet transform is a process of performing frequency conversion while holding position information.

  FIG. 4 is a schematic view of the Haar wavelet transform. First, four types of filters (Equation 1-1 to 1-4) are prepared for the target region of the learning image.

  Equation 1-1 is the vertical direction number component filter (HL), equation 1-2 is the horizontal direction number component filter (LH), equation 1-3 is the diagonal direction number component filter (HH), and equation 1-4 is low frequency The component filter (LL) is shown. For the 2 × 2 pixels of the target area, the inner product is taken with the above filter. Four types of images, vertical component image, horizontal component image, diagonal direction component image, and low frequency component image, so that the 2 × 2 region is moved without overlapping and the resolution becomes half. Generate Then, from the generated low frequency component image, four types of images of the vertical direction component image, the horizontal direction component image, the diagonal direction component image, and the low frequency component image of the next layer are generated.

By repeating such image generation, four types of component images of vertical direction component image, horizontal direction component image, diagonal direction component image, and low frequency component image are generated in each layer. At this time, since the resolution is halved, it is preferable to set the image size to a multiple of 2 to the eighth power, for example, when repeating Haar wavelet transform eight times. As a result, Haar wavelet transform is performed eight times, and four images are generated from each layer, so 32 images are generated. In addition to this, since the input image is added, the following total 33 images are generated.
1) Input image 2) Longitudinal component image of each hierarchy of first to eighth hierarchy 3) Horizontal component image of each hierarchy of first to eighth hierarchy 4) Diagonal of each hierarchy of first to eighth hierarchy Direction component image 5) Low frequency component image of each of the first to eighth layers For a total of 33 types of images generated from one image, maximum value, maximum value-minimum value, and the following equation 2 Seven feature quantities shown in Equation 6 are extracted. As a result, 7 × 33 = 231 feature quantities are extracted from one image (hereinafter, the number of feature quantities extracted from one image is N). The average of the pixel values is calculated by Equation 2, the variance by Equation 3, the kurtosis by Equation 4, the skewness by Equation 5, and the geometric average by Equation 6. Note that the size of the image is an image of a pixel in the vertical direction and b pixels in the horizontal direction, and the i-th horizontal pixel value and the j-th vertical pixel value are p (i, j).

  Here, the method using the Haar-wavelet transform has been described, but other transform methods such as wavelet transform, edge extraction, Fourier transform, and Gabor transform may be used. Further, as other statistical feature quantities, statistics such as a value obtained by subtracting the minimum value from the maximum value and a standard deviation may be used. By the above processing, a plurality of types of feature quantities can be extracted from the learning image.

(Step S302: Label Noise Cleansing)
In step S302, the cleansing unit 202 cleans the label image of the learning image using the plurality of feature amounts extracted from the learning image in step S301. That is, the cleansing unit 202 determines whether the learning image is noise data (data to which an incorrect label is added), and excludes the learning image determined as noise data. This determines the training data used in generating the identification model to identify the input image. Details of the process in this step will be described later.

(Step S303: Feature Selection)
In step S303, the feature amount selection unit 203 performs feature amount ranking on the learning image subjected to label noise cleansing in step S302. As a method of ranking feature amounts, Patent Document 1 selects feature amounts suitable for classification of input data, taking into consideration the compatibility between combinations of feature amounts from a plurality of feature amounts extracted from input data. Technology is disclosed. Specifically, the first evaluation value is calculated by combining the feature amounts, the first evaluation values are compared, and the second evaluation value indicating the compatibility between the feature amounts only for the upper I feature amounts is used. Vote and rank feature quantities based on the second evaluation value.

(Step S304: Generation of Identification Model)
In step S304, the identification model learning unit 204 generates an identification model using the feature quantities ranked in step S303. The identification model is an identification model (first identification model) for identifying an unknown input image, and in the present embodiment is an identification model for performing quality determination of an object included in the input image. Here, a projection distance method, which is one of the subspace methods, is used to generate a discrimination model. The projection distance is simply the shortest distance between the feature vector in the feature space whose axis is the respective feature amount and the hyperplane (principal plane) having the direction in which the distribution of the pattern is maximized. Hereinafter, this will be specifically described using formulas.

The mean vector m of normal data and the covariance matrix Σ can be shown using the number n of normal data and the feature vector x _i . An average vector m of normal data is shown in Equation 7 and a covariance matrix Σ is shown in Equation 8.

Here, it is assumed that the i-th eigenvalue and eigenvector of Σ are λ _i and φ _i , respectively, and the eigenvalues are arranged in descending order. In this embodiment, a discrimination model is generated from the mean vector m of the normal data and the covariance matrix て using the normal data for learning.

(Step S305: Parameter Determination by Cross Confirmation Method)
In step S305, the parameter setting unit 205 determines the number of feature quantities selected by the feature quantity selection in step S303 and the projection dimension of the subspace in step S304 using the cross-validation method.

  Specifically, parameters are determined here using k-Fold cross validation. That is, the learning data is randomly divided into k, and of the k-divided data sets, the identification model is generated using (k-1) data sets, and is verified with one data set. Then, the performance of the recognition rate is evaluated while sequentially changing the parameters to be determined (the number of selected features and the number of dimensions of the subspace), and the AUC (area under curve: area under the recognition performance curve) is the best parameter. select. Incidentally, it is appropriate to set about k = 5.

(Step S306: Normal / abnormal judgment on test data)
In step S306, the normality / abnormality determination unit 206 performs feature amount extraction on test data using the feature amount selected in step S303, and using the discrimination model generated in step S304, the normality is determined according to the parameters determined in step S305. Perform an anomaly judgment. Using the principal plane calculated by Equation 7 and the average vector m calculated by Equation 8, the projection dimension number l and the feature vector of the test data

Thus, the projection distance d (x) is expressed by Equation 9 below. Here, the projection distance represented by Formula 9 is calculated, threshold processing is performed, and the normal / abnormal determination is performed.

  In the present embodiment, although the configuration for generating the identification model using the subspace method has been described, the identification model may be generated using another identifier such as SVM.

  Here, although the information processing apparatus 103 according to the present embodiment performs learning of the identification model when performing identification on actual input data after the label noise cleansing process, the information processing apparatus 103 also performs processing on the actual input data. The learning of the identification model at the time of identification may be performed by another device.

  Here, the details of the label noise cleansing process in step S302 will be described. FIG. 5 is a flowchart showing the details of the label noise cleansing process of this embodiment.

(Step S501: setting of learning data)
In step S501, M pieces of image data (learning images) are prepared (set). Here, it is assumed that M learning data are prepared, and N feature quantities are extracted from each learning data as described above. The learning data set here is all or part of the learning data acquired in S301 described above.

(Step S502: Feature Selection)
In step S502, a selection process is performed to select R feature quantities at random among the N feature quantities extracted from each of the M learning data set in step S501 (R is an integer of 1 or more). At this time, since the feature quantities are randomly selected, it is possible to reduce the calculation time required to select feature quantities, and to suppress an increase in calculation time even if the number of repetitive processes is large. The combination of R feature amounts selected in each learning image is the same.

(Step S503: Data Division)
In step S503, M pieces of learning data prepared are randomly divided into L data sets (groups) (L is an integer of 1 or more). Then, L-1 data sets are set as learning data, and one data set is set as verification data. Then, while changing the combination, L combinations of learning data and verification data are generated. Although it has been stated here that division into L data sets is premised, it is possible to use the prepared M learning images for both learning data and verification data without division. good. The order of processing in steps S502 and S503 may be reversed.

(Step S504: Identification Model Generation)
In step S504, using the data of the data set set as learning data in step S503, a generation process of generating a discrimination model by a subspace method is executed. The discrimination model generated in this step is a discrimination model (second discrimination model) used in label noise cleansing, unlike the discrimination model for identifying the input image described above. Note that the first identification model and the second identification model may be different types of models, or may be the same type of model as in this embodiment.

(Step S505: Calculation of recognition score)
In step S505, calculation processing is performed to calculate the recognition score of each data set as verification data using the identification model generated in step S504. Here, the recognition score is a score indicating that the data is normal (likelihood of being normal). That is, in the case where the label attached to the learning data is normal (first information) or abnormal (second information), the recognition score is the first information. It indicates the likelihood. In the present embodiment, the projection distance method described in step S304 is used for identification model generation and recognition score calculation, but a method different from step S304 may be used.

(Step S506: recognition score integration)
In step S506, the recognition scores for each piece of data calculated in step S505 are added together to integrate the scores. Specifically, the recognition scores calculated for each data are added to obtain the total of the recognition scores. Although it has been stated that the sum of recognition scores of verification data is determined, the sum of recognition scores may be calculated using both learning data and recognition scores of verification data.

(Step S507: noise data candidate determination)
In step S507, noise data candidates (candidates of learning images to be excluded) are determined using the recognition score calculated in step S506. From the recognition score calculated for each data, the score of only the data to which the normal label is attached is extracted, and the average value and the standard deviation of the scores of the data to which the normal label is attached are calculated. Then, a value obtained by multiplying the standard deviation of the scores by a is added to the average value of the scores to obtain a first threshold. Further, a value obtained by multiplying the standard deviation of the score by a times the average value of the score is subtracted to obtain a second threshold. It is determined whether the score of the data to which the normal label is assigned is a value between the first and second threshold values, and if it is the sandwiched data, it is determined that the data is truly normal data. On the other hand, if the value is other than that, it is determined that the data is noise data. In consideration of the fact that normal noise data is outside the distribution of normal data, threshold processing is performed by setting, for example, a = 3, and data labeled as normal is normal data or noise data. Determine if there is.

  The threshold processing is similarly performed on data to which an abnormal label is attached. First, the average value and the standard deviation of the scores of the data labeled with the abnormality are calculated, and a value obtained by multiplying the standard deviation of the scores by b is added to the average value of the scores to obtain a third threshold. In addition, a value obtained by multiplying the standard deviation of the score by b with respect to the average value of the score is subtracted to obtain a fourth threshold. It is determined whether the score of the data to which the abnormal label is attached is a value between the third and fourth threshold values, and if it is the sandwiched data, it is determined that the data is noise data. On the other hand, if the value is other than that, it is determined that the data is really abnormal data. Consider that the noise data of the abnormality is inside the normal data distribution, for example, set b = 1, perform threshold processing, and indicate whether the data with the label of the abnormality is abnormal data or noise data Determine if it is.

(Step S508: Confirmation of End Condition)
In step S508, the process from step S502 to step S507 is repeated to check whether the end condition is satisfied. As the termination condition, for example, whether or not the processing of steps S502 to S507 is repeatedly performed a predetermined number of times (for example, 100 times) or more can be mentioned.

  In the present embodiment, feature amounts are randomly selected in S502, and a discrimination model is generated based on the selected feature amounts to determine noise data. Therefore, the noise data can be determined more robustly than the configuration in which the noise data is determined based on one predetermined feature amount. Therefore, in the identification model for identifying the object, which is generated based on the learning data from which the noise data is removed, the identification can be performed with high accuracy. In addition, by selecting the feature amounts at random, it is possible to reduce the calculation time required for the repetitive processing.

  In the present embodiment, the feature amount is randomly selected in step S502. However, when S502 to S508 are executed a plurality of times, the feature amount may be selected so that the combination is different each time. You do not have to do it.

(Step S509: Determination of noise data)
In step S509, noise data is determined based on the ratio determined to be a noise data candidate with respect to the number of times of repetitive processing from step S502 to step S507. In the present embodiment, assuming that the number of times of repetitive processing from step S502 to step S507 is 100, data determined as noise data candidates at a rate of x% is determined as noise data. Here, it is preferable to set about x = 50. And the learning data (learning image) determined as noise data here is excluded from the data of object so that it may not be utilized for the production | generation of the identification model in S305.

  As described above, according to the information processing apparatus according to the present embodiment, the feature amount is selected a plurality of times while making the combination different among the plurality of feature amounts. Then, different combinations of feature amounts are used to determine whether the data is noise data, and the determination results are integrated to determine noise data. As a result, noise data can be determined robustly and accurately, so that a discrimination model with high discrimination accuracy can be generated.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, the configuration in which the feature amount is randomly selected in step S502 has been described. In the present embodiment, in order to determine noise data more accurately, it is confirmed that the process of determining noise data candidates is correctly performed in each process of the iterative process. The components already described in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 6 is a flowchart showing details of label noise cleansing processing in the present embodiment. The processes in steps S601 to S606 and S608 in FIG. 6 are the same as the processes in steps S501 to S506 and step S508 in FIG. 5 described in the first embodiment, and thus the description thereof is omitted.

(Step S 607: Determine noise data candidates and good data candidates)
In step S607, good data candidates and noise data candidates are determined from the normal data and the abnormal data.

  First, noise data candidates are determined for data labeled as normal. The determination of the noise data candidate here may be performed by, for example, threshold processing using the first and second thresholds as described in the first embodiment. Next, noise data candidates are determined for the data to which the abnormal label is attached. The determination of the noise data candidate here may be performed by, for example, threshold processing using the third and fourth thresholds as described in the first embodiment. As described above, the noise data candidate is determined from each of the data to which the normal label is attached and the data to which the abnormal label is attached.

  Next, good data candidates are determined for the data to which the normal label is assigned. Specifically, first, the score of only the data to which the normal label is attached is extracted, and the average value and the standard deviation of the scores of the data to which the normal label is attached are calculated. Then, a value obtained by multiplying the standard deviation of the score by c with respect to the average value of the scores is added to obtain a fifth threshold. Further, a value obtained by multiplying the standard deviation of the score by c with respect to the average value of the score is subtracted to obtain a sixth threshold. It is determined whether the score of the data to which the normal label is assigned is a value between the fifth and sixth threshold values, and if it is the sandwiched data, it is determined that the data is good data. For example, in consideration of the fact that normal good data are in the distribution of normal data, threshold processing is performed after setting c = 1, and good data candidates are determined from data to which normal labels are attached. .

  Finally, good data candidates are determined for the data that is labeled as abnormal. From the scores calculated for each data, the score of only the data to which the abnormal label is assigned is extracted. The average value and the standard deviation of the scores of the data labeled as abnormal are calculated, and a value obtained by multiplying the standard deviation of the scores by d is added to the average value of the scores to obtain a seventh threshold. Further, a value obtained by multiplying the standard deviation of the score by d with respect to the average value of the score is subtracted to obtain an eighth threshold. If the score of the data labeled as abnormal is not between the seventh and eighth threshold values, it is determined that the data is good data of the abnormal data. In consideration of the fact that the good data of abnormality is outside the distribution of normal data, for example, threshold processing is performed after setting f = 3, and good data candidates are determined from the data to which the label of abnormality is attached. .

  As understood from the above description, the good data candidate is learning data having a relatively high possibility of correctly recognizing the label attached to each learning image. Conversely, the noise data candidate is learning data that is relatively low in the possibility of correctly recognizing the label attached to each learning image.

(Step S609: setting of noise data)
In step S609, noise data is determined from the noise data candidates selected in step S607.

  Noise data is determined based on a ratio determined to be a noise data candidate with respect to the number of times of repetitive processing from step S602 to step S607. Here, each data is summed up whether it is a noise data candidate or not, and in the repetition from step S602 to step S607, if it is determined as noise data at a rate of x%, it is set as noise data.

(Step S610: Confirmation of noise data)
In step S610, it is checked in step S609 whether the noise data can be correctly determined. With regard to the data determined to be noise data in step S609, if it is determined as a good data candidate even once in step S607 in the repetition of steps S602 to S607, it is determined that the data is not correct noise data.

(Step S611: Resetting noise data)
In step S611, the noise data is reset by excluding the noise data determined in step S609 that is determined not to be the correct noise data in step S610.

  In the present embodiment, good data is not used in label noise cleansing, but good data may be determined from good data candidates and data cleansing may be performed using noise data and good data as in Non-Patent Document 1. .

  According to the information processing apparatus according to the present embodiment, the noise data is reset using the noise data candidate and the good data candidate in the iterative process which satisfy the selected setting criteria. As a result, noise data can be determined robustly and accurately, so that a discrimination model with high discrimination accuracy can be generated.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the present embodiment, from among the iterative processes, which process is used to determine noise data candidates is selected using a distance scale that calculates the degree of similarity related to the order of character strings. Then, noise data is determined from the noise data candidates of the selected process. The components already described in the first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

  FIG. 7 is a flowchart showing details of label noise cleansing processing in the present embodiment. The processes in steps S601 to S606 in FIG. 7 are the same as the processes in steps S501 to S506 in FIG. 5 shown in the first embodiment, and therefore the description thereof is omitted.

(Step S 707: memorize the order of recognition score)
In step S 707, the index assigned to the data in step S 705 of each iteration of the repetitive processing is used to store the order of the recognition score from the recognition score calculated for each data.

(Step S 708: Confirmation of End Condition)
In step S 708, the processing from step S 702 to step S 707 is repeated to check whether the end condition is satisfied. The processing from step S702 to step S708 is repeated to determine noise data candidates corresponding to the selected combination of feature amounts.

(Step S 709: Calculate the degree of similarity regarding the order and select the repetitive process)
In step S 709, based on the order of the recognition scores stored in step S 707, the similarity relating to the order is calculated, and among the repetitive processes, which process to use is selected. Details will be described later with reference to FIG.

(Step S710: Determination of noise data)
In step S710, noise data is determined from the noise data candidates using only the process selected in step S709. Each data is subjected to processing of selected times to count whether it is a noise data candidate or not, and when it is determined as noise data at a ratio of b%, it is determined to be noise data. Here, it is preferable to set b = about 50.

  FIG. 8 is a flowchart showing a process for selecting a process to be used to determine noise data from among the iterative processes. The processes from step S801 to step S802 will be described below.

(Step S801: Calculation of similarity matrix regarding order)
In step S801, using the evaluation value used for the similarity of the character string, a similarity matrix regarding order is calculated using the evaluation value as an element. The evaluation value used for the similarity of character strings is used to obtain the similarity regarding the order. For example, the minimum editing distance or the like is used as the evaluation value. The minimum editing distance is a distance determined based on how much change is made to the arranged number string to become another number string. The greater the number of changes, the greater the distance. Here, for the indexes arranged in descending order of recognition score, the numeral strings arranged in different repetition processing are compared, and the processes of “insertion”, “deletion” and “replacement” are performed so that they become identical numeral strings. For example, when changing to the numeral string in which repeat process 1 “3, 6, 7, 1” and repeat process 2 “5, 6, 8” are arranged, “3” in repeat process 1 is replaced with “5”, Replace "7" with "8" and delete "1". Then, the numeral string of the repeat process 1 is changed to the numeral string of the repeat process 2. Since the number of times of processing at this time is three, the minimum editing distance is three. Thus, the recognition scores of different repeated processes are compared to calculate the minimum editing distance. Although it is assumed here that the minimum editing distance is used, Levenshtein distance may be used as another distance scale. The minimum editing distance gives the same cost in each of the "insert", "delete" and "replace" processes, but the Levenshtein distance gives different costs to "insert", "delete" and "replace". The distance measure is used to calculate the score of each process of the iterative process.

Using the minimum edit distance, the iterative process i is compared with the iterative process j to calculate the similarity DIST _ij of the iterative process i and the iterative process j. Then, the similarity matrix Y related to the order is calculated using Equation 10.

(Step S802: Calculation of Scores of Repetitive Processing)
In step S802, using the similarity matrix calculated in step S801, the score of each process of the iterative process is calculated. As shown in Equation 11 below, the similarity score Y _ij is added to the similarity matrix Y for each column to obtain the sum, and the score DIST_SUM (i) for the iterative process i is calculated.

  In this way, since the score corresponding to each process of the iterative process has been calculated, the score is arranged in descending order, and the top a% is not used, and the iterative process from step S701 to step S706 is selected. At this time, it is set to about a = 10.

  According to the information processing apparatus according to the present embodiment, using the distance measure for calculating the similarity with respect to the order of the character string, the process of determining the noise data candidate is selected, and the noise of the selected process is selected. Determine noise data from data candidates. Thereby, the noise data can be determined with high accuracy.

Fourth Embodiment
Next, a fourth embodiment of the present invention will be described. In the present embodiment, noise data candidates are determined and noise data is determined with high accuracy by combining a feature amount selection method that does not require a load on repetitive processing. The same reference numerals are given to the configurations already described in the first to third embodiments, and the descriptions thereof will be omitted.

  FIG. 9 is a flowchart showing details of label noise cleansing processing in the present embodiment. The processes in steps S901 and S903 to S909 in FIG. 9 are the same as the processes in steps S501 and S503 to S509 in FIG. 5 described in the first embodiment, and thus the description thereof is omitted.

(Step S902: Select a feature selection method and select a feature)
In step S902, for the learning data set in step S901, a feature amount selection method that does not impose a load on repetitive processing is selected, and feature amount selection is performed. As a feature quantity selection method that does not require calculation load, feature quantities are evaluated one by one according to the evaluation value criteria (Bayesian error probability estimated value or intraclass variance / interclass variance ratio), and feature quantities are ordered in order from the best There is a method of selecting

Here, the Bayesian error probability estimated value will be described as one of the examples of the evaluation value. Here, let w ₁ and w ₂ denote a normal class and an abnormal class, respectively, and let X = [x ₁ ,..., X _n ] ^t be a vector having n features. The conditional probability distributions P (x | w ₁ ) and P (x | w ₂ ) in w ₁ and w ₂ corresponding to the distributions of the probability belonging to the normal class w ₁ and the abnormal class w ₂ are expressed by a histogram, from which The posterior probability distributions P (w ₁ | x) and P (w ₂ | x) are calculated. Posterior probability distribution P | a _(w i x) shown in Equation 12.

Then, a Bayesian error probability estimated value corresponding to the overlap of the posterior probability distributions P (w ₁ | x) and P (w ₂ | x) is calculated using Expression 13.

Bayes = ∫min {P (w ₁ │x), P (w ₂ │x)} dx (Equation 13)
The calculation of the probability estimated value is performed on each of the N feature quantities. The Bayesian error probability estimated value calculated here can be regarded as a combination suitable for the classification of the non-defective product and the non-defective product as the value is lower.

Next, the in-class variance / inter-class variance ratio will be described in detail. For example, in the case of 2-class problem, the two classes as _w 1, _{w 2,} characterized observed _{x0 = [x 1, x 2} , ···, x k, ···, x N] when the, The intra-class variance / inter-class variance ratio regarding the feature quantity x _k is determined. Also, let n _{i be} the number of patterns belonging to class w _i , and let m _i be the average of x _k of patterns belonging to class w _i . Furthermore, let m be the average of x _k of all patterns. At this time, intraclass dispersion

And interclass distribution

Can be calculated as Equation 14 and Equation 15.

  From Equation 14 and Equation 15, the intraclass variance / interclass variance ratio is

It can be calculated by In this way, the intraclass variance / interclass variance ratio is determined, and feature quantities are selected in descending order of values. Here, as the evaluation value, the use of the Bayesian error probability estimated value and the in-class variance / inter-class variance ratio has been described, but an evaluation value based on the deviation of the Gaussian distribution may be used.

  Next, as a method that does not apply a calculation load, there is a method of evaluating the feature quantities two by two and selecting the feature quantities two by two from the feature quantities having good evaluation criteria. Also in this case, feature selection is performed using the Bayesian error probability estimated value or the intraclass variance / interclass variance ratio.

  Lastly, there is a method disclosed in Patent Document 1 as a method that does not require a calculation load. Patent Document 1 is a method of evaluating the compatibility of combinations of feature amounts, calculating a score for each feature amount, and determining the order of selecting the feature amounts. Here too, feature value selection is performed using the Bayesian error probability estimated value or the intraclass variance / interclass variance ratio.

  Among the plurality of feature amount selection methods described above, the feature amount selection method is set in advance, or the feature amount selection method is randomly selected for each process to perform feature amount selection.

  According to the information processing apparatus of the present embodiment, feature amount selection is performed using a feature amount selection method that does not require a calculation load. In this manner, the calculation time for selecting the feature amount is reduced, and the iterative process of selecting the noise data candidate is performed. Thereby, the noise data can be determined with high accuracy.

Fifth Embodiment
Next, a fifth embodiment of the present invention will be described. In the present embodiment, it is confirmed that the processing is correctly performed by changing the feature amount to be selected for each divided data set to determine the noise data candidate. The same reference numerals are given to the configurations that have already been described in the first to fourth embodiments, and the descriptions thereof will be omitted.

  FIG. 10 is a flowchart showing details of label noise cleansing processing in the present embodiment. The processes in steps S1004 to S1009 in FIG. 9 are the same as the processes in steps S504 to S509 in FIG. 5 described in the first embodiment, and thus the description thereof is omitted.

(Step S1002: Data Division)
In step S1002, the data set in step S1001 is divided into a plurality of data sets. The learning data is randomly divided into L data sets for the selected R feature amounts. At this time, although divided into learning data and verification data, L-1 data sets are set as learning data, and one data set is set as verification data. Then, while changing the combination, a combination of L pieces of learning data and verification data is generated.

(Step S1003: feature amount selection)
In step S1003, a feature amount corresponding to the combination of L pieces of learning data and verification data divided in step S1002 is randomly selected. This generates L sets of random feature amounts.

  According to the information processing apparatus according to the present embodiment, it is confirmed that the processing is correctly performed by changing the feature amount to be selected for each divided data set and determining the noise data candidate. Thereby, the noise data can be determined with high accuracy.

Sixth Embodiment
Next, a sixth embodiment of the present invention will be described. In each of the above-mentioned embodiments, the case of generating the identification model used for the quality determination (normal / abnormal determination) of the object has been described as an example. This embodiment shows the case of generating a discrimination model used for image diagnosis. The same reference numerals are given to the configurations already described in the first to fifth embodiments, and the descriptions thereof will be omitted.

  The information processing system according to the present embodiment detects an abnormal portion characteristic of a medical image acquired by the image acquisition device. Therefore, the system includes an image acquisition apparatus and an information processing apparatus, and the information processing apparatus determines whether a medical image acquired by the image acquisition apparatus 1101, here, a medical image has a characteristic abnormal part. The hardware configuration and the functional configuration of the information processing apparatus according to the present embodiment are the same as those of the first embodiment.

  FIG. 11 is a flowchart showing details of identification model generation processing according to the present embodiment.

(Step S1101: feature amount extraction for learning data)
In step S1101, the learning data setting unit 201 generates learning data from the acquired medical image. For example, in the fundus image, a region in which an abnormal portion characteristic of diabetes is manually marked in advance is determined as abnormal data, and a local feature amount is extracted. Then, the region not marked is judged as normal data, and the local feature amount is extracted. As feature quantities, SIFT feature quantities which are one of size invariant and rotation invariant feature quantities are used. Here, learning data is generated using N-dimensional SIFT feature quantities for each region including normal and abnormal parts.

(Step S1102: Label Noise Cleansing)
In step S1102, the cleansing unit 202 cleans the learning data from the learning data generated in step S1101 using a label noise cleansing technique. The method of performing label noise cleansing is the same as that of the first embodiment.

(Step S1103: Generation of Identification Model)
In step S1103, the identification model learning unit 204 generates an identification model using the SVM for the learning data generated in step S1102.

(Step S1104: Diagnostic imaging on test data)
In step S1104, the normal / abnormality determination unit 206 extracts local regions from the test image, extracts features using SIFT feature amounts, and determines whether there is an abnormal part using the discrimination model generated in step S1103.

  According to the information processing apparatus according to the present embodiment, in diagnosis of a medical image, erroneous labeling is performed using the feature amount of the local area of the learning image used when detecting the local area of the characteristic abnormal portion. Perform label noise cleansing to remove out-of-date data. With this configuration, it is possible to perform highly accurate image diagnosis.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Processing is also feasible. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

201 Learning data setting unit 202 Cleansing unit 203 Feature amount selection unit 204 Identification model learning unit 205 Parameter setting unit 206 Normal abnormality judgment unit

Claims (16)

An information processing apparatus for determining learning data used when the first generation means generates a first identification model for identifying input data, the information processing apparatus comprising:
Acquisition means for acquiring a plurality of learning data;
Extracting means for extracting a plurality of types of feature quantities from each of the plurality of acquired learning data;
Selection means for executing a selection process of selecting one or more feature amounts from a plurality of types of feature amounts extracted from each of the learning data;
Second generation means for executing generation processing for generating a second identification model based on the selected feature amount;
Calculation means for executing calculation processing for calculating a recognition score of the generated second identification model;
It is used when generating a first identification model from among the plurality of learning data based on the plurality of recognition scores obtained by performing the selection process, the generation process, and the calculation process a plurality of times. Determining means for determining learning data;
An information processing apparatus comprising: The apparatus further comprises dividing means for dividing the plurality of learning data into a plurality of groups,
The second generation unit generates the second identification model using learning data of a part of the plurality of groups.
The information processing apparatus according to claim 1, wherein the calculation unit calculates the recognition score using learning data of remaining groups of the plurality of groups.   The information processing apparatus according to claim 1, wherein the selection unit randomly selects one or more feature amounts from the plurality of types of feature amounts.   The information processing apparatus according to claim 1, wherein the selection unit selects a feature amount from the plurality of types of feature amounts based on evaluation values of the plurality of types of feature amounts.   The information processing apparatus according to claim 4, wherein the evaluation value is a Bayesian error probability estimated value, an in-class variance, an inter-class variance ratio, or a deviation of distribution in a Gaussian distribution.   The information processing apparatus according to claim 4, wherein the evaluation value is a distance measure when the recognition score is assigned a unique index, and the index is rearranged.   The information processing apparatus according to claim 6, wherein the distance measure is a minimum editing distance or a Levenshtein distance.   8. The information processing apparatus according to claim 6, wherein the selection unit selects a feature amount from the plurality of types of feature amounts using a matrix having the distance measure as an element.   8. The information processing apparatus according to claim 6, wherein the selection unit selects a feature quantity from the plurality of types of feature quantities by converting a matrix having the distance measure as an element into a vector. .   The information processing apparatus according to any one of claims 1 to 5, wherein the second generation unit generates the second identification model using an SVM or a subspace method.   The determination means determines whether or not the learning data is excluded based on each of the plurality of recognition scores, and excludes the learning data based on a ratio determined to be excluded. The information processing apparatus according to any one of 1 to 10.   The selection unit determines whether the data is relatively likely to be correctly recognized, or the data is relatively unlikely to be likely, based on each of the plurality of recognition scores. The information processing apparatus according to any one of claims 1 to 10, wherein it is determined whether or not the learning data is excluded. The learning data is provided with a label indicating whether it is the first information or the second information,
The information processing apparatus according to any one of claims 1 to 12, wherein the recognition score indicates a likelihood that the label attached to learning data is the first information.   The information according to any one of claims 1 to 13, further comprising the first generation means for generating the first identification model based on the learning data determined by the determination means. Processing unit. An information processing method for determining learning data used when generating a first identification model for identifying input data, comprising:
Acquiring a plurality of learning data;
Extracting a plurality of types of feature quantities from each of the plurality of acquired learning data;
Executing a selection process of selecting one or more feature amounts from a plurality of types of feature amounts extracted from each of the learning data;
Executing a generation process of generating a second identification model based on the selected feature amount;
Executing a calculation process for calculating a recognition score of the generated second identification model;
It is used when generating a first identification model from among the plurality of learning data based on the plurality of recognition scores obtained by performing the selection process, the generation process, and the calculation process a plurality of times. Determining learning data;
An information processing method characterized by comprising:   A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 13.

Images