WO2025052642A1 - Training information creation assistance device, image classification device, and program - Google Patents

Abstract

This training information creation assistance device comprises: a category score calculation unit that uses a segmentation model for an image which is composed of a plurality of pixels to calculate, for each combination of a pixel and a category, a category score indicating the degree of confidence that the pixel belongs to the category; an inference image determination unit that calculates an inference image on the basis of the category score; an adjacent category calculation unit that receives input of a category of interest and calculates, on the basis of the inference image and the category of interest, an adjacent category which is likely to be adjacent to the category of interest; and a correction-required score calculation unit that determines a correction-required score for at least one pixel on the basis of the category of interest, the adjacent category, and the category score.

Description

Teacher information creation support device, image classification device and program

The present invention relates to a teacher information creation support device, an image classification device, and a program for reducing the cost of creating teacher data in, for example, an image classification system with a learning function.

　Creating a pixel-by-pixel image classification model using deep learning requires teacher information, but the cost of adding teacher information is high. As a countermeasure, there is interactive segmentation, where the model learns in collaboration with the user by having the user indicate the parts and content of corrections to be made to the model's inference results. One possible way to select the parts to be corrected is to select parts with a low probability of inference from among the parts inferred as the focus category, but since parts that are not inferred as the focus category are not selected, false negative parts are not selected and the accuracy does not improve.

International Publication No. 2020/129235 International Publication No. 2020/189269

In Patent Document 1, the tendency for inference probability to decrease is utilized to select areas with low model inference probability as candidates for correction areas, but misclassified areas unrelated to the target category may be selected, making it difficult to improve the accuracy of the target category.

In Patent Document 2, among the parts classified as the target category, parts with a low probability of inference are selected as candidates for correction parts, but parts that are not inferred to be in the target category are not selected, so false negative parts are not selected and accuracy is not improved.

The purpose of this invention is to present areas requiring correction, including false negatives, that are expected to improve the accuracy of the focus category.

An example of a teacher information creation support device according to the present invention is:
a category score calculation unit that calculates, for an image consisting of a plurality of pixels, a category score indicating a degree of certainty that each combination of a pixel and a category belongs to the corresponding category, using a segmentation model;
An inference image determination unit that calculates an inference image based on the category score;
an adjacent category calculation unit that receives an input of a focus category and calculates an adjacent category that is likely to be adjacent to the focus category based on the inference image and the focus category;
a correction-required score calculation unit that determines a correction-required score for at least one pixel based on the focus category, the neighboring categories, and the category scores;
has.

An example of an image classification device according to the present invention includes:
an image category inference unit that uses a segmentation model to calculate a category score indicating a degree of certainty that a pixel belongs to a category for each combination of a pixel and a category for an image consisting of a plurality of pixels, and calculates an inferred image based on the category score;
A teacher information creation GUI having a focus category setting unit;
having
The attention category setting unit receives an input of an attention category,
The image classification device calculates a portion to be corrected based on the attention category and the category score,
The teacher information creation GUI includes:
Output the part that needs to be corrected,
The teacher information is accepted via the teacher information providing unit.

One example of the program according to the present invention causes a computer to function as the teacher information creation support device described above.

One example of the program according to the present invention causes a computer to function as the image classification device described above.

The present invention can suggest areas that need correction to improve accuracy against false negatives in a category of interest.

1 is an example of a functional configuration of a teacher information creation support device according to a first embodiment of the present invention. 13 shows an example of processing by an adjacent category calculation unit. 13 shows an example of a processing result of an adjacent category calculation unit. 13 illustrates an example of a correction-required score calculation unit. Another example of how to calculate uncertainty. Another example of how to calculate the Needs Revision score. Yet another example of how to calculate the Needs Revision score. 13 is an example of a functional configuration of an image classification device according to a second embodiment of the present invention. 13 is an example of a functional configuration of an image classification device according to a third embodiment of the present invention. 13 is another example of a functional configuration of an image classification device according to a third embodiment of the present invention. 13 is an example of a teacher information creation GUI. 13 is another example of a teacher information creation GUI.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[Example 1]
1 shows an example of the functional configuration of a teacher information creation support device according to a first embodiment of the present invention. First, an overview of the functional configuration in Fig. 1 will be given. The teacher information creation support device 100 receives an input image D1, a segmentation model D2, and a focus category D4 as inputs, and includes a category score calculation unit 101, an inference image determination unit 102, an adjacent category calculation unit 103, and a correction required score calculation unit 104.

The category score calculation unit 101 uses a segmentation model D2 to calculate a category score D3 for each combination of pixel and category for an input image D1 consisting of multiple pixels, which indicates the degree of certainty that the pixel belongs to that category. The category score D3 is a vector having values indicating the likelihood of each category being identified for each pixel in the input image D1.

The inference image determination unit 102 calculates an inference image D5 based on the category score D3. The inference image D5 is an image in which a single category has been identified for each pixel through inference. Specifically, the inference image D5 can be calculated by determining the category with the maximum value for the category score D3 for each pixel as the identification category.

The adjacent category calculation unit 103 receives the input of the focused category D4, and calculates the adjacent category D6 based on the inference image D5 and the focused category D4. An adjacent category means a category that is likely to be adjacent to the focused category (i.e., a category with a high adjacent likelihood), and a specific example will be described later.

The correction score calculation unit 104 determines the correction score D7 based on the focused category D4, the adjacent category D6, and the category score D3. The correction score is calculated, for example, for a portion that is difficult for the segmentation model D2 to identify in relation to the focused category, and a specific example will be described later.

The teacher information creation support device 100 can be configured, for example, using a computer. The computer has a hardware configuration as a known computer, and includes, for example, a calculation means and a storage means. The calculation means includes, for example, a processor, and the storage means includes, for example, a storage medium such as a semiconductor memory device and a magnetic disk device. Some or all of the storage medium may be a non-transitory storage medium.

The computer may also be equipped with input/output means. The input/output means may include, for example, input devices such as a keyboard and a mouse, output devices such as a display and a printer, and communication devices such as a network interface.

The storage means may store a program. When the processor executes this program, the computer functions as the teacher information creation support device 100 according to this embodiment or a device according to another embodiment.

The details of each component in Figure 1 are described below.

The input image D1 is an image for which a correction score is to be calculated. For at least one pixel (pixel or group of pixels) in this image, a correction score that is thought to promote improvement of the accuracy of the segmentation model D2 is calculated.

Segmentation model D2 is a trained model that has undergone prior machine learning and has the ability to discriminate images. However, the prior training may be performed using a dataset that has similar characteristics to input image D1, or may be performed using a dataset that is unrelated to input image D1.

The segmentation model D2 calculates features for each pixel of the input image D1, and calculates the confidence level when classifying each category from the features. The confidence level may be a value that indicates the inference probability after being normalized for each pixel using a softmax function, or it may be a logit function before being normalized using softmax. Either is acceptable.

The focus category D4 indicates the category that should be prioritized for improving accuracy among the categories identified by the segmentation model. For example, in the case of a semiconductor measurement image, this would be a category that indicates the structure of the measurement target.

FIG. 2 shows an example of the processing of the adjacent category calculation unit. First, in processing step S201, the adjacent category calculation unit 103 extracts adjacent pixel groups.

First, the adjacent category calculation unit 103 extracts pixels from the inference image D5 that belong to the attention category. Then, for each pixel inferred to belong to the attention category, pixels adjacent to that pixel are extracted as adjacent pixels (however, if the adjacent pixels also belong to the attention category, those pixels are not extracted as adjacent pixels).

Next, in processing step S202, the adjacent category calculation unit 103 determines adjacent categories. Here, the adjacent category calculation unit 103 obtains the frequency of categories in the inference image D5 for the adjacent pixel groups, and determines the category with a high frequency as the adjacent category for the focused category D4. In this way, the category that is likely to be adjacent to the focused category D4 is determined. Here, if there are multiple frequently occurring categories, it is also possible to determine multiple adjacent categories. In this way, it is possible to calculate the categories that are likely to be adjacent to the focused category D4.

In this way, in this embodiment, a category that is likely to be adjacent to the focused category D4 refers to a category that frequently has pixels adjacent to pixels belonging to the focused category D4 in the inference image D5. The definition of "frequent" can be determined appropriately by those skilled in the art, but it refers to, for example, a case where the frequency exceeds a predetermined threshold.

Figure 3 shows an example of the processing results of the adjacent category calculation unit. Figure 3 shows an example in which an inference image in which each pixel is classified as category 1, 2, or 3 is input, and the focus category is category 1.

First, in S201, the group of pixels adjacent to the location inferred to be category 1 is calculated as adjacent pixel group 300. Next, in S202, the frequency of the category to which adjacent pixel group 300 belongs is calculated, and category 2, which is the category with the highest frequency, is determined as the adjacent category.

For example, if the category indicates the structure of the measurement target in a semiconductor measurement image where similar structures are repeated within the image, the number of categories adjacent to a particular category is limited, so the above method can be used to determine the most frequent category as the adjacent category.

For example, if the white lines in an image of a road are set as the focus category, the asphalt parts of the road are frequently seen as an adjacent category, and the above method will register the asphalt parts as an adjacent category.

FIG. 4 shows an example of the correction score calculation unit 104. First, in processing step S401, the correction score calculation unit 104 extracts a category score related to the attention category D4 and a category score related to the adjacent category D6 for each pixel inferred to belong to the attention category D4 in the inference image D5 and each pixel inferred to belong to the adjacent category D6 in the inference image D5.

Then, in processing step S402, the correction score calculation unit 104 calculates uncertainty based on the extracted category scores as a value indicating the degree to which the segmentation model is insufficiently trained (e.g., contains insufficiently trained features). For example, the uncertainty is calculated as the smallness of the category scores for the specified categories (focus category D4 and adjacent category D6) from among the extracted category scores.

For example, if the category score is an inference probability, then the uncertainty value is 1 - inference probability (the value obtained by subtracting that inference probability from 1). Generally, in a trained segmentation model, the inference probability of a pixel with sufficiently trained features will be large, and the inference probability of a pixel with insufficiently trained features will be small, so it can be said that areas where 1 - inference probability is large (areas with large uncertainty) are areas where learning is insufficient.

By inputting the focus category and adjacent categories as the specified categories, the uncertainty of the part inferred to be the focus category and the uncertainty of the part inferred to be the adjacent category are obtained. Then, based on these uncertainties, processing step S403 identifies parts that need correction and have a large degree of uncertainty, and calculates a correction score for the identified parts that need correction.

The parts requiring correction may be parts where the uncertainty is equal to or greater than a predetermined threshold. A specific method for calculating the score requiring correction may be designed as appropriate by a person skilled in the art, and may be, for example, the sum of the uncertainty related to the focus category and the uncertainty related to the adjacent category.

　Areas with large uncertainty values indicate areas that are difficult for the segmentation model to identify, and by prioritizing learning from these areas, it is expected that the segmentation model's identification accuracy will improve. This makes it possible to calculate the score that needs correction from candidates that include adjacent categories and inferred areas that may result in false negatives, in addition to the focus category and inferred areas that may result in false positives.

Figure 5 shows another example of a method for calculating uncertainty. As shown in Figure 5, a method for calculating uncertainty regarding adjacent categories is to use the magnitude of the category score for the focus category as the uncertainty at an inferred location that belongs to the adjacent category.

In the example of FIG. 5, in processing step S501, the correction score calculation unit 104 extracts a group of category scores for each pixel inferred to belong to an adjacent category in the inference image. Then, in processing step S502, the uncertainty (second uncertainty) is calculated based on the category score related to the focused category from the extracted group of category scores. The second uncertainty can be, for example, the magnitude of the category score related to the focused category.

Note that for the parts inferred to belong to the attention category, the same calculation method as in FIG. 4 can be used. That is, the correction score calculation unit 104 calculates the uncertainty (first uncertainty) for each pixel inferred to belong to the attention category in the inference image, based on the category score related to the attention category.

In this case, the correction score calculation unit 104 calculates the correction score based on the first uncertainty and the second uncertainty.

In places where there are false negatives, the input image may contain features of the target category. In other words, in such places, even if the category with the highest category score is an adjacent category, the category score of the target category may also be high. Therefore, it is possible to calculate the score requiring correction only for places within the adjacent category that are more likely to be false negatives for the target category.

In Example 1, the method described above makes it possible to calculate a score requiring correction that is expected to improve accuracy with respect to false negatives as well as false positives.

FIG. 6 shows another example of a method for calculating the correction-needed score. In the example of FIG. 6, in processing step S601, the correction-needed score calculation unit 104 smoothes the correction-needed score D7 for each pixel within a predetermined range including the pixel, and calculates the part where the value of the smoothed correction-needed score D8 is large as the part that needs correction.

When a user makes corrections, it is more efficient to correct a large range at once than to correct only one isolated pixel, so by calculating the points with a high correction score within a specified range, it is possible to perform efficient corrections. A specific method for determining the specified range and a specific calculation for smoothing within the specified range can be designed appropriately by a person skilled in the art, but for example, a two-dimensional convolution process with a Gaussian function or a simple average calculation process may be used.

FIG. 7 shows yet another example of a method for calculating the correction-needed score. In the example of FIG. 7, the correction-needed score calculation unit 104 identifies a portion having a high correction-needed score as a correction-needed portion D9 in processing step S701. The definition of a portion having a high correction-needed score can be appropriately determined by a person skilled in the art, but it may be, for example, a portion having a correction-needed score equal to or greater than a threshold value, or an area having a high standard deviation value of the correction-needed score.

By determining the areas that need correction in this way, users can efficiently identify areas that have a high correction score.

[Example 2]
In the second embodiment, a GUI is proposed for creating teacher information using the correction-required score calculated in the first embodiment. Description of parts common to the first embodiment may be omitted.

FIG. 8 shows an example of the functional configuration of an image classification device according to the second embodiment of the present invention. The image classification device 800 also functions as a teacher information creation support device similar to that of the first embodiment. For example, the image classification device 800 may further have a configuration similar to that of the teacher information creation support device 100 of FIG. 1.

First, to outline the functional configuration in Figure 8, the image classification device 800 receives an input image D801 and a segmentation model D802, and is equipped with an image category inference unit 801 and a teacher information creation GUI 802.

The image category inference unit 801 uses a segmentation model D802 to calculate a category score D804 indicating the degree of certainty that a pixel belongs to a category for each combination of pixel and category for an input image D801 consisting of multiple pixels, and calculates an inference image D803 based on the category score D804. The inference image D803 and category score D804 are calculated in the same way as the inference image D5 and category score D3 in Figure 1, respectively.

The teacher information creation GUI802 is a GUI intended for the user to create teacher information D805 for each pixel using the input image D801, inference image D803, and category score D804. The teacher information creation GUI802 has a function of simultaneously displaying the input image D801 and inference image D803 to make it easier to find errors in the inference results, and a function of having the machine find areas that need correction from the category score D804, making it easy to select the areas to be corrected.

Figure 11 shows an example of a teacher information creation GUI 802. The teacher information creation GUI has an input image display unit that displays the input image D801, an inference image display unit that displays the inference image D803, a teacher information attachment unit that has a function of attaching teacher information creation, a focus category setting unit that accepts input of a focus category, a correction-required part display unit that outputs (for example, displays) a part that needs correction calculated based on the teacher information creation support device 100 shown in Figure 1 from the input focus category, input image, and category score, an uncertainty display unit that similarly displays the uncertainty calculated, and a model learning start unit that accepts an operation to start learning the segmentation model.

The uncertainty display section and the section displaying areas requiring correction are displayed as reference information when adding supervised information, and may not be necessary.

The device may also have an automatic correction part setting unit that accepts operations for automatically calculating the correction parts.

The image classification device 800 can calculate the areas that need to be corrected based on the focus category and category score, for example, in a manner similar to that of the teacher information creation support device 100 of Example 1.

The teacher information creation GUI 802 accepts the assignment of teacher information via the teacher information assignment unit. The teacher information assignment unit can set a category for each pixel using a click process on a computer or a pen tablet. In the example of Figure 11, the input image is displayed as the background, and the assigned teacher information is displayed superimposed on the foreground of the input image, but it is not necessary to display the input image as the background. In addition, it may have a function to assign the same category to multiple areas requiring correction with a single operation.

The uncertainty display section displays information to be used as a reference when adding teacher information. Similar to the teacher information creation support device 100 of Example 1, the image classification device 800 calculates the uncertainty that indicates the degree to which the learning of the segmentation model is insufficient, and the uncertainty display section displays this uncertainty. In the example of Figure 11, white areas represent areas with high uncertainty.

　Areas that are not selected as areas requiring correction but have relatively high uncertainty are often areas where the inference is incorrect, and even if there is no error, the segmentation model is often insufficiently trained. For this reason, the user can improve the accuracy of the segmentation model by referring to the display in the uncertainty display section and adding teaching information to areas with relatively high uncertainty. However, if areas requiring correction are selected automatically, it is not necessary to display the uncertainty display section.

The uncertainty display unit may also display the uncertainty in the focus category (first uncertainty) and the uncertainty in the adjacent category (second uncertainty) shown in FIG. 4 separately. Furthermore, if there are multiple adjacent categories, the second uncertainty may be displayed divided into multiple regions.

The section that needs correction displays areas that are highly uncertain as areas that need correction. In the example of Figure 11, the areas that need correction are displayed as white areas. The user can refer to the areas that need correction when adding teaching information. However, if the areas that need correction are selected automatically, the section that needs correction does not need to be displayed.

Figure 12 shows another example of the teaching information creation GUI 802. In this example, the GUI in Figure 11 is applied to detecting defects in structures during the semiconductor manufacturing process. As in Figure 11, three categories are displayed, and in particular, the categories "Structure 1" and "Structure 2" represent appropriate semiconductor structures, and the category "Defect" represents a structural defect.

With this type of GUI, users can efficiently create training information that can be used to detect defects in semiconductor structures.

[Example 3]
In the third embodiment, an image classification device is proposed for a segmentation model in the middle of learning in interactive segmentation in which learning of a segmentation model and annotation by a user are performed mutually. Descriptions of parts common to the first and second embodiments may be omitted.

FIG. 9 shows an example of the functional configuration of an image classification device according to the third embodiment of the present invention. The image classification device 900 first trains a segmentation model D802 using training information D805 calculated by the method shown in the second embodiment and an input image.

The model learning unit 901 learns a segmentation model D802 using an input image D801 and training information D805.

The segmentation model D802 used is one that has been trained in advance at the input stage. For the pre-training, a small amount of teacher information for the input image D801 may be used, or training may be performed using a public dataset, etc.

In the image classification device 900, the model learning unit 901 learns the segmentation model D802, and then creates teacher information D805 again through processing by the image category inference unit 801, etc., and trains the segmentation model D802 using the teacher information D805. By repeating this process until there are no more misclassifications in the inference image D803, it is possible to obtain a segmentation model D802 with good classification accuracy.

FIG. 10 shows another example of the functional configuration of an image classification device according to the third embodiment of the present invention. The image classification device 1000 calculates the uncertainty in the same manner as in the first and second embodiments, and performs enhancement learning using the uncertainty for each pixel.

Areas with high uncertainty are areas where the segmentation model has not been sufficiently trained. The model training unit 1001 trains the segmentation model based on the uncertainty information D1001 so that the loss of pixels with high uncertainty is smaller than the loss of pixels with low uncertainty.

This type of learning is expected to improve accuracy. For example, by increasing the loss per pixel in proportion to the uncertainty and training the model according to that loss, it is possible to more strongly learn the features of areas with high uncertainty.

100: Teacher information creation support device 101: Category score calculation unit 102: Inference image determination unit 103: Adjacent category calculation unit 104: Correction score calculation unit 300: Adjacent pixel group 800: Image classification device 801: Image category inference unit 802: Teacher information creation GUI
900: Image classification device 901: Model learning unit 1000: Image classification device 1001: Model learning unit D1: Input image D2: Segmentation model D3: Category score D4: Category of interest D5: Inferred image D6: Adjacent category D7: Score requiring correction D8: Score requiring correction D9: Area requiring correction D801: Input image D802: Segmentation model D803: Inferred image D804: Category score D805: Teacher information D1001: Uncertainty information S201: Processing step S202: Processing step S401: Processing step S402: Processing step S403: Processing step S501: Processing step S502: Processing step S601: Processing step S701: Processing step

Claims (13)

A teacher information creation support device,
a category score calculation unit that calculates, for an image consisting of a plurality of pixels, a category score indicating a degree of certainty that each combination of a pixel and a category belongs to the corresponding category, using a segmentation model;
An inference image determination unit that calculates an inference image based on the category score;
an adjacent category calculation unit that receives an input of a focus category and calculates an adjacent category that is likely to be adjacent to the focus category based on the inference image and the focus category;
a correction-required score calculation unit that determines a correction-required score for at least one pixel based on the focus category, the neighboring categories, and the category scores;
A teacher information creation support device comprising: The teacher information creation support device according to claim 1,
The correction score calculation unit,
extracting a category score related to the attention category and a category score related to the adjacent category for each pixel inferred to belong to the attention category in the inference image and each pixel inferred to belong to the adjacent category in the inference image;
Calculating uncertainty based on each extracted category score as a value indicating the degree to which the segmentation model is insufficiently trained;
calculating the revision score based on the uncertainty;
A teacher information creation support device comprising: The teacher information creation support device according to claim 1,
The correction score calculation unit,
Calculating a first uncertainty for each pixel inferred to belong to the attention category in the inference image based on a category score related to the attention category;
Calculating a second uncertainty for each pixel inferred to belong to the adjacent category in the inference image based on a category score related to the target category;
calculating the revision score based on the first uncertainty and the second uncertainty;
A teacher information creation support device comprising: The teacher information creation support device according to claim 1, wherein the correction score calculation unit identifies the parts with a large correction score as parts that need correction. The teacher information creation support device according to claim 1, characterized in that the correction score calculation unit smoothes the correction score for each pixel within a predetermined range including the pixel, and calculates the part with the large smoothed value as the part requiring correction. An image classification device, comprising:
an image category inference unit that uses a segmentation model to calculate a category score indicating a degree of certainty that a pixel belongs to a category for each combination of a pixel and a category for an image consisting of a plurality of pixels, and calculates an inferred image based on the category score;
A teacher information creation GUI having a focus category setting unit;
having
The attention category setting unit receives an input of an attention category,
The image classification device calculates a portion to be corrected based on the attention category and the category score,
The teacher information creation GUI includes:
Output the part that needs to be corrected,
Accepting assignment of teacher information via a teacher information assignment unit;
1. An image classification device comprising: The image classification device according to claim 6, characterized in that the image classification device has a model learning unit that uses the teacher information to learn a segmentation model. The image classification device according to claim 7,
The image classification device calculates an uncertainty indicating a degree to which the segmentation model is insufficiently trained;
The model learning unit learns the segmentation model so that a loss of pixels with large uncertainty is smaller than a loss of pixels with small uncertainty.
1. An image classification device comprising: 7. The image classification device according to claim 6,
The image classification device calculates an uncertainty representing a degree to which the segmentation model is insufficiently trained;
indicating said uncertainty;
1. An image classification device comprising: 7. The image classification device according to claim 6,
The image classification device includes:
Calculating an adjacent category that is likely to be adjacent to the attention category based on the inference image and the attention category;
Calculating a first uncertainty for each pixel inferred to belong to the attention category in the inference image based on a category score related to the attention category;
Calculating a second uncertainty for each pixel inferred to belong to the adjacent category in the inference image based on a category score related to the target category;
The teacher information creation GUI displays the first uncertainty and the second uncertainty.
1. An image classification device comprising: The image classification device according to claim 8, wherein the teaching information creation GUI displays the areas with high uncertainty as the areas requiring correction. A program that causes a computer to function as the teacher information creation support device described in claim 1. A program that causes a computer to function as the image classification device described in claim 6.

Images