JP2021111360A - Method for generating quality characteristic map - Google Patents

Abstract

To provide a method of capturing an image by an electron microscope or an optical microscope and predicting quality characteristics of a part captured in the image to generate a quality characteristic map.SOLUTION: The method for generating a quality characteristic map includes: a machine learning step of reading an image group for learning consisting of a plurality of images for learning and quality characteristics of material for learning for the images for learning in association with each other, cutting out a plurality of learning regions from the images for learning, extracting a feature quantity for learning and learning a regression model of predicting the quality characteristics of the material for learning; and a map generation step of reading an image for evaluation of one or more material for evaluation, cutting out a plurality of evaluation regions from the images for evaluation, extracting a feature quantity for evaluation, predicting quality characteristics of the material for evaluation for each evaluation region according to the regression model, and using the predicted quality characteristics of the material for evaluation to perform mapping for distribution of the quality characteristics of the material for evaluation.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique of imaging a material such as a magnet or ceramics with an image imaging device and generating a quality characteristic map from the image.

In recent years, the technical field of materials informatics, which develops new materials by effectively utilizing information science, especially data science, has attracted attention.
In materials informatics, data such as various experimental conditions and experimental results are associated and accumulated in a database, and statistical analysis, machine learning, simulation, etc. are used to extract information useful for the development of new materials.
In particular, images obtained by imaging the manufactured material with an image imaging device such as an electron microscope or an optical microscope, and quality characteristics representing the performance of the material measured by the analyzer are accumulated as useful information.

Here, as a method of analyzing an image obtained by imaging a material and obtaining useful information, for example, Patent Document 1 discloses a method of enhancing an image captured by an electron microscope. In the method of Patent Document 1, an image conversion process is prepared in advance according to the relationship between the energy signal and the angle signal obtained from the electron microscope and the composition information and the tissue structure, and the developer of the new material is paying attention. It is possible to generate an image that emphasizes the part where the desired composition information and tissue structure are reflected.

Further, as a method of applying machine learning to the analysis of an image obtained by imaging a material, for example, Patent Document 2 discloses a method of predicting quality characteristics using a neural network. In the method of Patent Document 2, a large number of pairs of image groups obtained by imaging a plurality of materials and their respective quality characteristics are prepared in advance, and the neural network is trained to newly image the trained neural network. By inputting the obtained image, the quality characteristics of the material can be predicted.

Japanese Unexamined Patent Publication No. 2018-133174 Japanese Unexamined Patent Publication No. 2019-012037

In order to generate an image by the method of Patent Document 1, it is necessary to execute an image conversion process according to the relationship between the energy signal and the angle signal obtained from the electron microscope in advance and the composition information and the tissue structure. In addition, the developer of the new material must have knowledge of the principles, composition information, tissue structure, etc. of the electron microscope.

Further, the method of Patent Document 2 has a problem that it cannot be utilized unless a large amount of materials, images thereof, and quality characteristics thereof are measured in advance for learning a neural network. In the development of new materials, it is difficult to prepare as many materials as necessary for learning neural networks.

Therefore, in the present invention, the characteristics of the material, that is, the quality characteristics, can be obtained without having to measure a large amount of materials, images, and quality characteristics, and without having knowledge of principles such as an electron microscope, composition information, and tissue structure. Provided is a method of generating a quality characteristic map capable of generating a map representing a distribution.

The method for generating a quality characteristic map according to the present invention includes a learning image group consisting of a plurality of learning images obtained by imaging one or more learning materials, and the quality characteristics of the learning material in the learning image. To learn a regression model that predicts the quality characteristics of the learning material from the learning feature amount by extracting multiple learning areas from the learning area and extracting the learning feature amount from the learning feature amount. The machine learning step to be performed and the evaluation image of one or more evaluation materials are read, a plurality of evaluation areas are cut out from the evaluation image, the evaluation feature amount is extracted from the evaluation area, and the evaluation feature amount and the evaluation feature amount are used. A map that predicts the quality characteristics of the evaluation material for each evaluation area by the regression model and maps the distribution of the quality characteristics of the evaluation material in the evaluation image using the predicted quality characteristics of the evaluation material. It has a generation step.

Further, it is preferable to extract the learning feature amount and the evaluation feature amount using a learned convolutional neural network.

Further, it is preferable to train the regression model using a random forest.

Further, it is preferable that the mapping in the map generation step is a contour map.

Further, the method for generating a quality characteristic map according to the present invention includes a learning image group composed of a plurality of learning images obtained by imaging one or more learning materials, and the learning image in the learning image. The quality characteristics of the material are associated with each other and read, and a plurality of learning areas are cut out from the learning image.
A machine learning step in which a learning feature amount is extracted from the learning area and a regression model for predicting the quality characteristics of the learning material is learned from the learning feature amount, and an evaluation image of one or more evaluation materials. Is read, a plurality of evaluation regions are cut out from the evaluation image, a first evaluation feature amount is extracted from the evaluation area, and the evaluation feature amount and the regression model are used for the evaluation. For each region, the first prediction step of predicting the quality characteristics of the evaluation material and the brightness value of a part of the pixels in the evaluation region are replaced with arbitrary values, and the second evaluation feature amount is used. A second prediction step of extracting and predicting the quality characteristics of the evaluation material for each evaluation region by the second evaluation feature amount and the regression model, the first prediction step, and the above. It may have a map generation step of mapping the difference in quality characteristics of the evaluation material predicted in the second prediction step by changing the value of the brightness of some of the pixels.

Further, it is preferable to extract the learning feature amount, the first evaluation feature amount, and the second evaluation feature amount by using a learned convolutional neural network.

Further, the difference in quality characteristics of the evaluation material predicted in the first prediction step and the second prediction step is preferably a SHAP value.

According to the present invention, a map showing the distribution of quality characteristics in an image, even if only one quality characteristic, that is, the performance of the material, can be obtained with respect to an image of the produced material. It will be possible to generate them, which will be extremely useful in materials informatics as useful information in the development of new materials.
In addition, a quality characteristic map can be drawn even for materials with little information such as images.

This is an example of an electron microscope image and a generated quality characteristic map. This is an example of a material list. This is an example of a flowchart showing a learning procedure of a machine learning step for learning a regression model. This is an example of feature data for learning. This is an example of a diagram showing the relationship between an image and a rectangular region in the present invention. This is an example of a flowchart showing the procedure of the map generation step. This is an example of predicted quality characteristic data. This is an example of a flowchart showing the procedure for generating an image of a Voronoi diagram by Monte Carlo simulation. This is an example of an image in which a Voronoi diagram is drawn and a quality characteristic map generated by using it as an input. This is an example of a flowchart showing a procedure for generating an image of a grain boundary structure by Monte Carlo simulation. This is an example of an image in which line segments and dots are drawn and a quality characteristic map generated by inputting the image. This is an example of an electron microscope image and a generated quality characteristic map. This is an example of a flowchart showing the procedure of the map generation step. This is an example of an image in which a Voronoi diagram is drawn and a quality characteristic map generated by inputting it.

Hereinafter, the method for generating the quality characteristic map of the present invention will be described in detail using five embodiments. In the first embodiment, an example in which a quality characteristic map showing the distribution of the magnetic flux density and the coercive force, which are important as the quality characteristics of the magnet, is generated from the electron microscope image of the magnet will be described. In the second and third embodiments, an example of a simulation carried out for verifying the certainty of the method of the present invention will be described. In the fourth embodiment, an example in which a quality characteristic map showing the distribution of magnetic flux density and coercive force is generated from an electron microscope image of a magnet by a method of improving the resolution will be described. Then, in the fifth embodiment, an example of the simulation carried out for verifying the certainty of the method used in the fourth embodiment will be described.

(First Embodiment)
FIG. 1 is an example of an electron microscope image to be input and a quality characteristic map generated based on the present embodiment. The input image 10 is an image obtained by capturing a part of the material of the magnet with an electron microscope, and the magnetic flux density map 11 and the coercive force map 12 are the magnetic flux density map and the coercive force generated by using the present invention, respectively. It is a map.
As described later, the quality characteristic map of the magnetic flux density map 11 and the coercive force map 12 cannot generate the edge of the image according to the size of the rectangular region used at the time of generation.

The generated magnetic flux density map 11 and coercive force map 12 indicate that the darker the shaded part, the higher the predicted quality characteristic, and the lighter the shaded part, the lower the predicted quality characteristic.
By visually comparing the image captured by the electron microscope of the input image 10 with the quality characteristic map of the magnetic flux density map 11 and the coercive force map 12, what kind of tissue structure is the part where the magnetic flux density is predicted to be high? What kind of tissue structure is the part where the magnetic flux density is predicted to be low, what kind of tissue structure is the part where the coercive force is predicted to be high, and what kind of structure is the part where the coercive force is predicted to be low? It can be compared whether it is a structure.

It is known that the magnetic flux density of a magnet increases as the area of the input image 10 becomes agglomerates and the area becomes wider, and this phenomenon appears in the map of the magnetic flux density map 11. It is also known that the coercive force becomes higher as the shaded portion of the input image 10 becomes more complicated, and this phenomenon appears in the coercive force map 12.

The quality characteristic maps of the magnetic flux density map 11 and the coercive force map 12 are shown in shades in the present embodiment, but the magnitude relationship of the quality characteristics becomes easier to understand when a pseudo color is used.

FIG. 2 is an example of a material list. The material list describes the material number, the magnetic flux density and coercive force measured as quality characteristics, and the image file name of the electron microscope. Used for attachment.
The data for each material is described vertically. In this example, one measurement result of magnetic flux density, one measurement result of coercive force, and one image are managed for one material.

The machine learning step for learning the regression model in the present embodiment includes the learning image group obtained by imaging a plurality of learning materials and the quality characteristics of the learning material in each learning image of the learning image group. , A plurality of learning rectangular areas are cut out from the read learning image, a learning feature amount is extracted using the cut out learning rectangular area, and the quality of the learning material is obtained from the learning feature amount. It is to learn a regression model that predicts the characteristics.

FIG. 3 is an example of a flowchart showing a learning procedure of a machine learning step for learning a regression model in this embodiment.

In step 101, a learned convolutional neural network (CNN) feature extractor published on the Internet is read.
This feature extractor was not learned to analyze images of magnets, ceramics, etc., which is the object of the present invention, but to identify animals, plants, etc., and to identify automobiles, pedestrians, etc. It was learned in. For example, in the R language, which is prominent in the field of statistical analysis, this feature extractor can be read by executing the function application_vgg16 incorporated in the deep learning library Keras.
In step 102, the material list shown in FIG. 2 is read.

The period from step 103 to step 111 is repeated in a loop for the number of lines in the material list, that is, for the number of images.
In step 104, the image file captured by the electron microscope is read.
From step 105 to step 110, and from step 106 to step 109, the read image is processed while moving the X and Y coordinates of the center of the learning rectangular area (hereinafter, abbreviated as rectangular area). ..
In step 107, the data of the rectangular area is cut out based on the X coordinate and the Y coordinate, and in step 108, the learning feature for the cut out rectangular area is used by using the feature amount extractor of the learned convolutional neural network read in step 101. The amount (hereinafter, abbreviated as feature amount) is extracted.

In step 112, the types of features are reduced. When the features are extracted using the feature extractor of the trained convolutional neural network, not only the effective features are extracted. When the size of the rectangular region to be cut out is 256 pixels × 256 pixels × 3 layers (RGB), 32768 kinds of feature quantities are extracted. However, depending on the type of feature quantity, there are many cases where only zero is contained as a value. Therefore, for example, the rectangular region containing zero as the feature quantity value is 90% or more of the total rectangular region. Features are deleted.

In step 113, a regression model for predicting the magnetic flux density is learned. Further, in step 114, a regression model for predicting the coercive force is learned.

Both steps 113 and 114 can be applied as regression models by any regression system machine learning method such as multiple regression analysis, random forest, and support vector machine.
These machine learning methods have the advantage that they can learn with a smaller number of materials than neural networks.
Since there is a limit to the number of types of features that can be input in multiple regression analysis, it is advisable to reduce the dimensions in advance by principal component analysis.
From this point of view, Random Forest is easy to use.

That is, an image is input, and a regression model that predicts quality characteristics, that is, magnetic flux density and coercive force, is generated in a random forest via a convolutional neural network.

FIG. 4 is an example of feature data after the completion of step 112. This feature amount data has a plurality of pairs of X and Y coordinates for one material number. The quality characteristic 302 has the same value for each item if the material number is the same. The feature amount 303 is arranged in columns as many as the number of types of the feature amount.

The trained convolutional neural network feature extractor extracts 32768 types of features from F1 to F3268, but the rectangular area containing zero as the feature value is 90% of the total rectangular area. As a result of deleting the above-mentioned feature amounts, as shown in this example, the number of feature amounts is reduced to several hundred types such as F28 and F56.

FIG. 5 is a diagram for explaining the relationship between the input image, the rectangular area, and the coordinates.
With respect to the image read in step 104, the X coordinate is taken in the horizontal direction and the Y coordinate is taken in the vertical direction, and the coordinates are determined according to the number of pixels with the upper left portion as the origin. In the case of an image having 2560 pixels horizontally and 1920 pixels vertically as shown in the figure, the X coordinate is from 1 to 2560 and the Y coordinate is from 1 to 1920.
The rectangular area 13 is a small piece of an image obtained by cutting out a part of the image.

For example, the illustrated rectangular region 13 has central coordinates of X = 2304 and Y = 256, and is an region of 256 pixels horizontally and 256 pixels vertically.
The size of the rectangular region 13 is not limited to 256 pixels in the horizontal direction and 256 pixels in the vertical direction, and may be freely determined according to the size of the image, the size of the tissue reflected in the image, and the like. The smaller the size of the rectangular region 13, the higher the resolution of the generated quality characteristic map.
However, if the size of the rectangular region 13 is small, the prediction accuracy of the quality characteristic is lowered, and the credibility of the generated quality characteristic map is lowered.
Therefore, it is advisable to confirm the prediction accuracy of the quality characteristics and determine an appropriate size of the rectangular region 13.

In the map generation step in the present embodiment, at least one evaluation image of the evaluation material is read, a plurality of evaluation rectangular areas are cut out from the read evaluation image, and the evaluation feature amount is cut out from the cut out evaluation rectangular area. Is extracted, the quality characteristics of the evaluation material are predicted for each evaluation rectangular area by the extracted evaluation feature quantity and the regression model, and the predicted quality characteristics of the evaluation material are used for evaluation in the evaluation image. It is to map the distribution of the quality properties of the material.

FIG. 6 is an example of a flowchart showing the procedure of the map generation step in the present embodiment.

In step 201, the same trained convolutional neural network feature extractor as in step 101 is read.
In step 202, a regression model for predicting the magnetic flux density learned in step 113 is read.
In step 203, a regression model for predicting the coercive force learned in step 114 is read.
In step 204, an image of the material for which a quality characteristic map is to be generated is loaded.
The image read here may be one of the images of the learning material used for learning the regression model, or may be an image of another material not used for learning.
However, the image imaging conditions such as magnification and size must be the same as the image used for learning the regression model.

From step 205 to step 212, and from step 206 to step 211, the read image is processed while moving the X and Y coordinates of the center of the evaluation rectangular area (hereinafter, abbreviated as rectangular area). ..
In step 207, a rectangular area with respect to the X coordinate and the Y coordinate is cut out.
In step 208, the evaluation feature amount (hereinafter abbreviated as feature amount) for the rectangular region is extracted using the feature amount extractor of the trained convolutional neural network, and is the same as the type of feature amount reduced in step 112. To delete.
In step 209, the magnetic flux density in the rectangular region is predicted.
In step 210, the coercive force of the rectangular region is predicted.

FIG. 7 is an example of quality characteristic data predicted in stages 209 and 210.
The predicted magnetic flux density and the predicted coercive force are listed for each combination of the X coordinate and the Y coordinate.

At the end of the loop, step 213 creates a magnetic flux density map using the X and Y coordinates and the predicted magnetic flux density.
In step 214, a coercive force map is generated using the X coordinate, the Y coordinate, and the predicted coercive force. In this example, the X and Y coordinates were moved in the same way as when training the regression model, but in the training, the X and Y coordinates were moved roughly, and when the quality characteristic map was generated, they were moved more finely than in the case of training. You may.
The finer the movement, the higher the resolution of the quality characteristic map can be generated.

Here, the mapping in the present invention is an image of the distribution of quality characteristics from the quality characteristic data predicted as described above. Further, the mapping is preferably imaged with a contour map, and the result is the magnetic flux density map of 11 and the coercive force map of 12 shown in FIG.

The contour map can be easily drawn by using the graph drawing library matplotlib or seaborn in the Python language, the basic function contour in the R language, or ggplot2 in the graph drawing library.
However, when they cannot be used, for example, a smooth B-spline curved surface may be approximated to the quality characteristic data and imaged.
More simply, a scatter plot may be drawn with the X and Y coordinates of the quality characteristic data, and each hit point may be colored with the predicted magnetic flux density and the predicted coercive force.
Further, the mapping may be imaged not only in the two-dimensional display but also in the three-dimensional display.

(Second Embodiment)
Next, in order to verify the certainty of the quality characteristic map generated by the method of the present embodiment, an image depicting a boronoy diagram similar to the structure of the magnet is generated by Monte Carlo simulation, and the image is input. A quality characteristic map was generated.

FIG. 8 is a flowchart showing a procedure for generating an image of a Voronoi diagram by Monte Carlo simulation. Here, an example of generating a square image of 256 pixels × 256 pixels will be described.

From step 401 to step 408, the number of tissues is repeated as a variable.
For example, when the variable N is 40, 40 tissues are generated in one image.
That is, the average area of the tissue is 1/40 of the image size.
Between steps 402 and 407, it is repeated 20 times to generate 20 images with the same number of tissues.
In step 403, N random X, Y coordinates are generated based on the variable N. An integer from 1 to 256 is randomly generated as the X coordinate. In addition, the Y coordinate also randomly generates an integer from 1 to 256.
In step 404, the vertices of the Voronoi diagram are calculated from N random coordinates.
In step 405, the calculated vertex groups are connected by line segments and a Voronoi diagram is drawn.
Save the image in which the Voronoi diagram was drawn in step 406. By executing the process of this flowchart, 100 images of different organizational structures can be generated.

FIG. 9 shows an image in which a Voronoi diagram is drawn and an example of a quality characteristic map generated by using the image as an input.
The Voronoi diagram image 421 is an example of a 256-pixel × 256-pixel image on which the Voronoi diagram saved in step 406 is drawn. The quality characteristic map 422 is a quality characteristic map generated by inputting the image of the Voronoi diagram image 421.
The regression model generated the objective variable as the mean area of the tissue.

As a result, even if a regression model is generated with the average area of the tissue for each image, as in the quality characteristic map 422, the part with a large tissue area has a dark shade, and the part with a small tissue area has a shade. It is drawn lightly.
For example, if there is a structure having a large area such as the upper right corner of the Voronoi diagram image 421, the upper right corner of the quality characteristic map 422 is drawn in dark shades to correspond to it.
On the other hand, the lower right corner of the quality characteristic map 422 is lightly shaded so as to correspond to the portion where the tissues having a small area are densely formed, such as the lower right corner of the Voronoi diagram image 421.
Therefore, it can be seen that the quality characteristic map 422 generated by using the present invention well represents the characteristics regarding the area of the Voronoi diagram image 421.

(Third Embodiment)
Next, in order to verify the certainty of the quality characteristic map generated by the method of the present embodiment with another object, an image drawing a diagram similar to the grain boundary structure of ceramics is generated by Monte Carlo simulation, and the image is displayed. Input to generate a quality characteristic map.

As the grain boundary structure of ceramics, an image in which small dots are randomly scattered is captured.
However, depending on the sintering conditions, particles that grow abnormally in a straight line are generated, which reduces the bending strength. Therefore, we generated an image in which random dots and random line segments were mixed, and generated and verified a regression model that predicts the total length of the line segments.

FIG. 10 is a flowchart showing a procedure for generating an image of a grain boundary structure by a Monte Carlo simulation. Here, an example of generating a square image of 256 pixels × 256 pixels will be described.

Between steps 501 and 507, the process is repeated for the number of images to be generated.
In step 502, 50 pairs of random XY coordinate pairs are generated. That is, the coordinates (X1, Y1) and the coordinates (X2, Y2) are generated, and the line segment connecting them is defined as an abnormally grown granular material. However, there is no overlong overgrowth.
Therefore, the length of each line segment is calculated in step 503, and in step 504, 50 line segments are arranged, only 10 are selected from the shorter one, and 10 line segments are drawn.
In step 505, 200 random dots are drawn.
In step 506, an image in which the line segment and the dot are drawn is saved.

FIG. 11 shows an image in which line segments and dots are drawn and an example of a quality characteristic map generated by using the image as an input.
The pseudo-ceramic grain boundary structure image 521 is an example of a 256-pixel × 256-pixel image in which the line segments and dots saved in step 506 are drawn.
The quality characteristic map 522 is a quality characteristic map generated by inputting an image of the pseudo-ceramic grain boundary structure image 521.
The regression model generated the objective variable as the total length of multiple line segments.

As a result, although the regression model is generated as the total length of a plurality of line segments, the quality characteristic map 522 shows a part where a long line segment exists or a part where a plurality of line segments are densely packed. It can be confirmed that the shades are drawn deeply.
Therefore, it can be seen that the quality characteristic map 522 generated using the present embodiment well represents the characteristics related to the pseudoceramic grain boundary structure image 521.

(Fourth Embodiment)
FIG. 12 is an example of an electron microscope image to be input and a quality characteristic map generated based on the present embodiment. The input image 15 is an image obtained by capturing a part of the material of the magnet with an electron microscope, and the magnetic flux density map 16 and the coercive force map 17 are the magnetic flux density map and the coercive force map generated by using the present invention, respectively. It is a map.
The quality characteristic maps of the magnetic flux density map 16 and the coercive force map 17 have higher resolution than the magnetic flux density map 11 and the coercive force map 12, and can also generate the edges of the image. different.

The generated magnetic flux density map 16 and coercive force map 17 show that the lighter the shade, the higher the quality characteristic, and the darker the shade, the lower the quality characteristic.
What kind of tissue structure is the part that has the effect of increasing the magnetic flux density by visually comparing the image captured by the electron microscope of the input image 15 with the quality characteristic map of the magnetic flux density map 16 and the coercive force map 17. What kind of tissue structure is the part that has the effect of lowering the magnetic flux density, what kind of tissue structure is the part that has the effect of increasing the coercive force, and what kind of tissue structure is the part that has the effect of lowering the coercive force? You can compare what kind of organizational structure it is.

It is known that the magnetic flux density of a magnet increases as the area of the input image 15 is dark, that is, the particle portion is agglomerates and the area is large. The magnetic flux density map 16 shows that there is an action of increasing the magnetic flux density at the end of the particle portion having a large area. In particular, the left and right ends of the particle portion have the effect of increasing the magnetic flux density, and the shading is lightened. On the other hand, it can be seen that even at the end of the particle portion, the upper end and the lower end are shaded. This indicates that the orientation of the easy-to-magnetize axis of the electron microscope image 15 is left and right. It is also known that the coercive force becomes higher in the light shaded portion of the input image 15, that is, the grain boundary portion. The phenomenon appears in the coercive force map 17. From these quality characteristic maps, it can be seen that both the magnetic flux density and the coercive force can be increased if the vertically long particle portion can be formed.

The quality characteristic maps of the magnetic flux density map 16 and the coercive force map 17 are shown in shades in the present embodiment, but the magnitude relationship of the quality characteristics becomes easier to understand when a pseudo color is used. For example, it is easy to understand if the part that has the effect of improving the quality characteristics is colored red and the part that has the work of lowering the quality characteristics is colored blue.

In the procedure for generating the magnetic flux density map 16 and the coercive force map 17, the machine learning step is the same as the flowchart of FIG. 3 illustrated for generating the magnetic flux density map 11 and the coercive force map 12. On the other hand, the map generation steps are slightly different.

FIG. 13 is an example of a flowchart showing the procedure of the map generation step in the present embodiment.
For the steps 113 and 114 shown below, refer to the flowchart of FIG.
In step 201, the learned convolutional neural network feature extractor is read.
In step 202, a regression model for predicting the magnetic flux density learned in step 113 is read.
In step 203, a regression model for predicting the coercive force learned in step 114 is read.
In step 204, an image of the material for which a quality characteristic map is to be generated is loaded.
The image read here may be one of the images of the learning material used for learning the regression model, or may be an image of another material not used for learning.
However, the image imaging conditions such as magnification and size must be the same as the image used for learning the regression model.

From step 205 to step 212, and from step 206 to step 211, the read image is processed while moving the X and Y coordinates of the center of the evaluation rectangular area (hereinafter, abbreviated as rectangular area). ..
In step 207, a rectangular area with respect to the X coordinate and the Y coordinate is cut out.
In step 208, the first evaluation feature amount (hereinafter, abbreviated as feature amount 1) for the rectangular region is extracted using the feature amount extractor of the trained convolutional neural network, and the feature amount reduced in step 112 is extracted. Delete the same type.

Further, the luminance value of a part of the pixels in the rectangular region is replaced with an arbitrary value, and the second evaluation feature quantity (hereinafter, abbreviated as feature quantity 2) for the rectangular region is the feature of the trained convolutional neural network. Extract using a quantity extractor, and delete the same type of feature quantity reduced in step 112.

Next, the quality characteristics of the evaluation material are predicted for each rectangular region by the feature quantity 1 and the regression model learned in the steps 113 and 114 (first prediction step), and similarly, the feature quantity 2 is also obtained. The quality characteristics of the evaluation material are predicted for each rectangular region by the regression model (second prediction step). Further, the difference in quality characteristics of the evaluation material predicted in the first prediction step and the second prediction step is mapped by changing the luminance value of some pixels (map generation step).

Here, the difference in the quality characteristics of the evaluation materials predicted in the first prediction step and the second prediction step corresponds to, for example, the SHAP value that can be calculated using the regression model learned in steps 113 and 114. The value.

The SHAP value is an example of a numerical value calculated by XAI (Explainable AI), and is an evaluation material predicted in prediction step 1 and prediction step 2 using XAI such as LIMITE (Local Interpretable Model-agnostic Explanations). The difference in quality characteristics of may be calculated.

Therefore, in step 231 of the present embodiment, a matrix of SHAP values of the magnetic flux density in the rectangular region is calculated using the regression model for predicting the magnetic flux density read in step 202. If the SHAP value is a rectangular area of 256 × 256 pixels, 256 × 256 SHAP values are similarly calculated. That is, since the SHAP value is calculated for each pixel, a quality characteristic map with high resolution can be generated.
In step 232, a matrix of SHAP values of the coercive force in the rectangular region is calculated using the regression model for predicting the coercive force read in step 203.

When the loop is completed, in step 233, a magnetic flux density map is generated using a matrix of X-coordinates, Y-coordinates, and the calculated SHAP value.
In step 234, a coercive force map is generated using a matrix of X-coordinates, Y-coordinates, and calculated SHAP values.

(Fifth Embodiment)
Next, in order to verify the certainty of the quality characteristic map generated by the method of the present embodiment, an image depicting a Boronoi diagram similar to the structure of the magnet is generated by Monte Carlo simulation, and the image is input. A quality characteristic map was generated. The procedure for generating an image of a Voronoi diagram by Monte Carlo simulation is the same as that in FIG.

FIG. 14 shows an image in which the Voronoi diagram is drawn and an example of a quality characteristic map generated by using the image as an input and using the fourth embodiment.
The Voronoi diagram image 431 is an example of a 256-pixel × 256-pixel image on which the Voronoi diagram saved in step 406 is drawn. The quality characteristic map 432 is a quality characteristic map generated by inputting the image of the Voronoi diagram image 431. The regression model generated the objective variable as the mean area of the tissue.

As a result, even if a regression model is generated with the average area of the tissue for each image, as in the quality characteristic map 432, the part with a large tissue area has a dark shade, and the part with a small tissue area has a shade. It is drawn lightly. In particular, the grain boundary of the portion where the area of the tissue was small was almost white in shade.
Therefore, it can be seen that the quality characteristic map 432 in the present embodiment well represents the characteristics related to the area of the Voronoi diagram image 431.

The present invention is not limited to an electron microscope, and can be applied to any image imaging device.
Further, the present invention does not require knowledge of the principle, composition information, tissue structure, etc. of the image capturing apparatus.
Further, it is possible to generate a map showing the distribution of quality characteristics in an image even when only one quality characteristic is obtained for the image obtained by capturing the material produced in the experiment.

As a result, it is possible to compare the good part and the bad part from the generated map. For example, overgrowth of grain boundary structures has been found to have the greatest effect on quality characteristics such as ceramic strength.
However, there are other factors that affect the strength other than the abnormal growth of the grain boundary structure.

The present invention can generate a map of quality characteristic distribution, that is, intensity distribution, from an image obtained by imaging a material even for such a material, and can compare a strong portion and a weak portion. .. As a result, it is possible to give awareness to a phenomenon that the developer of the new material does not anticipate.

Further, the present invention can be utilized without measuring a large number of materials, their images, and their quality characteristics as in the method of Patent Document 2.

Although the present invention has been described above using the above-described embodiment, the present invention is not limited to the above-described embodiment. The contents can be changed within the technical scope included in the claims.
For example, in the above embodiment, the quality characteristics of the magnetic flux density and the coercive force of the magnet are mapped, but it can also be used to map the quality characteristics of the thermal conductivity and the bending strength of ceramics.

10 Input image 11 Magnetic flux density map 12 Coercive force map 13 Rectangular area 421 Boronoi map image 422 Quality characteristic map 521 Pseudoceramics grain boundary structure image 522 Quality characteristic map 15 Input image 16 Magnetic flux density map 17 Coercive force map 431 Boronoi map image 432 Quality Characteristic map

Claims (7)

A learning image group consisting of a plurality of learning images obtained by imaging one or more learning materials and a quality characteristic of the learning material in the learning image are read in association with each other.
A plurality of learning areas are cut out from the learning image, and a plurality of learning areas are cut out.
A machine learning step in which a learning feature amount is extracted from the learning area and a regression model for predicting the quality characteristic of the learning material is learned from the learning feature amount.
Load the evaluation image of one or more evaluation materials,
A plurality of evaluation areas are cut out from the evaluation image, and a plurality of evaluation areas are cut out.
The evaluation feature amount is extracted from the evaluation area, and the evaluation feature amount is extracted.
The quality characteristics of the evaluation material are predicted for each evaluation area by the evaluation feature amount and the regression model.
A map generation step of mapping the distribution of the quality characteristics of the evaluation material in the evaluation image using the predicted quality characteristics of the evaluation material.
A method of generating a quality characteristic map, which comprises having. The method for generating a quality characteristic map according to claim 1, wherein the learning feature amount and the evaluation feature amount are extracted using a trained convolutional neural network. The method for generating a quality characteristic map according to claim 2, wherein the learning of the regression model is performed using a random forest. The method for generating a quality characteristic map according to any one of claims 1 to 3, wherein the mapping in the map generation step is a contour map. A learning image group consisting of a plurality of learning images obtained by imaging one or more learning materials and a quality characteristic of the learning material in the learning image are read in association with each other.
A plurality of learning areas are cut out from the learning image, and a plurality of learning areas are cut out.
A machine learning step in which a learning feature amount is extracted from the learning area and a regression model for predicting the quality characteristic of the learning material is learned from the learning feature amount.
Load the evaluation image of one or more evaluation materials,
A plurality of evaluation areas are cut out from the evaluation image, and a plurality of evaluation areas are cut out.
The first evaluation feature amount is extracted from the evaluation area, and the evaluation feature amount is extracted.
The first prediction step of predicting the quality characteristics of the evaluation material for each evaluation region by the first evaluation feature amount and the regression model.
The brightness value of a part of the pixels in the evaluation region is replaced with an arbitrary value, the second evaluation feature amount is extracted, and the evaluation feature amount is used with the second evaluation feature amount and the regression model. A second prediction step for predicting the quality characteristics of the evaluation material for each region,
A map generation step that maps the difference in quality characteristics of the evaluation material predicted in the first prediction step and the second prediction step by changing the brightness value of some of the pixels.
A method of generating a quality characteristic map, which comprises having. The quality characteristic according to claim 5, wherein the learning feature amount, the first evaluation feature amount, and the second evaluation feature amount are extracted by using a learned convolutional neural network. How to generate a map. The quality characteristic map according to claim 5 or 6, wherein the difference in quality characteristics of the evaluation material predicted in the first prediction step and the second prediction step is a SHAP value. How to generate.

Images