JP2019012037A - Material characteristic estimation device and material characteristic estimation method - Google Patents

Abstract

A material property estimation apparatus and a material property estimation method capable of efficiently estimating a property value using an image of a material observed with a scanning electron microscope or the like. An information reading unit 211 that reads an image of a material imaged by a microscope 600, an estimation unit 212 that estimates a material property of the material based on a neural network model based on the image, and an output that outputs an estimation result by the estimation unit 212 The estimation result includes the certainty information of the estimation. [Selection] Figure 1

Description

  The present invention relates to a material property estimation device and a material property estimation method.

  In recent years, the characteristics of steel products required for thin steel plates used for automobiles, etc., and for thick steel plates used for line pipes, etc. have been improved, and variations in allowable material property values (mechanical property values) have also occurred. It has become stricter. Further, the development of steel products for improving characteristic values has been demanded to be more efficient. For efficient steel product development, steel materials are actually estimated and predicted using a model before actually producing steel materials and measuring properties such as Tensile Strength (TS) using a testing device. Technology is important. For example, Patent Document 1 proposes a method for estimating a characteristic value of a steel product from an operation condition such as a steel material temperature in the steel process.

JP 2002-236119 A

  Although there is a method for estimating the characteristic value from the operation condition as described above, it has not been proposed to estimate the characteristic value efficiently by using an image of a material observed with a scanning electron microscope or the like.

  Accordingly, an object of the present invention made in view of the above-described problems and the like is a material property estimation apparatus and material capable of efficiently estimating a property value using an image of a material observed with a scanning electron microscope or the like. It is to provide a characteristic estimation method.

In order to solve the above problems, a material property estimation apparatus according to an embodiment of the present invention is
An input unit for receiving input of an image of a material imaged by a microscope;
An estimation unit for estimating material properties of the material by a neural network model based on the image;
An output unit that outputs an estimation result and a certainty factor by the estimation unit;
Is provided.

Moreover, the material property estimation method according to an embodiment of the present invention includes:
Reading an image of the material imaged by a microscope;
Estimating material properties of the material by a neural network model based on the image;
Outputting an estimation result of the estimating step;
Including
The estimation result includes certainty information of estimation.

  According to the material property estimation apparatus and the material property estimation method according to an embodiment of the present invention, it is possible to efficiently estimate a property value using an image of a material observed with a scanning electron microscope or the like.

1 is a schematic diagram of a system including a material property estimation apparatus according to an embodiment of the present invention. It is an example of a microstructure image. It is a flowchart which shows a prediction model creation process. It is a conceptual diagram of the process of a neural network model. The processing flow figure of a neural network model is shown. It is a flowchart which shows the estimation process of a material characteristic. It is an example of the micro structure image used as the object of an estimation process. It is an example of the result display of an estimation process. It is another example of the result display of an estimation process. It is an example of another format of the table | surface which shows the estimation precision of a neural network model. It is an example of another format of the table | surface which shows the estimation precision of a neural network model.

(Embodiment)
Embodiments of the present invention will be described below.

(Configuration of this system)
FIG. 1 is a schematic diagram of a system 100 including a material property estimation apparatus 200 according to an embodiment of the present invention. As shown in FIG. 1, a system 100 according to an embodiment of the present invention includes a material property estimation device 200, a display device 300, an input device 400, a storage device 500, a microscope 600, and a first information processing device. 601, a material property measuring apparatus 700, and a second information processing apparatus 701.

  The material property estimation device 200 is a device that performs a material property estimation process. The material property estimation device 200 is generally connected to the display device 300, the input device 400, the storage device 500, the first information processing device 601, and the second information processing device 701 by wire. The material property estimation device 200 performs transmission / reception of necessary information by communicating with each of these devices, and estimates material properties. In addition, although the material property estimation apparatus 200 and each apparatus are connected by the wire via the bus | bath in FIG. 1, the aspect of a connection is not restricted to this, You may connect by radio | wireless or combined wired and radio | wireless A connection mode may be used. Each detailed structure of the material property estimation apparatus 200 will be described later.

  The display device 300 is an arbitrary display such as a liquid crystal display and an organic EL display. The display device 300 can display a screen based on input data and a video signal. The display device 300 displays the processing result by the material property estimation device 200 and the like.

  The input device 400 is an arbitrary input interface that can detect the operation of the administrator of the system, such as a keyboard, a pen tablet, a touch pad, and a mouse. Note that the input device 400 may be a touch panel provided integrally with the display device 300. The input device 400 receives operations related to various processing instructions to the material property estimation device 200.

  The storage device 500 is, for example, a hard disk drive, a semiconductor drive, an optical disk drive, or the like, and is a device that stores information necessary for this system.

  The microscope 600 is, for example, a scanning electron microscope or a scanning optical microscope, and is an apparatus that images a microstructure of a material (hereinafter also referred to as a test piece) whose material characteristics are estimated by the present system. The test piece is, for example, a steel material. In this embodiment, an example in which the test piece is a steel material will be described. FIG. 2 shows an example of a microstructure image of a test piece imaged by the microscope 600 (hereinafter referred to as a microstructure image and corresponding to the “image” of the present invention). For example, as shown in FIG. 2, the microstructure image is a two-dimensional image obtained by enlarging the crystal structure of the surface of the test piece to a visual field of 150 μm.

  The first information processing apparatus 601 is an apparatus for storing a microstructure image captured by the microscope 600 as electronic data and transmitting the electronic data to the material property estimation apparatus 200. Here, the first information processing apparatus 601 stores the microstructure image and the test piece in association with each other. For example, the first information processing apparatus 601 stores an ID unique to each test piece and a microstructure image of the test piece in association with each other.

  The material property measuring apparatus 700 includes, for example, a tensile test device that performs a tensile test of a test piece, and is a device that can measure the material property of the test piece. In the case where the material property measuring apparatus 700 is a tensile test device, the material property is tensile strength. In the present embodiment, an example in which the material property is tensile strength will be described below. The material properties are not limited to tensile strength, and may be ductility, toughness, compressive strength, shear strength, or any other material property. In this embodiment, an example in which 112 samples having a tensile strength of 560 to 700 MPa and a ductility (EL) of 29 to 39% with respect to a 60 k hot-rolled high tensile material, 5 charges, 19 coils, and a characteristic value range is used as a test piece. Show.

  The second information processing apparatus 701 is an apparatus for storing the material characteristic value measured by the material characteristic measuring apparatus 700 as electronic data and transmitting the electronic data to the material characteristic estimating apparatus 200. Here, the second information processing apparatus 701 stores the material characteristic value and the test piece in association with each other. Specifically, for example, the second information processing apparatus 701 stores the ID unique to the test piece and the material characteristic value of the test piece in association with each other.

(Configuration of material property estimation device)
As shown in FIG. 1, the material property estimation apparatus 200 includes an arithmetic processing unit 201, a ROM 202, and a RAM 204. The ROM stores a material property estimation program 203. The arithmetic processing unit 201, the ROM 202, and the RAM 204 are connected to each other by a bus 205. The display device 300, the input device 400, the storage device 500, the first information processing device 601, and the second information processing device 701 are also connected by the bus 205.

  The arithmetic processing unit 201 includes one or more processors such as a general-purpose processor and a dedicated processor specialized for specific processing. The arithmetic processing unit 201 reads the material property estimation program 203 from the ROM 202 and realizes a specific function using the RAM 204 which is a temporary storage unit. The arithmetic processing unit 201 controls the overall operation of the material property estimation apparatus 200.

  As shown in FIG. 1, the arithmetic processing unit 201 includes, as functional blocks, an information reading unit 206, an image preprocessing unit 207, a discretization processing unit 208, a prediction model creation unit 209, an output unit 210, and an information reading unit. 211, an estimation unit 212, and an output unit 213. When receiving a prediction model creation processing instruction based on an operation of the input device 400, the arithmetic processing unit 201 receives an information reading unit 206, an image preprocessing unit 207, a discretization processing unit 208, and a prediction model creation unit 209. And the output unit 210 functions. On the other hand, the arithmetic processing unit 201 causes the information reading unit 211, the estimation unit 212, and the output unit 213 to function when receiving an instruction for estimation processing based on an operation of the input device 400.

(Prediction model creation process)
Next, the material property estimation process by the system 100 will be described. Before estimating the material properties, the system first creates a prediction model that links the microstructure image and the material properties. FIG. 3 shows a flowchart relating to the prediction model creation process. The arithmetic processing unit 201 of the material property estimation apparatus 200 executes the process according to FIG. 3 when receiving an instruction for a prediction model creation process based on an operation of the input apparatus 400.

  When receiving the prediction model creation instruction, the information reading unit 206 of the arithmetic processing unit 201 first reads the microstructure image of the test piece stored by the first information processing apparatus 601. The information reading unit 206 reads a material characteristic value corresponding to the read microstructure image. Specifically, the information reading unit 206 specifies information corresponding to the read microstructure image based on the ID of the test piece. And the material characteristic value of the said test piece preserve | saved by the 2nd information processing apparatus 701 is read (step S201).

  Next, the image preprocessing unit 207 processes the microstructure image input in step S201 for the prediction model creation process (step S202). Specifically, the image preprocessing unit 207 compresses the microstructure image. For example, when the microstructure image input in step S201 is an image having a color depth of 1 channel and 8-bit color (256 colors) and an image size of 1252 horizontal by 990 pixels vertical, the image preprocessing unit 207 has a width of 120. X Compress to 94 pixels in height. The image pre-processing unit 207 changes the color depth from 1 channel to 3 channel true color. The image pre-processing unit 207 also deletes the lower 18 pixels that are printed from the microstructure image. By such processing, binary data of 120 [pixels] × 76 [pixels] × 3 [channels] × 112 [samples] = 3,064,320 [bytes] is created for each microstructure image.

  Subsequently, the image preprocessing unit 207 selects one image subjected to the above processing at random, and performs a change process. Specifically, the image pre-processing unit 207 vertically inverts the selected image with a probability of 50%, horizontally inverts with a probability of 50%, changes the brightness randomly in 64 steps, and changes the contrast randomly. Then, a small image of 24 × 24 pixels is randomly extracted (crop) from the changed image. By repeating the above processing 50,000 times, 50,000 small images of 24 × 24 pixels are created. These 50,000 images are called learning data. Furthermore, the image pre-processing unit 207 creates 1,0008 images independently from the learning data by performing the same processing as when the learning data is created. This is called evaluation data.

  Subsequently, the discretization processing unit 208 discretely converts the material property value of the test piece read in step S201 into category data (step S203). When the material property values (here, tensile strength) of all the test pieces are values within the range of 560 MPa to 699 MPa, the discretization processing unit 208 sets the material property values of 11 categories from category 0 to category 10. Data is discretely transformed by assigning to either. Specifically, category 0 is assigned to data having a tensile strength of 560 MPa or more and less than 570 MPa. Further, category 1 is assigned to data having a tensile strength of 570 MPa or more and less than 580 MPa. Further, category 2 is assigned to data having a tensile strength of 580 MPa or more and less than 590 MPa. Further, category 3 is assigned to data having a tensile strength of 590 MPa or more and less than 600 MPa. Further, category 4 is assigned to data having a tensile strength of 600 MPa or more and less than 610 MPa. Further, category 5 is assigned to data having a tensile strength of 610 MPa or more and less than 620 MPa. Further, category 6 is assigned to data having a tensile strength of 620 MPa or more and less than 630 MPa. Further, category 7 is assigned to data having a tensile strength of 630 MPa or more and less than 640 MPa. Further, category 8 is assigned to data having a tensile strength of 660 MPa or more and less than 670 MPa. Further, category 9 is assigned to data having a tensile strength of 670 MPa or more and less than 680 MPa. Further, category 10 is assigned to data having a tensile strength of 680 MPa or more and less than 690 MPa. Thus, one category is assigned in the range where the tensile strength is 10 MPa.

  Subsequently, the prediction model creation unit 209 creates a neural network model that links the microstructure image and material characteristics (step S204). Specifically, the prediction model creation unit 209 inputs the image (learning data) created in step S202, and creates a neural network model that outputs the category values of the material characteristics discretized in step S203. More specifically, the prediction model creation unit 209 performs optimization calculation of the parameters of the neural network model. As an optimization calculation, the prediction model creation unit 209 first inputs learning data (50,000 pieces of image data) to a neural network model, and estimates of categories output from the neural network model and test pieces corresponding to the images. The sum is calculated by adding the squares of the differences from the category value. For example, the sum of squares of the difference between the output (0 to 1) values of each category is calculated with respect to correct values where the value of a specific category is 1 and the others are 0. In other words, the estimated value of the category output by the neural network model (output value of 0 to 1 for each category) and the correct value of the category value of the test piece (the value of the specific category to which the test piece belongs is 1, the specific category For the category values other than 0), the sum of squares of the difference between the values of each category is calculated. The prediction model creation unit 209 optimizes the parameters so that the sum is minimized.

  FIG. 4 shows a conceptual diagram of processing of the neural network model in this system. The neural network model has a structure in which inputs and outputs of a plurality of filters are connected in multiple stages. As shown in FIG. 4, the neural network model first receives an input image 41, determines the value of each pixel 44 through a filter 43, and outputs a first feature map 45. Next, the value of each pixel 47 is determined for the first feature map 45 through the filter 46, and the second feature map 48 is output. Such processing is repeated, and the classifier 50 finally outputs the category related to the material characteristic value through the bonding layer 49.

  FIG. 5 shows a processing flow diagram of the neural network model in this system. The neural network model according to the present embodiment includes a first convolution layer 301, a first pooling layer 302, a first local response normalization filter 303, a second convolution layer 304, and a second convolution layer in order from the input side. It includes a local response normalization filter 305, a second pooling layer 306, a first fully coupled layer 307, a second fully coupled layer 308, and a classifier 309.

  The first convolution layer 301 receives an image of horizontal 24 × vertical 24 pixels and outputs a first feature map of 19 × 19 pixels by a convolution operation. The first feature map indicates what local features are present at which locations in the input image. In the convolution calculation, for example, 60 kinds of filters of 5 × 5 pixels and 3 channels are used.

  The first pooling layer 302 receives the first feature map output from the first convolution layer 301 as an input, and sets the maximum value within 3 × 3 pixels of the first feature map as a new pixel. Such an operation is performed over the entire map while shifting every pixel. As a result, the first pooling layer 302 outputs a second feature map obtained by smoothing the first feature map.

  The first local response normalization filter 303 performs normalization processing between different second feature maps at the same position of the second feature map so that the value of the second feature map output from the first pooling layer 302 is not saturated.

  The second convolution layer 304 uses the second feature map subjected to the normalization process as an input image, and outputs a third feature map through a convolution operation. The third feature map indicates what local features are present at which locations in the input image. In the convolution calculation, for example, 60 kinds of filters of 5 × 5 pixels and 3 channels are used.

  The second local response normalization filter 305 performs normalization processing between different third feature maps at the same position of the third feature map so that the value of the third feature map output from the second convolution layer 304 is not saturated.

  The second pooling layer 306 receives the third feature map normalized by the second local response normalization filter 305 as an input, and sets the maximum value within 3 × 3 pixels of the third feature map as a new pixel. . Such an operation is performed over the entire map while shifting every pixel. Accordingly, the second pooling layer 306 outputs a fourth feature map obtained by smoothing the third feature map.

  The first fully connected layer 307 and the second fully connected layer 308 combine the pixels of the fourth feature map output from the second pooling layer 306 with a predetermined algorithm. The combined fifth feature map is output to the classifier 309. Specifically, the first fully connected layer 307 receives all the pixels of the fourth feature map output from the second pooling layer 306 as input, and is a neural network in which the input pixels and another 384 neurons are fully connected. 384 values calculated from this neural network are output to the second fully connected layer 308. The second fully connected layer 308 is a neural network in which 384 inputs inputted from the first fully connected layer 307 and another 192 neurons are fully connected, and 192 calculated from this neural network. The value is output to the classifier 309.

  The classifier 309 outputs a category (11 classifications in the present embodiment) related to the material characteristic value from the fifth feature map. Here, the classifier 309 outputs a value for each category, and each output value indicates the probability of belonging to each category.

  When the parameters of the neural network model are optimized based on the result output by the above processing and the category of the test piece based on the measurement value, the weighting of the filter is optimized. By performing the optimization process, local features of the microstructure such as grain boundaries, phase fractions, and grain shapes that affect the tensile strength can be extracted by the neural network model.

  After optimizing the parameters of the neural network model, the prediction model creation unit 209 inputs the evaluation data created in step S203 to the neural network model and obtains an estimation result for the evaluation data. The prediction model creation unit 209 stores the learning data, evaluation data, neural network model parameters, the output result of the neural network model for the learning data, and the output result of the neural network model for the evaluation data in the storage device 500. Let

  Subsequently, the output unit 210 outputs the estimation result of the evaluation data based on the neural network model created in step S204 to the display device 300 (step S205). The display device 300 outputs the estimation result in a table format, for example.

(Material property estimation process)
The arithmetic processing unit 201 of the material property estimation apparatus 200 performs material property estimation processing when receiving an instruction for estimation processing based on an operation of the input device 400. FIG. 6 is a flowchart showing a material property estimation process.

  First, the information reading unit 211 reads a microstructure image to be estimated (step S401). FIG. 7 shows an example of a microstructure image that is an object of estimation processing. The microstructure image to be estimated is acquired by the microscope 600 and stored in the first information processing apparatus 601. The information reading unit 211 acquires the microstructure image from the first information processing device 601. Further, the information reading unit 211 acquires various data related to the neural network model stored in the storage device 500.

  Next, the estimation unit 212 estimates the material characteristics of the material related to the image based on the microstructure image read in step S401 using a neural network model (step S402). Specifically, the estimation unit 212 estimates the material property value category to which the material related to the image belongs.

  In the neural network model according to the embodiment of the present invention, as described above, the output value of the classifier 309 is output for each category, and each output value indicates the probability of belonging to each category. For example, the classifier 309 outputs a continuous value between 0 and 1 for each category. In this case, the closer the value output by the classifier 309 is to 1, the higher the possibility that it belongs to that category.

  Since the classifier 309 outputs the probability independently for each category, the estimation unit 212 compiles and outputs the output of the classifier 309 as an expected value distribution. That is, the estimation unit 212 sums up the outputs of each category, and divides the output value of each category by the total value to obtain an expected value distribution.

Here, a histogram is displayed with each category on the horizontal axis and the expected value (probability) to which the input image data belongs to each category on the vertical axis. The estimation result shown in FIG. 8 shows that the expected value belonging to category 3 is approximately 60% for the tensile strength of the steel material indicated by the microstructure image. It also indicates that an expected value belonging to another category is 10% or less. In this way, the material identification can be grasped based on the display of the estimation result.

  Another example of the result display of the estimation process is shown in FIG. The estimation result shown in FIG. 9 shows that the expected value belonging to category 1 or 2 is about 35% for the tensile strength of the steel material indicated by the microstructure image. It also indicates that the expected value belonging to category 3 is about 20%. That is, in this case, it can be understood that the probability of the estimation result is low compared to FIG.

  FIGS. 10 and 11 show examples of estimation accuracy tables obtained by extracting only samples of evaluation data having an expected value for one category of 60% or more, that is, samples having high estimation certainty. As shown in FIGS. 10 and 11, the estimation accuracy of categories 1 to 3 is 100% for samples whose expected value is 60% or more. That is, when the certainty factor is high, it can be confirmed that the estimation accuracy is extremely high.

  Thus, according to this embodiment, the microstructure image of the test piece imaged with the microscope is read, and the material characteristic of the material which concerns on the said microstructure image is estimated with a neural network model. Thereby, the characteristic value can be efficiently estimated using the microstructure image of the material observed with a scanning electron microscope or the like. It is also possible to grasp how probable the estimation result is based on the certainty of estimation.

  In the present embodiment, a small image of 24 × 24 pixels is used as the learning data and the evaluation data by cropping. The small image corresponds to an area of about 30 μm square. In other words, in the present embodiment, the material characteristics are estimated based on information about a region about 30 μm square in the image captured by the microscope. The size is determined by estimating the material properties depending on the crystal grain size, since the crystal grain size of the steel material targeted in the present embodiment is about 10 μm. In the estimation of material properties, the acquired image size and resolution (number of pixels) depending on the subject of interest, such as when the subject property depends on the crystal grain size of the steel material, or when it is necessary to consider even fine precipitates, etc. ) Must be changed accordingly. Therefore, in the present embodiment, the size of the image region is determined based on necessary information on the crystal structure, and the material characteristics of the material are estimated based on the information on the image region. Similar to the size of the image region, the resolution of the image is appropriately determined based on necessary information on the crystal structure. As described above, in the present embodiment, the imaging size and resolution are selected so that the information necessary for estimating the material characteristics is sufficiently included and the calculation load is taken into consideration.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the functions included in each means, each step, etc. can be rearranged so that there is no logical contradiction, and a plurality of means, steps, etc. can be combined or divided into one. .

  For example, the system 100 according to the present embodiment has been described as an example including the first information processing device 601 and the second information processing device 701. However, these devices may be combined and processed by one information processing device. Good. In this case, the information processing apparatus saves the microstructure image captured by the microscope 600 as electronic data, and transmits the electronic data to the material property estimation apparatus 200. Further, the material property value measured by the material property measuring apparatus 700 is stored as electronic data, and the electronic data is transmitted to the material property estimating apparatus 200. Moreover, the material characteristic estimation apparatus 200 may perform the processing in the first information processing apparatus 601 and the second information processing apparatus 701.

DESCRIPTION OF SYMBOLS 100 System 200 Material characteristic estimation apparatus 201 Arithmetic processing part 202 ROM
203 Material property estimation program 204 RAM
205 Bus 206 Information Reading Unit 207 Image Preprocessing Unit 208 Discretization Processing Unit 209 Prediction Model Creation Unit 210 Output Unit 211 Information Reading Unit 212 Estimation Unit 213 Output Unit 300 Display Device 400 Input Device 500 Storage Device 600 Microscope 601 First Information Processing Device 700 Material property measuring device 701 Second information processing device 301 First convolution layer 302 First pooling layer 303 First local response normalization filter 304 Second convolution layer 305 Second local response normalization filter 306 Second pooling layer 307 First 1 Fully connected layer 308 2nd fully connected layer 309 Classifier 41 Input image 43 Filter 44 Each pixel 45 First feature map 46 Filter 47 Pixel 48 Second feature map 49 Connected layer 50 Classifier (category output)

Claims (6)

An information reading unit for reading an image of a material imaged by a microscope;
An estimation unit for estimating material properties of the material by a neural network model based on the image;
An output unit for outputting an estimation result by the estimation unit;
With
The said estimation result is a material characteristic estimation apparatus containing the reliability information of estimation.   The material property estimation apparatus according to claim 1, wherein the estimation unit estimates a material property value category to which the material belongs.   2. The material property estimation apparatus according to claim 1, wherein the output unit outputs an expected value distribution of a category of a material property value to which the material belongs as the certainty factor information.   The material property estimation apparatus according to claim 1, wherein the material property includes tensile strength.   The said estimation part determines the size of an image area | region based on the information of the crystal structure currently imaged by the said image, and estimates the material characteristic of the said material based on the information of this image area | region. The material property estimation apparatus according to one item. Reading an image of the material imaged by a microscope;
Estimating material properties of the material by a neural network model based on the image;
Outputting an estimation result of the estimating step;
Including
The estimation result includes a material property estimation method including reliability information of estimation.