KR102803897B1 - Method and apparatus for training deep learning model to infer rule from visual information - Google Patents

Abstract

A method for training a deep learning model for visual inference according to one embodiment of the present invention may include: obtaining first problem information including first rule image information and first correct answer candidate image information; determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information; determining first pseudo-label information based on the first learning matrix information; and training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information by receiving the first learning matrix information based on the first pseudo-label information.

Description

{METHOD AND APPARATUS FOR TRAINING DEEP LEARNING MODEL TO INFER RULE FROM VISUAL INFORMATION}

The present invention relates to a method and apparatus for training a deep learning model to infer rules from visual information.

The Raven Matrices Test (RPM) is one of the psychological test tools for assessing matrix reasoning ability, and is used to measure human reasoning, problem solving, and abstract thinking ability. In addition, the Raven Matrices Test does not rely heavily on language ability, and can be used in various age and language groups, and can also be used mainly for intelligence testing, and to assess talent potential in educational and occupational situations.

Recently, Raven's matrix test has been used to measure the visual reasoning ability of AI models to discover rules in a given problem. Accordingly, many attempts have been made to improve the visual reasoning ability of AI models.

However, according to conventional technology, since the difference in domain between input data was not considered, the number of correct and incorrect labels was imbalanced during the learning process of the artificial intelligence model, which resulted in the artificial intelligence model tending to over-predict.

Accordingly, there is a need to develop technologies to alleviate the overprediction of deep learning models and improve their generalization ability to infer rules contained in visual information.

Korean Patent Publication No. 10-2023-0163614 (2023.12.01)

The problem to be solved by the present invention is to secure stability in fields with high prediction sensitivity, such as medical image classification and autonomous driving, by improving the generalization ability of a deep learning model for visual inference by using label smoothing that reflects domain differences.

However, the problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the description below.

A method for training a deep learning model to infer rules from visual information according to one embodiment of the present invention may include: obtaining first problem information including first rule image information and first correct answer candidate image information; determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information; determining first pseudo-label information based on the first learning matrix information; and training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information by receiving the first learning matrix information based on the first pseudo-label information.

In addition, the first rule image information may include six first rule image panels representing rules regarding objects, properties, and relationships, and the first problem information may include a first problem matrix having a size of 3X3 configured to infer a first target image panel to be located next to two first problem image panels based on the six first rule image panels.

Here, the first problem matrix may be configured such that the first target image panel is determined as one of a plurality of first correct answer candidate image panels included in the first correct answer candidate image information.

Meanwhile, the step of determining the first learning matrix information may include a step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value.

In addition, the step of determining the first learning matrix information may include the step of determining the first learning matrix by respectively inputting a plurality of first correct answer candidate image panels and a plurality of second correct answer candidate image panels included in the changed first correct answer candidate image information into the first target image panel.

Meanwhile, the step of determining the first pseudo-label information may include: a step of extracting a feature for each row included in the first learning matrix; a step of determining a binarized hard label based on the feature for each row included in the first learning matrix; and a step of determining the first pseudo-label information including a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information from the binarized hard label using label smoothing.

In addition, the step of training the deep learning model may include a step of training the deep learning model through backpropagation to minimize a loss function determined using features for each row included in the first learning matrix and the first pseudo-label information.

In addition, the first correct answer matrix information may include information about a row among each row included in the first learning matrix that includes only the first rule image panel, the first correct answer candidate matrix information may include information about a row among each row included in the first learning matrix that includes the first correct answer candidate image panel, and the first incorrect answer matrix information may include information about a row among each row included in the first learning matrix that includes the second correct answer candidate image panel.

Meanwhile, a visual inference method using a pre-learned deep learning model according to one embodiment of the present invention includes a step of receiving problem information including rule image information and correct answer candidate image information; and a step of determining a panel corresponding to a target image panel from among correct answer candidate image panels included in the correct answer candidate image information using a pre-learned deep learning model, wherein the deep learning model may be pre-learned by a learning process including a step of obtaining first problem information including first rule image information and first correct answer candidate image information; a step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in second problem information; a step of determining first pseudo-label information based on the first learning matrix information; and a step of training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo-label information by receiving the first learning matrix information.

According to another embodiment of the present invention, a device for training a deep learning model to infer rules from visual information includes: a memory in which a deep learning model training program is stored; and a processor for loading the deep learning model training program from the memory and executing the deep learning model training program, wherein the processor obtains first problem information including first rule image information and first correct answer candidate image information, determines first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in second problem information, determines first pseudo label information based on the first learning matrix information, and trains the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo label information.

In addition, the first rule image information may include six first rule image panels representing rules regarding objects, properties, and relationships, and the first problem information may include a first problem matrix having a size of 3X3 configured to infer a first target image panel to be located next to two first problem image panels based on the six first rule image panels.

Here, the first problem matrix may be configured such that the first target image panel is determined as one of a plurality of first correct answer candidate image panels included in the first correct answer candidate image information.

Meanwhile, the processor may change the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value.

Additionally, the processor can determine a first learning matrix by inputting a plurality of first correct answer candidate image panels and a plurality of second correct answer candidate image panels included in the changed first correct answer candidate image information into the first target image panel, respectively.

Meanwhile, the processor may extract features for each row included in the first learning matrix, determine a binarized hard label based on the features for each row included in the first learning matrix, and determine the first pseudo-label information including a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information from the binarized hard label using label smoothing.

Additionally, the processor can train the deep learning model through backpropagation to minimize a loss function determined using features for each row included in the first learning matrix and the first pseudo-label information.

In addition, the first correct answer matrix information may include information about a row among each row included in the first learning matrix that includes only the first rule image panel, the first correct answer candidate matrix information may include information about a row among each row included in the first learning matrix that includes the first correct answer candidate image panel, and the first incorrect answer matrix information may include information about a row among each row included in the first learning matrix that includes the second correct answer candidate image panel.

According to another embodiment of the present invention, a visual inference device using a pre-learned deep learning model comprises: a memory in which a visual inference program is stored; and a processor for loading the visual inference program from the memory and executing the visual inference program, wherein the processor receives problem information including rule image information and correct answer candidate image information, and determines a panel corresponding to a target image panel from among correct answer candidate image panels included in the correct answer candidate image information using a pre-learned deep learning model, wherein the deep learning model comprises: a step of obtaining first problem information including first rule image information and first correct answer candidate image information; a step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in second problem information; a step of determining first pseudo-label information based on the first learning matrix information; And based on the first pseudo-label information, the first learning matrix information may be input and the deep learning model may be trained to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information.

A computer-readable recording medium storing a computer program according to another embodiment of the present invention may include instructions for causing the processor to perform a method of training a deep learning model to infer rules from visual information, the method comprising: obtaining first problem information including first rule image information and first correct answer candidate image information; determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information; determining first pseudo-label information based on the first learning matrix information; and training a deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo-label information.

According to another embodiment of the present invention, a computer program stored in a computer-readable recording medium may include instructions for causing the processor to perform a method of training a deep learning model to infer rules from visual information, the method comprising: obtaining first problem information including first rule image information and first correct answer candidate image information; determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information; determining first pseudo-label information based on the first learning matrix information; and training a deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo-label information.

According to an embodiment of the present invention, by training a deep learning model to increase label values for correct answer candidates included in the same domain using label smoothing, the flexibility of visual inference using the deep learning model can be secured, and over-prediction can be prevented.

FIG. 1 is a block diagram showing a deep learning model training device according to an embodiment of the present invention.
FIG. 2 is a block diagram conceptually illustrating the function of a deep learning model training program according to an embodiment of the present invention.
Figure 3 is a flowchart illustrating a deep learning model training method according to one embodiment of the present invention.
FIG. 4 is a diagram exemplarily showing training a deep learning model using label smoothing according to one embodiment of the present invention.
FIG. 5 is a block diagram conceptually illustrating the function of a visual inference program according to one embodiment of the present invention.
Figure 6 is a flowchart illustrating a visual inference method according to one embodiment of the present invention.
FIG. 7 is a diagram exemplarily showing inferring rules from given visual information using a pre-learned deep learning model according to one embodiment of the present invention.

The advantages and features of the present invention, and the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are provided only to make the disclosure of the present invention complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

In describing embodiments of the present invention, if it is judged that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the embodiments of the present invention, and these may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification.

FIG. 1 is a block diagram showing a deep learning model training device according to an embodiment of the present invention.

Referring to FIG. 1, a deep learning model training device (100) may include a processor (110), an input/output device (120), and a memory (130).

The processor (110) can control the overall operation of the deep learning model training device (100).

The processor (110) can receive problem information (e.g., first problem information and second problem information) using an input/output device (120).

In the present invention, the visual information is visual information representing objects, properties, and relationships for measuring human intelligence, and may include a plurality of image panels. Here, the image panel is an image representing a shape contained within a panel (or plate) of a certain size, and the shape, size, color, position, and number of the shape may be set according to the purpose.

In addition, in the present invention, the problem information may include rule image information and correct answer candidate image information. Here, the rule image information may include a plurality of image panels representing rules regarding objects, properties, and relationships, and the correct answer candidate image information may include a plurality of image panels in which there is a possibility of a correct answer to the problem.

Specifically, the problem information may include a problem matrix of a predetermined size, and the problem matrix may be configured to infer a plurality of rule image panels, a plurality of problem image panels, and a target image panel located next to the plurality of problem image panels. Here, the problem matrix may be configured such that a common rule exists in each row.

In the present invention, the first problem information and the second problem information are described as being input through the input/output device (120), but are not limited thereto. That is, according to an embodiment, the deep learning model learning device (100) may include a transceiver (not shown), and the deep learning model learning device (100) may receive at least one of the first problem information and the second problem information using the transceiver (not shown), and at least one of the first problem information and the second problem information may be generated within the deep learning model learning device (100).

Here, the first problem information and the second problem information are information representing a problem for inferring rules from given visual information, and the first problem and the second problem are in different domains and may have different visual information.

The processor (110) obtains first problem information including first rule image information and first correct answer candidate image information, determines first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information, determines first pseudo-label information based on the first learning matrix information, and trains a deep learning model to receive the first learning matrix information and classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo-label information.

The input/output device (120) may include one or more input devices and/or one or more output devices. For example, the input devices may include a microphone, a keyboard, a mouse, a touch screen, etc., and the output devices may include a display, a speaker, etc.

The memory (130) can store a deep learning model training program (200) and information required for executing the deep learning model training program (200).

In this specification, a deep learning model training program (200) may mean software including commands for training a deep learning model by inputting problem information including rule image information and correct answer candidate image information and using label information determined to infer rules from given visual information.

The processor (110) can load the deep learning model training program (200) and information necessary for executing the deep learning model training program (200) from the memory (130) to execute the deep learning model training program (200).

The processor (110) executes a deep learning model training program (200) to input first problem information including first rule image information and first correct answer candidate image information, and train a deep learning model to classify correct answer matrix information, correct answer candidate matrix information, and incorrect answer matrix information based on first label information determined using second correct answer candidate image information included in second problem information.

The functions and/or operations of the deep learning model training program (200) will be examined in detail with reference to Fig. 2.

FIG. 2 is a block diagram conceptually illustrating the function of a deep learning model training program according to an embodiment of the present invention.

Referring to FIG. 2, a deep learning model training program (200) may include a problem information acquisition unit (210), a learning matrix information determination unit (220), and a deep learning model training unit (230).

The problem information acquisition unit (210), the learning matrix information determination unit (220), and the deep learning model learning unit (230) illustrated in Fig. 2 conceptually divide the functions of the deep learning model learning program (200) in order to easily explain the functions of the deep learning model learning program (200), but are not limited thereto. According to embodiments, the functions of the problem information acquisition unit (210), the learning matrix information determination unit (220), and the deep learning model learning unit (230) can be merged/separated, and can also be implemented as a series of commands included in one program.

First, the problem information acquisition unit (210) can acquire first problem information including first rule image information and first correct answer candidate image information.

Specifically, the first rule image information may include six first rule image panels representing rules regarding objects, properties, and relationships, and the first correct answer candidate image information may include a plurality of first correct answer candidate image panels.

In addition, the first problem information may include a first problem matrix of size 3X3, which is configured to infer a first target image panel to be located next to two first problem image panels based on six rule image panels. The first problem matrix may be expressed as in the following mathematical expression 1.

Here, is the first rule image panel, is the first problem image panel, may refer to a first target image panel.

Specifically, the first problem matrix can be configured such that the first target image panel is determined as one of a plurality of first correct answer candidate image panels included in the first correct answer candidate image information.

Meanwhile, a common rule may exist for each row of the first problem matrix. The rule according to one embodiment of the present invention may include at least one of a constant rule regarding the constant property value of an object across a row, a progression rule regarding the increasing or decreasing property value of an object toward the right image panel of a row, an arithmetic rule regarding the result of addition or subtraction of property values of the left image panel and the center image panel of a row being the same as the property value of the right image panel of the row, and a distribute three rule regarding the selection of three values for a specific property and the arrangement of the three values in each panel of a row with only a different order.

Next, the learning matrix information determination unit (220) can determine the first learning matrix information from the first problem information by using the second correct answer candidate image information included in the second problem information.

Specifically, the learning matrix information determination unit (220) can change the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value.

For example, if the preset value is 4 and there are 8 first correct answer candidate image panels, the learning matrix information determining unit (220) can change the first correct answer candidate image information by replacing the 4 first correct answer candidate image panels with 4 second correct answer candidate image panels. Here, the changed first correct answer candidate image information can include 4 first correct answer candidate image panels and 4 second correct answer candidate image panels.

Meanwhile, the preset values according to one embodiment of the present invention are only examples and may be variously changed within the scope that can achieve the purpose of the present invention.

In addition, the learning matrix information determining unit (220) can determine the first learning matrix by inputting a plurality of first correct answer candidate image panels and a plurality of second correct answer candidate image panels included in the changed first correct answer candidate image information into the first target image panel, and the first learning matrix can be expressed as in the following mathematical expression 2.

Here, N is the number of rows of the first learning matrix, may refer to the first correct answer candidate image panel, may mean a second correct answer candidate image panel, and M may mean the number of first correct answer candidate image panels included in the changed first correct answer candidate image information.

For example, if the number of first correct answer candidate image panels and second correct answer candidate image panels included in the first correct answer candidate image information is 8, the number of rows of the first learning matrix can be determined as 10.

At this time, all the first problem image panels existing in the first column of the first learning matrix are the same, and all the first problem image panels existing in the second column are the same.

Meanwhile, the first learning matrix information may include first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information as information for each row included in the first learning matrix (i.e., information for three image panels).

More specifically, the first correct answer matrix information may mean information about a row that contains only the first rule image panel among each row included in the first learning matrix.

Additionally, the first correct answer candidate matrix information may mean information about a row that includes a first correct answer candidate image panel among each row included in the first learning matrix.

Additionally, the first error matrix information may mean information about a row that includes a second correct answer candidate image panel among each row included in the first learning matrix.

Meanwhile, the learning matrix information determination unit (220) can sequentially sort the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information included in the first learning matrix information.

For example, if the number of first correct candidate image panels and second correct candidate image panels included in the changed first correct candidate image information is 8, the first correct matrix information may mean information about the first row and the second row of the first learning matrix, the first correct candidate matrix information may mean information about the third to sixth rows of the first learning matrix, and the first incorrect matrix information may mean information about the seventh to tenth rows of the first learning matrix.

Next, the deep learning model learning unit (230) can determine first pseudo label information based on the first learning matrix information. Here, the first pseudo label information can be determined by an unsupervised learning method.

Specifically, the deep learning model learning unit (230) can extract features for each row included in the first learning matrix. For example, the deep learning model learning unit (230) can extract feature vectors for each row included in the first learning matrix using a deep learning model (e.g., ResNet-18).

Additionally, the features for each row included in the first learning matrix can be normalized based on the features for the first correct answer matrix included in the first learning matrix.

For example, the deep learning model learning unit (230) can normalize by performing an operation (e.g., a subtraction operation) on the average of the feature vector for the first row and the feature vector for the second row with the feature vector for each row included in the first learning matrix.

Additionally, the deep learning model learning unit (230) can determine a binarized hard label based on the features for each row included in the first learning matrix.

For example, the deep learning model learning unit (230) may determine the label value for the first correct answer matrix information as 1, and the label values for the first correct answer candidate matrix information and the first incorrect answer matrix information as 0.

Meanwhile, the deep learning model learning unit (230) can determine first pseudo-label information including a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information from the binarized hard label using label smoothing.

Specifically, the deep learning model learning unit (230) can determine a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information by considering a smoothing value set by the user and the number of first correct answer candidate image panels to be replaced. Here, the label value for the first incorrect answer matrix information may still be 0.

For example, the soft label for the first correct answer matrix information and the soft label for the first correct answer candidate matrix information can be expressed as in the following mathematical expression 3.

Here, can mean the soft label value for the first correct answer matrix information, can mean a soft label value for the first correct answer candidate matrix information, can mean a smoothing value, may mean the number of first correct answer candidate image panels to be replaced.

In addition, the deep learning model learning unit (230) can receive first learning matrix information based on the first pseudo-label information and train the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information.

Specifically, the deep learning model learning unit (230) can train the deep learning model through backpropagation to minimize a loss function determined using the feature vector and first pseudo-label information for each row included in the first learning matrix.

In this way, by training the deep learning model to lower the label value for the first correct answer matrix information and to raise the label value for the first correct answer candidate matrix information, the effect of securing flexibility in visual inference using the deep learning model and preventing over-prediction can be achieved.

Figure 3 is a flowchart illustrating a deep learning model training method according to one embodiment of the present invention.

Referring to FIGS. 2 and 3, the problem information acquisition unit (210) can acquire first problem information including first rule image information and first correct answer candidate image information (S310).

Next, the learning matrix information determination unit (220) can determine first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information using second correct answer candidate image information included in the second problem information (S320).

Next, the deep learning model learning unit (230) can determine the first pseudo-label information based on the first learning matrix information (S330). Here, the first pseudo-label information is determined based on an unsupervised learning method, and can include a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information.

Next, the deep learning model learning unit (230) can receive first learning matrix information based on the first pseudo-label information and train the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information (S340).

FIG. 4 is a diagram exemplarily showing training a deep learning model using label smoothing according to one embodiment of the present invention.

Referring to FIGS. 2 and 4, first problem information (410) is illustrated, which includes a plurality of first rule image panels (corresponding to identification numbers 1 to 6), a plurality of first correct answer candidate image panels (corresponding to identification numbers 9 to 12), and a plurality of second correct answer candidate image panels (corresponding to identification numbers 13 to 16).

Here, the learning matrix information determining unit (220) can determine the first learning matrix information (411) by inputting a plurality of first correct answer candidate image panels (corresponding to identification numbers 9 to 12) and a plurality of second correct answer candidate image panels (corresponding to identification numbers 13 to 16) into the first target image panel, respectively.

In addition, the deep learning model learning unit (230) can input each row included in the first learning matrix into the deep learning model (420) to extract features for each row included in the first learning matrix.

In addition, the deep learning model learning unit (230) can determine a binarized hard label (431) based on the features for each row included in the first learning matrix. Here, the label value corresponding to the first correct answer matrix information is 1, and the label value corresponding to the first correct answer candidate matrix information and the first incorrect answer matrix information is 0.

In addition, the first pseudo-label information (430) including the soft labels (432) for the first correct answer matrix information and the first correct answer candidate matrix information can be determined by using label smoothing. Here, the label value corresponding to the first correct answer matrix information is 0.95, the label value corresponding to the first correct answer candidate matrix information is 0.05, and the label value corresponding to the first incorrect answer matrix information is 0.

Meanwhile, the deep learning model learning unit (230) can receive first learning matrix information (411) based on first pseudo-label information (430) and train the deep learning model (420) to classify first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information.

FIG. 5 is a block diagram conceptually illustrating the function of a visual inference program according to one embodiment of the present invention.

A visual reasoning program (300) according to one embodiment of the present invention may be executed in a visual reasoning device (not shown) for knowledge distillation, and the visual reasoning device may include a processor (not shown) and a memory (not shown).

The above processor (not shown) can control the overall operation of the visual inference device (not shown).

A processor (not shown) obtains problem information including rule image information and correct answer candidate image information, and inputs a problem matrix included in the problem information into a pre-trained deep learning model to determine a panel corresponding to a target image panel among correct answer candidate image panels included in the correct answer candidate image information.

Here, the pre-trained deep learning model may be trained to receive first learning matrix information based on first pseudo-label information and classify first correct matrix information, first correct candidate matrix information, and first incorrect matrix information.

Memory (not shown) can store a visual inference program (300) and information necessary for executing the visual inference program (300).

In this specification, a visual inference program (300) may mean software including commands for obtaining problem information including rule image information and correct answer candidate image information, inputting first learning matrix information based on first pseudo-label information into a problem matrix included in the problem information, classifying the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information into a pre-trained deep learning model, and determining a panel corresponding to a target image panel among correct answer candidate image panels included in the correct answer candidate image information.

A processor (not shown) can load a visual inference program (300) and information necessary for executing the visual inference program (300) from a memory (not shown) to execute the visual inference program (300).

A processor (not shown) may execute a visual inference program (300) to determine a panel corresponding to a target image panel among the candidate image panels to infer a rule from given visual information.

The functions and/or operations of the visual inference program (300) will be examined in detail with reference to Fig. 5.

The problem information acquisition unit (310) and the visual inference unit (320) illustrated in Fig. 5 conceptually divide the functions of the visual inference program (300) in order to easily explain the functions of the visual inference program (300), but are not limited thereto. According to embodiments, the functions of the problem information acquisition unit (310) and the visual inference unit (320) can be merged/separated, and can also be implemented as a series of commands included in one program.

First, the problem information acquisition unit (310) can receive problem information including rule image information and correct answer candidate image information. Here, unlike the learning of the deep learning model, the correct answer candidate image information is not changed.

Next, the visual inference unit (320) can determine a panel corresponding to a target image panel among the correct answer candidate image panels included in the correct answer candidate image information using a pre-learned deep learning model.

Specifically, the visual inference unit (320) can input each of a plurality of correct answer candidate image panels included in the problem information into the problem matrix to determine a rule inference matrix. In addition, the visual inference unit (320) can determine the correct answer candidate image panel with the highest predicted value, which is output by a pre-learned deep learning model by inputting the rule inference matrix, as the panel corresponding to the target image panel.

Figure 6 is a flowchart illustrating a visual inference method according to one embodiment of the present invention.

Referring to FIGS. 5 and 6, the problem information acquisition unit (310) can receive problem information including rule image information and correct answer candidate image information (S610).

Next, the visual inference unit (320) can input the pre-learned deep learning model to determine a panel corresponding to the target image panel among the correct answer candidate image panels included in the correct answer candidate image information (S620).

Here, the deep learning model may be pre-trained to obtain first problem information including first rule image information and first correct answer candidate image information, determine first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information, determine first pseudo-label information based on the first learning matrix information, and classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information based on the first pseudo-label information.

FIG. 7 is a diagram exemplarily showing inferring rules from visual information using a pre-learned deep learning model according to one embodiment of the present invention.

Referring to FIGS. 5 and 7, the problem information acquisition unit (310) can receive problem information (710) including a plurality of rule image panels (corresponding to identification numbers 1 to 6) and a plurality of correct answer candidate image panels (corresponding to identification numbers 9 to 16).

The visual inference unit (320) can determine a rule inference matrix (711) by inputting each of the plurality of correct answer candidate image panels included in the problem information (710) into the problem matrix.

In addition, the visual inference unit (320) can input each row included in the rule inference matrix (711) into a pre-learned deep learning model (720) to output a prediction value regarding rule inference from the given visual information.

Specifically, the visual inference unit (320) can use a pre-learned deep learning model (720) to input a rule inference matrix (711) and determine the correct answer candidate image panel with the highest output predicted value as the panel corresponding to the target image panel.

Here, since the prediction value (721) for the third row included in the rule inference matrix (711) is the highest at 0.87, the visual inference unit (320) can determine the correct answer candidate image panel corresponding to the identification number 9 as the target image panel for inferring the rule from the given visual information.

The combination of each block of the block diagram and each step of the flow diagram attached to the present invention may be performed by computer program instructions. These computer program instructions may be installed in an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions executed by the encoding processor of the computer or other programmable data processing equipment create a means for performing the functions described in each block of the block diagram or each step of the flow diagram. These computer program instructions may also be stored in a computer-available or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement the functions in a specific manner, so that the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes an instruction means for performing the functions described in each block of the block diagram or each step of the flow diagram. Since the computer program instructions can also be installed on a computer or other programmable data processing apparatus, a series of operational steps are performed on the computer or other programmable data processing apparatus to create a computer-executable process, and the instructions that cause the computer or other programmable data processing apparatus to perform the steps for executing the functions described in each block of the block diagram and each step of the flowchart can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

Additionally, each block or step may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specified logical function(s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps depicted in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order, depending on the functionality they serve.

The above description is merely an illustrative description of the technical idea of the present invention, and those skilled in the art will appreciate that various modifications and variations may be made without departing from the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

100: Deep Learning Model Training Device
200: Deep Learning Model Training Program
210: Problem information acquisition department
220: Learning matrix information decision unit
230: Deep Learning Model Training Section
300: Visual Reasoning Program
310: Problem information acquisition department
320: Visual Reasoning Unit

Claims (20)

A method for training a deep learning model to infer rules from visual information.
A step of obtaining first problem information including first rule image information and first correct answer candidate image information;
A step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information;
A step of determining first pseudo-label information based on the first learning matrix information; and
A step of training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information by inputting the first learning matrix information based on the first pseudo-label information,
The step of determining the above first learning matrix information is:
A step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value,
The second problem information is characterized in that it has visual information of a different domain from the first problem information.
How to train a deep learning model.
In the first paragraph,
The above first rule image information includes six first rule image panels representing rules regarding objects, properties and relationships,
The above first problem information includes a first problem matrix of size 3X3, which is configured to infer a first target image panel to be located next to two first problem image panels based on the six first rule image panels,
The above first problem matrix is expressed as mathematical expression 1.
How to train a deep learning model.
[Mathematical formula 1]

(Here, is the first rule image panel, is the first problem image panel, refers to the first target image panel, and there is a common rule for each row of the first problem matrix.)
In the second paragraph,
The above first problem matrix is configured such that the first target image panel is determined as one of a plurality of first correct answer candidate image panels included in the first correct answer candidate image information.
How to train a deep learning model.
delete In the third paragraph,
The step of determining the above first learning matrix information is:
Including a step of determining a first learning matrix by inputting a plurality of first correct answer candidate image panels and a plurality of second correct answer candidate image panels included in the changed first correct answer candidate image information into the first target image panel,
The above first learning matrix is expressed as mathematical expression 2.
How to train a deep learning model.
[Mathematical formula 2]

(Here, N is the number of rows of the first learning matrix, refers to the first correct answer candidate image panel, means the second correct answer candidate image panel, M means the number of first correct answer candidate image panels included in the changed first correct answer candidate image information, all first question image panels existing in column 1 are the same, and all first question image panels existing in column 2 are the same.)
In clause 5,
The step of determining the first medical label information is:
A step of extracting features for each row included in the first learning matrix;
A step of determining a binarized hard label based on the features for each row included in the first learning matrix; and
A step of determining the first pseudo-label information including a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information from the binarized hard label using label smoothing.
How to train a deep learning model.
In Article 6,
The steps for training the above deep learning model are:
A step of training the deep learning model through backpropagation to minimize a loss function determined using the features for each row included in the first learning matrix and the first pseudo-label information.
How to train a deep learning model.
In Article 7,
The above first correct answer matrix information includes information about a row that contains only the first rule image panel among each row included in the first learning matrix,
The above first correct answer candidate matrix information includes information about a row that includes the first correct answer candidate image panel among each row included in the first learning matrix,
The above first error matrix information includes information about a row that includes the second correct answer candidate image panel among each row included in the first learning matrix.
How to train a deep learning model.
A visual inference method using a pre-trained deep learning model,
A step of obtaining problem information including rule image information and correct answer candidate image information; and
A step of determining a panel corresponding to a target image panel among the candidate image panels included in the candidate image information using a pre-trained deep learning model,
The above deep learning model is,
A step of obtaining first problem information including first rule image information and first correct answer candidate image information;
A step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information;
A step of determining first pseudo-label information based on the first learning matrix information; and
A step of training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information by inputting the first learning matrix information based on the first pseudo-label information,
The step of determining the above first learning matrix information is:
A step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value,
The above second problem information is pre-learned by a learning process characterized by having visual information of a different domain from the above first problem information.
Visual reasoning methods.
A device that trains a deep learning model to infer rules from visual information.
Memory where the deep learning model training program is stored; and
A processor for loading the deep learning model training program from the memory and executing the deep learning model training program,
The above processor,
Obtain first problem information including first rule image information and first correct answer candidate image information,
Using the second correct answer candidate image information included in the second problem information, first learning matrix information including the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information is determined from the first problem information,
Determine the first pseudo-label information based on the first learning matrix information,
Based on the first pseudo-label information, the deep learning model is trained to receive the first learning matrix information and classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information.
The above processor,
By replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information, based on a preset value, the first correct answer candidate image information is changed,
The second problem information is characterized in that it has visual information of a different domain from the first problem information.
Deep learning model training device.
In Article 10,
The above first rule image information includes six first rule image panels representing rules regarding objects, properties and relationships,
The above first problem information includes a first problem matrix of size 3X3, which is configured to infer a first target image panel to be located next to two first problem image panels based on the six first rule image panels,
The above first problem matrix is expressed as mathematical expression 1.
Deep learning model training device.
[Mathematical formula 1]

(Here, is the first rule image panel, is the first problem image panel, refers to the first target image panel, and there is a common rule for each row of the first problem matrix.)
In Article 11,
The above first problem matrix is configured such that the first target image panel is determined as one of a plurality of first correct answer candidate image panels included in the first correct answer candidate image information.
Deep learning model training device.
delete In Article 12,
The above processor,
A first learning matrix is determined by inputting a plurality of first correct answer candidate image panels and a plurality of second correct answer candidate image panels included in the changed first correct answer candidate image information into the first target image panel, respectively.
The above first learning matrix is expressed as mathematical expression 2.
Deep learning model training device.
[Mathematical formula 2]

(Here, N is the number of rows of the first learning matrix, refers to the first correct answer candidate image panel, means the second correct answer candidate image panel, M means the number of first correct answer candidate image panels included in the changed first correct answer candidate image information, all first question image panels existing in column 1 are the same, and all first question image panels existing in column 2 are the same.)
In Article 14,
The above processor,
Extract features for each row included in the first learning matrix,
Determine a binarized hard label based on the features for each row included in the first learning matrix,
Using label smoothing, the first pseudo-label information including a soft label for the first correct answer matrix information and a soft label for the first correct answer candidate matrix information are determined from the binarized hard labels.
Deep learning model training device.
In Article 15,
The above processor,
The deep learning model is trained through backpropagation to minimize the loss function determined using the features for each row included in the first learning matrix and the first pseudo-label information.
Deep learning model training device.
In Article 16,
The above first correct answer matrix information includes information about a row that contains only the first rule image panel among each row included in the first learning matrix,
The above first correct answer candidate matrix information includes information about a row that includes the first correct answer candidate image panel among each row included in the first learning matrix,
The above first error matrix information includes information about a row that includes the second correct answer candidate image panel among each row included in the first learning matrix.
Deep learning model training device.
As a visual inference device using a pre-trained deep learning model,
Memory where the visual reasoning program is stored; and
A processor for loading the visual reasoning program from the memory and executing the visual reasoning program,
The above processor,
Enter problem information including rule image information and answer candidate image information,
Using a pre-trained deep learning model, a panel corresponding to a target image panel is determined among the candidate image panels included in the candidate image information.
The above deep learning model is,
A step of obtaining first problem information including first rule image information and first correct answer candidate image information;
A step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information;
A step of determining first pseudo-label information based on the first learning matrix information; and
A step of training the deep learning model to classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information by inputting the first learning matrix information based on the first pseudo-label information,
The step of determining the above first learning matrix information is:
A step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value,
The above second problem information is pre-learned by a learning process characterized by having visual information of a different domain from the above first problem information.
Visual reasoning device.
A computer-readable recording medium storing a computer program,
The above computer program, when executed by a processor,
A step of obtaining first problem information including first rule image information and first correct answer candidate image information;
A step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information;
A step of determining first pseudo-label information based on the first learning matrix information; and
Based on the first pseudo-label information, a step of training a deep learning model to receive the first learning matrix information and classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information is included.
The step of determining the above first learning matrix information is:
A step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value,
The second problem information is characterized in that it has visual information of a different domain from the first problem information.
A method comprising instructions for causing the processor to perform a method of training a deep learning model to infer rules from visual information.
Computer readable recording medium.
A computer program stored on a computer-readable recording medium,
The above computer program, when executed by a processor,
A step of obtaining first problem information including first rule image information and first correct answer candidate image information;
A step of determining first learning matrix information including first correct answer matrix information, first correct answer candidate matrix information, and first incorrect answer matrix information from the first problem information by using second correct answer candidate image information included in the second problem information;
A step of determining first pseudo-label information based on the first learning matrix information; and
Based on the first pseudo-label information, a step of training a deep learning model to receive the first learning matrix information and classify the first correct answer matrix information, the first correct answer candidate matrix information, and the first incorrect answer matrix information is included.
The step of determining the above first learning matrix information is:
A step of changing the first correct answer candidate image information by replacing at least some of the plurality of first correct answer candidate image panels with at least some of the plurality of second correct answer candidate image panels included in the second correct answer candidate image information based on a preset value,
The second problem information is characterized in that it has visual information of a different domain from the first problem information.
A method comprising instructions for causing the processor to perform a method of training a deep learning model to infer rules from visual information.
Computer program.

Images