JP2018190130A - Analyzer, analysis method, and analysis program - Google Patents

Abstract

A prediction model with high prediction accuracy can be constructed while reducing the time required for pipeline search. An extracting unit 151 extracts sample data from input data. The estimation unit 154 estimates the required time for the entire process based on the required time when some processing is performed using the sample data, and increases the sample rate when the estimated time is less than the time limit. . When the sample rate increases, the extraction unit 151 further extracts sample data. In addition, the search unit 152 is a combination that increases the prediction accuracy of a prediction model based on sample data that is constructed when applied from a combination of setting contents of a plurality of processes that are executed when the prediction model is constructed. Explore. In addition, the selection unit 153 selects a combination of setting contents in which the prediction accuracy of the prediction model based on the input data constructed when applied is greater than or equal to a predetermined threshold from the searched combinations. [Selection] Figure 1

Description

  The present invention relates to an analysis apparatus, an analysis method, and an analysis program.

  In recent years, application examples of data analysis using machine learning are increasing. On the other hand, in order to acquire statistics and machine learning knowledge essential for data analysis, medium- to long-term education is required. Therefore, a technique for supporting data analysis is disclosed so that non-experts can easily engage in data analysis without acquiring statistics and machine learning knowledge.

  For example, a technique is known in which accuracy is evaluated for each pipeline using a sequential model-based optimization (SMBO) and an optimum pipeline is searched (for example, Non-Patent Document 1 and 2). Here, the pipeline is a series of processes for constructing a prediction model, and includes preprocessing for input data, data learning based on hyperparameters, and the like. In addition, a technique for presenting a user with a small number of pipelines suitable for data to be analyzed among a large number of pipelines designed in advance by experts is known.

  In general, as the number of candidate settings for data pre-processing and prediction algorithms that make up the pipeline increases, the possibility of finding a pipeline that can build a prediction model with higher prediction accuracy increases. The time required for the search increases.

Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter, “Efficient and Robust Automated Machine Learning”, NIPS'15 Proceedings of the 28th International Conference on Neural Information Processing Systems, December 2015, PP. 2755-2763 Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar, “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”, arXiv: 1603.06560v3, cs.LG, November 2016

  However, the conventional technology for automating data analysis has a problem that it may not be possible to construct a prediction model with high prediction accuracy while reducing the time required for pipeline search. For example, it may be possible to reduce the time required for pipeline search by simply reducing the setting contents that are candidates for search, but in this case, sufficient search may not be performed and the accuracy of the prediction model may be reduced. .

  When data is input, the analyzer of the present invention extracts a predetermined ratio of data from the input data as sample data, and when the predetermined ratio increases, the input increases from the input data. An extraction unit that further extracts a predetermined ratio of data as the sample data, and every time the sample data is extracted by the extraction unit, when a part of the predetermined process is performed using the sample data Based on the required time, the time required for performing the predetermined process using the input data is estimated, and when the estimated time is less than a preset time limit, the predetermined ratio From a combination candidate of setting contents applied to a plurality of processes executed when constructing a prediction model based on predetermined data and an estimation unit that increases A search unit for searching for a combination of setting contents such that the prediction accuracy of the prediction model based on the sample data constructed when applied to a plurality of processes satisfies a predetermined condition, and the setting contents searched by the searching unit A selection unit that selects a combination of setting contents in which the prediction accuracy of the prediction model based on the input data constructed when applied to the plurality of processes is equal to or higher than a predetermined threshold from the combination of It is characterized by.

  According to the present invention, it is possible to construct a prediction model with high prediction accuracy while reducing the time required for pipeline search.

FIG. 1 is a diagram illustrating an example of the configuration of the analyzer according to the first embodiment. FIG. 2 is a diagram illustrating an example of a data configuration of setting information according to the first embodiment. FIG. 3 is a diagram illustrating an example of a data configuration of predictor information according to the first embodiment. FIG. 4 is a diagram for explaining a processing outline of the analysis apparatus according to the first embodiment. FIG. 5 is a diagram for explaining a pipeline search according to the first embodiment. FIG. 6 is a diagram for explaining the cross-validation according to the first embodiment. FIG. 7 is a diagram for explaining a pipeline search according to the first embodiment. FIG. 8 is a diagram for explaining a promising area according to the first embodiment. FIG. 9 is a diagram for explaining selection of a pipeline according to the first embodiment. FIG. 10 is a diagram for explaining a promising area according to the first embodiment. FIG. 11 is a diagram for explaining the synthesis of the prediction model according to the first embodiment. FIG. 12 is a diagram for explaining the synthesis of the prediction model according to the first embodiment. FIG. 13 is a flowchart showing the flow of processing of the analyzer according to the first embodiment. FIG. 14 is a flowchart showing the flow of processing of the analyzer according to the first embodiment. FIG. 15 is a diagram for explaining the effect of the first embodiment. FIG. 16 is a diagram for explaining the effect of the first embodiment. FIG. 17 is a flowchart showing a flow of processing of the analyzer according to the second embodiment. FIG. 18 is a diagram illustrating an example of a computer that executes an analysis program.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

[First Embodiment]
[Configuration of analyzer]
The configuration of the analyzer 10 will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of the configuration of the analyzer according to the first embodiment. As shown in FIG. 1, the analysis apparatus 10 is realized by a general-purpose computer such as a workstation or a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15. Prepare.

  The input unit 11 is realized using an input device such as a keyboard or a mouse, and inputs various instruction information to the control unit 15 in response to an input operation by the operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, an information communication device, and the like, and outputs a result of data analysis to the operator.

  The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between an external device such as a management server and the control unit 15 via a telecommunication line such as a LAN (Local Area Network) or the Internet. .

  The storage unit 14 is realized by a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. In the storage unit 14, a processing program for operating the analysis apparatus 10, data used during execution of the processing program, and the like are stored in advance or temporarily stored for each processing. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. In addition, the storage unit 14 stores setting information 141 and predictor information 142.

  Here, the setting information 141 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a data configuration of setting information according to the first embodiment. As illustrated in FIG. 2, the setting information 141 includes an execution order for each step and setting content candidates. The setting content candidate is a setting content candidate of the setting item corresponding to each step.

  In the example of FIG. 2, the setting information 141 indicates that there are “missing value complement method search”, “normalization method search”, and “feature selection method search” as steps. These steps correspond to data preprocessing.

  In the example of FIG. 2, the setting information 141 indicates that the step “feature selection method search” is the third step to be executed. The setting information 141 indicates that “decision tree”, “L1 regularization”, “ANOVA” and “no processing” are candidates for the setting contents of the setting item corresponding to the step “feature selection method search”. Show. In the example of FIG. 2, the setting item corresponding to the step “feature selection method search” is a method used for feature selection.

Next, the predictor information 142 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a data configuration of predictor information according to the first embodiment. As shown in FIG. 3, the predictor information 142 includes default parameters for each predictor. The default parameter is a default value of a hyper parameter of each predictor. For example, the predictor information 142 indicates that the default value of the hyperparameter N of the predictor A ₁ is 100.

  As illustrated in FIG. 1, the control unit 15 executes a processing program stored in a memory by an arithmetic processing device such as a CPU (Central Processing Unit), so that an extraction unit 151, a search unit 152, an estimation unit 154, It functions as a synthesis unit 155 and a verification unit 156. The search unit 152 includes a step selection unit 152a, an accuracy calculation unit 152b, and a setting content determination unit 152c. Note that these functional units may be implemented on different or different hardware.

  When data is input, the extraction unit 151 extracts a predetermined ratio of data from the input data as sample data. When the predetermined ratio increases, the extraction unit 151 increases the predetermined ratio of the input data. The data is further extracted as sample data. As illustrated in FIG. 4, the extraction unit 151 extracts sample data from learning data that is all input data. FIG. 4 is a diagram for explaining a processing outline of the analysis apparatus according to the first embodiment. Here, it is assumed that a predetermined ratio at which the extraction unit 151 extracts sample data, that is, a sampling rate in sampling, is set in advance.

  The search unit 152 changes the sample data to be constructed when applied to a plurality of processes from a combination of setting contents applied to a plurality of processes executed when a prediction model based on predetermined data is constructed. A combination of setting contents is searched for such that the prediction accuracy of the prediction model based thereon satisfies a predetermined condition.

  As illustrated in FIG. 4, the search unit 152 performs a search for each predictor. Further, the search unit 152 performs parameter search after determining the validity of the preprocessing, and selects a pipeline with cross-validation accuracy by cross-validation.

  Here, the process of the search part 152 is demonstrated using FIG. FIG. 5 is a diagram for explaining a pipeline search according to the first embodiment. The processing in the search unit 152 is performed by the step selection unit 152a, the accuracy calculation unit 152b, and the setting content determination unit 152c.

As illustrated in FIG. 5, first, the search unit 152 determines a pipeline necessary for constructing a prediction model by executing Steps 1 to 3 for each predictor. For example, the search unit 152, after performing the steps 1 to 3 for the predictor _{A 1,} executes the parameter search.

  The step selection unit 152a corresponds to each of a plurality of processes executed when the prediction model is constructed, that is, the pipeline, and each time the setting contents are determined in the step of sequentially determining the setting contents of the corresponding processes. Next, the step to be executed is selected. The setting content determination unit 152c determines the setting content of each step from the setting content candidates included in the setting information 141. At this time, the step selection unit 152a selects the next step for which the setting content has been determined according to the execution order indicated in the setting information 141. If none of the steps has been executed, the step selection unit 152a selects the step that has the earliest execution order.

  For example, as shown in FIG. 5, since the next step of the step “normalization method search” is “feature selection method search”, when the setting content of the step “normalization method search” is determined, the step selection unit 152a Then, “search for feature selection method” is selected as the next step.

  In addition, the steps “search for missing value complementing method”, “search for normalizing method”, and “search for feature selection method” in FIG. 5 respectively perform missing value compensation, normalization, and preprocessing of data for learning and analysis, respectively. This is a step of determining setting contents for feature selection. In addition, the setting content candidates for the steps “search for missing value complement method”, “search for normalization method”, and “search for feature selection method” are methods used for missing value complement, normalization, and feature selection, respectively.

  The accuracy calculation unit 152b performs processing for which the setting content has been determined among the plurality of processings by applying the determined setting content, and performs processing corresponding to the step selected by the step selection unit 152a. Prediction accuracy is calculated for each prediction model that is constructed when each candidate is applied.

  For example, when the step “feature selection method search” is selected by the step selection unit 152a, the steps “missing value complement method search” and “normalization method search” that are executed before the step “feature selection method search”. Since the setting contents of have been determined, the setting contents determined in the steps “Find missing value interpolation method search” and “Normalization method search” and the setting contents candidates in the step “Feature selection method search” were applied. A prediction model can be constructed. At this time, since there are four setting contents candidates for the step “feature selection method search”, when the setting contents of the steps “missing value complement method search” and “normalization method search” are determined as one, At least four prediction models can be constructed.

  Then, the accuracy calculation unit 152b calculates the prediction accuracy for each predictable model that can be constructed. At this time, a plurality of setting contents of the steps “search for missing value complementing method” and “normalization method searching” may be determined. For example, when two types of setting contents of the steps “search for missing value complementing method” and “normalized method searching” are determined, the number of predictable models that can be constructed is at least eight. Further, the accuracy calculation unit 152b can calculate the prediction accuracy by performing cross-validation using learning data divided into a predetermined number. Here, the cross-validation will be described with reference to FIG. FIG. 6 is a diagram for explaining the cross-validation according to the first embodiment.

  As shown in FIG. 6, first, the accuracy calculation unit 152b divides the sample data into four pieces of divided data 20a, 20b, 20c, and 20d. Then, as the first process, the accuracy calculation unit 152b uses the prediction model to cause the predictor to learn the divided data 20b, 20c, and 20d, and measures the accuracy of the learned predictor using the divided data 20a. .

  Similarly, in the second process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20c, and 20d, and measures the accuracy of the learned predictor using the divided data 20b. In the third process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20b, and 20d, and measures the accuracy of the learned predictor using the divided data 20c. In the fourth process, the accuracy calculation unit 152b causes the predictor to learn the divided data 20a, 20b, and 20c, and measures the accuracy of the learned predictor using the divided data 20d. Then, the accuracy calculation unit 152b sets the cross-validation accuracy, which is an average value of the accuracy measured in four processes, as the prediction accuracy. In addition, the division | segmentation number in cross-validation is not limited to 4, It can be made into arbitrary numbers.

  The setting content determination unit 152c compares the prediction accuracy calculated by the accuracy calculation unit 152b, and sets the setting content candidate having the highest prediction accuracy among the setting content candidates to the step selected by the step selection unit 152a. Determine the setting contents of the corresponding process.

  For example, as shown in FIG. 5, in step “normalization method search”, the accuracy calculation unit 152 b calculates the prediction accuracy of the prediction model corresponding to the setting content “maximum minimum” as 72%, and the setting content “standardization”. The prediction accuracy of the prediction model corresponding to the setting content “Z score” is calculated as 78%, the prediction accuracy of the prediction model corresponding to the setting content “Z score” is calculated as 72%, and the prediction accuracy of the prediction model corresponding to the setting content “no processing”. Was calculated as 70%. At this time, since the prediction model having the highest prediction accuracy in the step “normalization method search” is a prediction model corresponding to the setting content “standardization”, the setting content determination unit 152 c corresponds to the step “normalization method search”. The setting content of the setting item is determined as “standardized”. That is, the setting content determination unit 152c determines to standardize the method used in normalization, which is data preprocessing.

  As described above, the step selection unit 152a selects a step to be executed next to the step whose setting content is determined by the setting content determination unit 152c. For example, when the setting content in the step “normalization method search” is determined by the setting content determination unit 152c, the step selection unit 152a selects the step “feature selection method search”.

  Further, the search unit 152 performs parameter search. In the parameter search, the search unit 152 comprehensively calculates the prediction accuracy for each hyperparameter combination, and searches for the best parameter with the highest prediction accuracy.

The search unit 152 finally searches the pipeline for which the prediction accuracy of the prediction model satisfies a predetermined condition among the pipelines. For example, when the threshold value of the prediction accuracy is 70% and only the pipeline having the prediction accuracy equal to or higher than the threshold value is searched, the search unit 152 uses the pipelines P _{1 to} P ₃ as search results as shown in FIG. . FIG. 7 is a diagram for explaining a pipeline search according to the first embodiment.

  The hyperparameter search result by the search unit 152 can be expressed as a region as shown in FIG. FIG. 8 is a diagram for explaining a promising area according to the first embodiment. For example, the vertical axis in FIG. 8A represents the first hyperparameter, and the horizontal axis represents the second parameter. FIG. 8A shows that the denser the pattern, the higher the prediction accuracy.

  At this time, the search unit 152 can use not only a pipeline with a prediction accuracy of 70% or more but also a pipeline with a prediction accuracy of 70% and a value that is not statistically significant. . That is, the search unit 152 has a combination of setting contents such that the prediction accuracy of the prediction model based on sample data constructed when applied to a plurality of processes is 70% or more, and the prediction accuracy is 70% or more. A search is made for a combination of setting contents that does not have a statistically significant difference from a certain prediction accuracy. Note that 70% here is an example of the second threshold. An area corresponding to a combination of hyperparameters corresponding to the search result in this case is called a promising area.

  Here, (a) of FIG. 8 represents a promising region of prediction accuracy when sample data is used. FIG. 8B shows a promising region of prediction accuracy when using learning data from which sample data is extracted. As shown in FIG. 8, the two promising regions are similar. The promising region is used in processing by the selection unit 153.

  The selection unit 153 uses the combination of setting contents searched by the search unit 152 to set the prediction content of the prediction model based on the input data that is constructed when applied to a plurality of processes at a predetermined threshold value or more. Select a combination.

  That is, the search unit 152 evaluates the prediction accuracy of the prediction model using the sample data, whereas the selection unit 153 evaluates the prediction accuracy of the prediction model using the learning data. At this time, the selection unit 153 further narrows down the prediction accuracy for each pipeline searched by the search unit 152 using a threshold value.

Here, the selection unit 153 may simply select a pipeline in which each prediction accuracy is equal to or higher than the threshold, or may select a pipeline in which a value obtained by dividing each prediction accuracy by the maximum prediction accuracy is equal to or higher than the threshold. Good. For example, as shown in FIG. 9, when the prediction accuracy of pipelines P ₁ , P ₂ , and P ₃ is 85%, 70%, and 83%, the maximum prediction accuracy is 85%. FIG. 9 is a diagram for explaining selection of a pipeline according to the first embodiment.

In this case, selection section 153 compares the 85/85 and the threshold for the pipeline _{P 1} is compared with 83/85 and the threshold for the pipeline _{P 2,} the 70/85 and the threshold for the pipeline _{P 1} The pipeline can be selected using the comparison result.

  Further, the selection unit 153 fixes the setting contents regarding the preprocessing of the data of the selected combination, and performs a hyper parameter search. Here, if the hyperparameter having the highest prediction accuracy as a result of the search by the search unit 152 is the best parameter 210, the promising region can be expressed as shown in FIG. FIG. 10 is a diagram for explaining a promising area according to the first embodiment.

  The filled portion in FIG. 10 represents a promising area. In addition, the hatched portion in FIG. 10 is a region that is a target for hyperparameter search by the search unit 152 but is excluded from a target for hyperparameter search by the selection unit 153. In addition, the search part 152 can determine the presence or absence of a significant difference by the t test using the prediction accuracy in the case of the best parameter 210.

  Each time sample data is extracted by the extraction unit 151, the estimation unit 154 uses the input data based on the time required when a part of the predetermined processing is performed using the sample data. A time required for performing a predetermined process is estimated, and when the estimated time is less than a preset time limit, a predetermined ratio is increased. The process by the estimation unit 154 is performed at any timing during the execution of the process of the search unit 152, before the execution of the process of the search unit 152, or after the execution of the process of the search unit 152 and performed by the selection unit 153. It may be performed before it is performed. For example, the estimation unit 154 can estimate the processing time of the search unit 152 and the selection unit 153 when the prediction accuracy of the “average value” for the step “search for missing value complementing method” in FIG. 5 is calculated. .

First, it is assumed that the increase amount of the calculation amount when the number of data increases for each predictor is known. For example, it is assumed that the increase amount of the calculation amount of the predictor A ₁ is O (n ² ). Further, it is assumed that the sample rate is 20% and the time required for calculating the prediction accuracy of the “average value” for the step “search for missing value complementing method” is 100 seconds. As shown in FIG. 5, since there are three steps and there are four setting content candidates in each step, the estimation unit 154 calculates the processing time of the search unit 152 for the predictor A ₁ by the equation Calculate as in (1).
100 (seconds) × 3 (steps) × 4 (setting contents) = 1200 (seconds) (1)

Further, the selecting unit 153, 100/20 times the search section 152, that is, the 5 times the data is of interest, estimating section 154, the processing time of the selection unit 153 of the predictor _{A 1,} equation (2) Calculate as follows.
100 (seconds) x 5 ² (increase in calculation amount) x 3 (steps) x 1 (setting contents)
= 7500 (seconds) (2)

  Here, the subsequent processing differs depending on whether or not the processing time (total of (1) and (2)) calculated by the estimation unit 154 is less than a predetermined time limit. When the required time estimated by the estimation unit 154 is less than the predetermined time limit, the search unit 152 further performs a search using the sample data extracted by the extraction unit 151 at a rate higher than a predetermined rate. .

  On the other hand, when the required time estimated by the estimation unit 154 is equal to or longer than the time limit, the search unit 152 executes a search by excluding a prediction algorithm having a similar prediction method. At this time, the search unit 152 excludes the prediction algorithm until the estimated time becomes less than the time limit.

  Specifically, first, the predictor information 142 stores a prediction algorithm, that is, a category for each predictor. Then, when the required time estimated by the estimation unit 154 is equal to or longer than the time limit, the search unit 152 refers to the predictor information 142 and excludes any of a plurality of predictors having the same category. For example, the search unit 152 excludes predictors in descending order of execution time based on information on execution time set in advance.

For example, assume that the predictor A ₁ in FIG. 3 is Random Forest, the predictor A ₂ is Extra Random Trees, the predictor A ₃ is Logistic Regression, and A ₄ is Linear SVM. Then, the predictor information 142, "decision tree category" is set as the category of A ₁ and A _2, and the "linear predictor category" is set as the category of A ₃ and A _4. Also, it is assumed that the predictor information 142 is set so that the order of the length of execution time is A ₂ , A ₁ , A ₃ , A ₄ .

At this time, if the required time estimated by the estimation unit 154 is equal to or longer than the time limit, the search unit 152 first sets A ₂ with a plurality of predictors in the same category and has the longest execution time. Is excluded. Here, if be excluded A ₁ is estimated to be required time is the time limit or more, the search unit 152, among the remaining predictor, a plurality of predictor is set to the same category, and, most execution time excludes a long a _3. The search unit 152 repeats exclusion of the predictors until there is no category in which a plurality of predictors are set, or until it is not estimated that the required time is equal to or longer than the time limit.

When a combination of a plurality of setting contents is selected by the selection unit 153, the combining unit 155 combines a plurality of prediction models corresponding to each of the plurality of setting content combinations by using a predetermined ensemble method. For example, as illustrated in FIG. 11, when two pipelines are selected by the selection unit 153, two prediction models Model A ₁ and Model A ₃ can be constructed. FIG. 11 is a diagram for explaining the synthesis of the prediction model according to the first embodiment. At this time, the synthesizing unit 155 synthesizes the model A ₁ and the model A ₃ with the weight w ₁ and the weight w ₂ , respectively, by a predetermined ensemble method.

Furthermore, as illustrated in FIG. 12, the synthesis unit 155 sets the prediction model having the highest generalization performance among Model A ₁ , Model A ₃ , and the synthesized Ensemble Model as the best model. FIG. 12 is a diagram for explaining the synthesis of the prediction model according to the first embodiment.

  The verification unit 156 verifies the prediction model constructed based on the pipeline selected by the selection unit 153. When a pipeline is selected by the selection unit 153, the verification unit 156 causes the predictor to learn learning data based on the determined pipeline, and builds a prediction model. Then, the verification unit 156 measures the prediction accuracy of the constructed prediction model as test accuracy using test data different from the learning data. For example, the analyzer 10 may use the test accuracy measured here as the final output. Further, by performing verification using test data different from the learning data, it is possible to check the overlearned state and the unlearned state.

[Process of First Embodiment]
A processing flow of the analyzer 10 according to the first embodiment will be described with reference to FIGS. 13 and 14 are flowcharts showing the flow of processing of the analyzer according to the first embodiment. As shown in FIG. 13, first, the analysis apparatus 10 reads learning data (step S <b> 101). Next, the extraction unit 151 extracts sample data having a predetermined sample size from the learning data (step S102).

  Next, when there is an unselected predictor (step S103, Yes), the search unit 152 refers to the predictor information 142 and selects the next predictor (step S104). Next, the search unit 152 determines a pipeline of the selected predictor using the read learning data (step S105). The search unit 152 may determine a plurality of pipelines.

  Next, a process (step S105 in FIG. 13) in which the search unit 152 determines a pipeline will be described in detail with reference to FIG. As illustrated in FIG. 14, when there is an unselected step (Yes in Step S201), the step selection unit 152a refers to the setting information 141 and selects the next step (Step S202). The next step is the step with the earliest execution order among the unselected steps. In the present embodiment, each step corresponds to each data pre-processing. On the other hand, when there is no unselected step (step S201, No), the analysis apparatus 10 searches for a hyper parameter (step S207).

  When there is an unselected setting content among the setting content candidates of the step selected by the step selection unit 152a (step S203, Yes), the accuracy calculation unit 152b selects the next setting content (step S204). On the other hand, when there is no unselected setting content (step S203, No), the setting content determination unit 152c determines the setting content with the highest prediction accuracy calculated by the accuracy calculation unit 152b by the step selection unit 152a. The setting contents are determined (step S206).

  When the setting content is selected, the accuracy calculation unit 152b calculates the prediction accuracy of the prediction model constructed based on the pipeline to which the selected setting content is applied (step S205). At this time, the accuracy calculator 152b can calculate the prediction accuracy by cross-validation using the learning data divided into a predetermined number. Then, the accuracy calculation unit 152b repeats the processes of steps S203 to S205 until there is no unselected setting content.

  Returning to FIG. 13, when there is no unselected predictor (step S103, No), the estimation unit 154 estimates the time of the entire process (step S106). When the estimated time is less than the time limit (step S107, Yes), the estimation unit 154 increases the sample rate (step S108). Then, the extraction unit 151 extracts sample data again at the increased sample rate (step S102). Thereafter, the analysis apparatus 10 repeats the process until the time estimated by the estimation unit 154 is less than the time limit.

  On the other hand, when the time estimated by the estimation unit 154 is not less than the time limit (No at Step S107), the search unit 152 reduces the prediction algorithm (Step S109). Then, the selection unit 153 selects a pipeline whose prediction accuracy is greater than or equal to the threshold value from the pipeline determined by the search unit 152, and further performs a hyperparameter search for the selected pipeline (step S110). Then, the verification unit 156 constructs a prediction model based on the selected pipeline (step S111), and verifies the constructed prediction model using test data (step S112).

[Effect of the first embodiment]
When data is input, the extraction unit 151 extracts a predetermined ratio of data from the input data as sample data. When the predetermined ratio increases, the extraction unit 151 increases the predetermined ratio of the input data. The data is further extracted as sample data. Each time sample data is extracted by the extraction unit 151, the estimation unit 154 uses the input data based on the time required when a part of the predetermined processing is performed using the sample data. A time required for performing a predetermined process is estimated, and when the estimated time is less than a preset time limit, a predetermined ratio is increased. In addition, the search unit 152 is a sample that is constructed when applied to a plurality of processes from candidate combinations of setting contents applied to a plurality of processes executed when a prediction model based on predetermined data is constructed. A combination of setting contents is searched for such that the prediction accuracy of the prediction model based on the data satisfies a predetermined condition. In addition, the selection unit 153 has a prediction accuracy of a prediction model based on input data constructed when applied to a plurality of processes based on a combination of setting contents searched by the search unit 152 becomes a predetermined threshold value or more. Select a combination of settings. In this way, it is possible to roughly determine a pipeline with high prediction accuracy using sample data, and select the final pipeline using all data, thereby reducing the time required for pipeline search. A prediction model with high prediction accuracy can be constructed while shortening.

  FIG. 15 shows the prediction accuracy of the prediction model constructed by analyzing the same execution time when using auto-skearn and when using the present embodiment. FIG. 15 is a diagram for explaining the effect of the first embodiment. As shown in FIG. 15, the prediction accuracy is higher in the present embodiment with 8 types of data out of 12 types. The remaining four types had almost the same expected accuracy.

  FIG. 16 shows the execution time and prediction accuracy when the processing of the search unit 152 and the selection unit 153 is performed without performing sample extraction (automated rule), and in the case of the present embodiment. FIG. 16 is a diagram for explaining the effect of the first embodiment. As shown in FIG. 16, in this embodiment, it was possible to reduce the processing time by about 70% without substantially reducing the prediction accuracy.

  When the required time estimated by the estimation unit 154 is equal to or longer than the time limit, the search unit 152 performs a search by excluding a prediction algorithm having a similar prediction method. Thereby, when there is no time, the processing time can be further shortened, and when there is time, a prediction model with higher prediction accuracy can be constructed.

  The search unit 152 includes a combination of setting contents such that the prediction accuracy of the prediction model based on the sample data constructed when applied to a plurality of processes is equal to or higher than the second threshold, and the prediction accuracy and the second threshold. A combination of setting contents that does not have a statistically significant difference from the above prediction accuracy is searched. Thereby, it becomes possible to search for the optimum hyperparameter in a short time.

  When a combination of a plurality of setting contents is selected by the selection unit 153, the combining unit 155 combines a plurality of prediction models corresponding to each of the plurality of setting content combinations by using a predetermined ensemble method. Thereby, compared with the case where each pipeline is selected independently, a prediction model with higher prediction accuracy can be constructed.

[Second Embodiment]
In the first embodiment, the case where the estimation of the processing time by the estimation unit 154 is performed after the pipeline is determined by the search unit 152 has been described. Here, as described above, the processing by the estimation unit 154 may be performed at an arbitrary timing during the processing of the search unit 152 or before the processing of the search unit 152 is executed. In the second embodiment, the estimation of the processing time is performed by the estimation unit 154 before the processing related to pipeline determination by the search unit 152 is started.

[Process of Second Embodiment]
A processing flow of the analyzer 10 according to the second embodiment will be described with reference to FIG. FIG. 17 is a flowchart showing a flow of processing of the analyzer according to the second embodiment. As shown in FIG. 17, first, the analysis apparatus 10 reads learning data (step S301). Next, the extraction unit 151 extracts sample data having a predetermined sample size from the learning data (step S302).

  Next, when there is an unselected predictor (step S303, Yes), the estimation unit 154 refers to the predictor information 142 and selects the next predictor (step S304). Next, the estimation unit 154 measures the execution time with the sample data, and predicts the execution time with the learning data (step S305).

  Here, the estimation unit 154 actually performs cross-validation using the sample data, and measures the time required at that time. Specifically, the estimation unit 154 performs processing similar to the processing by the accuracy calculation unit 152b as shown in FIG. However, since the estimation unit 154 is intended to measure the time required for the process, the prediction accuracy need not be calculated.

  In addition, the accuracy calculation unit 152b performs preprocessing on the sample data, and then causes the predictor to learn the sample data. In contrast, the estimation unit 154 directly causes the learning device to learn without performing preprocessing on the sample data. In addition, the estimation unit 154 predicts the execution time in the learning data using the above-described equations (1) and (2).

  The estimation unit 154 sequentially selects predictors and executes Step S305. When there is no unselected predictor (No at Step S303), the estimation unit 154 estimates the time of the entire process (Step S306). When the estimated time is less than the time limit (step S307, Yes), the estimation unit 154 increases the sample rate (step S308). Then, the extraction unit 151 extracts sample data again at the increased sample rate (step S302). Thereafter, the analysis apparatus 10 repeats the process until the time estimated by the estimation unit 154 is less than the time limit.

  On the other hand, when the time estimated by the estimation unit 154 is not less than the time limit (No at Step S307), the search unit 152 reduces the prediction algorithm (Step S309). Next, when there is an unselected predictor (step S310, Yes), the search unit 152 refers to the predictor information 142 and selects the next predictor (step S311). Next, the search unit 152 determines the pipeline of the selected predictor using the read learning data (step S312). The search unit 152 may determine a plurality of pipelines. In addition, the process in which the search part 152 determines a pipeline is the same as the process shown in FIG.

  When there is no unselected predictor (No in step S310), the selection unit 153 selects a pipeline whose prediction accuracy is equal to or greater than a threshold from the pipeline determined by the search unit 152, and further, for the selected pipeline The hyperparameter search is performed (step S313). Then, the verification unit 156 constructs a prediction model based on the selected pipeline (step S314), and verifies the constructed prediction model using test data (step S315).

[Other Embodiments]
When the feature amount of the input data has a predetermined data characteristic, the search unit 152 performs a search from the setting content combination candidates associated in advance with the predetermined data characteristic among the setting content combination candidates. May be. For example, in the case of data such as text data in which the proportion of zero in the data is large, that is, when the data is sparse, sufficient accuracy is often obtained even with a linear predictor. Therefore, when the learning data or the sample data is sparse, the search unit 152 excludes a non-linear predictor from the predictors. For example, the search unit 152, a sparse feature quantity ratio of zero is equal to or greater than the threshold r _f, when the ratio of total is the threshold value r _a more, determines that the data is sparse. Threshold _{r f} and _{r a} can be, for example, 0.9.

[System configuration, etc.]
Each component of each illustrated device is functionally conceptual and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or a part of the distribution / integration is functionally or physically distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. Furthermore, all or a part of each processing function performed in each device may be realized by a CPU and a program that is analyzed and executed by the CPU, or may be realized as hardware by wired logic.

  Also, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed. All or a part can be automatically performed by a known method. In addition, the processing procedure, control procedure, specific name, and information including various data and parameters shown in the above-described document and drawings can be arbitrarily changed unless otherwise specified.

[program]
As one embodiment, the analysis apparatus 10 can be implemented by installing an analysis program for executing the above analysis as package software or online software on a desired computer. For example, the information processing apparatus can be caused to function as the analysis apparatus 10 by causing the information processing apparatus to execute the above analysis program. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, the information processing apparatus includes mobile communication terminals such as smartphones, mobile phones and PHS (Personal Handyphone System), and slate terminals such as PDA (Personal Digital Assistant).

  The analysis apparatus 10 can also be implemented as an analysis server apparatus that uses a terminal device used by a user as a client and provides the client with the above-described analysis-related services. For example, the analysis server device is implemented as a server device that provides an analysis service that receives learning data as an input and outputs a pipeline or a prediction model. In this case, the analysis server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the above analysis by outsourcing.

  FIG. 18 is a diagram illustrating an example of a computer that executes an analysis program. The computer 1000 includes a memory 1010 and a CPU 1020, for example. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to the display 1130, for example.

  The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the analysis apparatus 10 is implemented as a program module 1093 in which a code executable by a computer is described. The program module 1093 is stored in the hard disk drive 1090, for example. For example, a program module 1093 for executing processing similar to the functional configuration in the analysis apparatus 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD.

  The setting data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 and executes them as necessary.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read out by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (LAN, WAN (Wide Area Network), etc.). Then, the program module 1093 and the program data 1094 may be read by the CPU 1020 from another computer via the network interface 1070.

DESCRIPTION OF SYMBOLS 10 Analysis apparatus 11 Input part 12 Output part 13 Communication control part 14 Storage part 15 Control part 141 Setting information 142 Predictor information 151 Extraction part 152 Search part 152a Step selection part 152b Accuracy calculation part 152c Setting content determination part 153 Selection part 154 Estimation 155 Synthesis unit 156 Verification unit

Claims (7)

When data is input, a predetermined ratio of data is extracted as sample data from the input data, and when the predetermined ratio is increased, the increased predetermined ratio of data is input from the input data. An extractor for further extracting as sample data;
Each time the sample data is extracted by the extraction unit, the predetermined data is input using the input data based on the time required for performing a part of the predetermined processing using the sample data. Estimating the time required when performing the process, and when the estimated time is less than a preset time limit, an estimation unit that increases the predetermined ratio;
A prediction model based on the sample data constructed when applied to the plurality of processes from a combination of setting contents applied to the plurality of processes executed when the prediction model based on the predetermined data is constructed A search unit for searching for a combination of setting contents such that the prediction accuracy of satisfies a predetermined condition;
From a combination of setting contents searched by the search unit, a combination of setting contents that the prediction accuracy of the prediction model based on the input data constructed when applied to the plurality of processes is a predetermined threshold or more. A selection section to select;
An analysis apparatus comprising:   When the feature amount of the input data has a predetermined data characteristic, the search unit selects a combination of setting contents previously associated with the predetermined data characteristic from among the combinations of setting contents. The analysis apparatus according to claim 1, wherein a search is performed.   3. The search unit according to claim 1, wherein when the required time estimated by the estimation unit is equal to or longer than the time limit, the search unit executes a search by excluding a prediction algorithm having a similar prediction method. Analysis equipment.   When a combination of a plurality of setting contents is selected by the selection unit, the apparatus further includes a synthesizing unit that synthesizes a plurality of prediction models corresponding to each of the plurality of setting contents combinations using a predetermined ensemble technique. The analyzer according to claim 1, wherein the analyzer is characterized by the following. The search unit
In the step of sequentially determining the setting contents of the corresponding process corresponding to each of the plurality of processes executed when constructing the prediction model, each time the setting contents are determined, the step to be executed next is selected. ,
Among the plurality of processes, a process for which setting contents have been determined is performed by applying the determined setting contents, and a process corresponding to the next executed step is applied to each of the setting contents candidates. Calculate the prediction accuracy for each of the prediction models that will be built,
The calculated prediction accuracy is compared, and the setting content candidate having the highest prediction accuracy among the setting content candidates is determined as the setting content of the process corresponding to the next executed step. The analyzer according to claim 1. An analysis method executed by an analyzer,
An extraction step of extracting a predetermined percentage of data from the input data as sample data;
A prediction model based on the sample data constructed when applied to the plurality of processes from a combination of setting contents applied to the plurality of processes executed when the prediction model based on the predetermined data is constructed A search step for searching for a combination of setting contents such that the prediction accuracy of
A combination of setting contents in which the prediction accuracy of the prediction model based on the input data constructed when applied to the plurality of processes is greater than or equal to a predetermined threshold from the combination of setting contents searched in the search step A selection process to select;
The analysis method characterized by including.   An analysis program for causing a computer to function as the analyzer according to any one of claims 1 to 5.

Images