JP5766753B2 - System parameter learning apparatus, information processing apparatus, method, and program - Google Patents

Description

  The present invention relates to a system parameter learning apparatus, information processing apparatus, method, and program, and more particularly, to a system parameter learning apparatus, information processing apparatus, method, and program for learning system parameters.

  As shown in FIG. 11, identification problems in information processing such as speech recognition, machine translation, character recognition, object recognition, DNA structure prediction, etc. predict output when input is given as shown in FIG. It can be regarded as a system.

  These systems can generally be divided into an execution phase and a construction phase. The construction phase refers to the work of designing a system in advance by hand and determining system parameters and the like. In the execution phase, the input is processed based on the design defined in the construction phase, and the output is determined depending on the system parameters.

  In the construction phase, the system can be constructed in various ways. For example, a method is conceivable in which a conversion rule is described manually, an input is converted into an output in accordance with the rule, and the output is output.

  However, manual preparation of conversion rules is very expensive to maintain completeness and consistency. Therefore, as shown in FIG. 13, a machine learning method for automatically constructing a system from data is used. In recent years, the method of automatically constructing a system has been mainstream.

  In the construction phase, first, an input pair of the target system and a corresponding output pair are prepared. This is generally called correct answer data or teacher data. Teacher data can be said to be data representing what kind of output should be made when input in teacher data is input to the system. Next, a system is constructed using the teacher data. The necessary requirement is that the system can perform correct output with respect to the input in the teacher data. Therefore, the construction phase based on machine learning results in learning a set of system parameters that can correctly discriminate the teacher data using the teacher data.

The above processing is expressed mathematically as follows.
First, the execution phase is shown. Let x ^ represent one input and let X be the set of all possible inputs x ^ that the system can accept. Note that “＾” attached to a symbol indicates that the symbol is a matrix, a multidimensional array, or a vector. Similarly, let y denote a single output, and let Y be the set of all possible outputs y that the system allows. Also, let Y (x) be the set of all outputs y ^ that can be taken when x ^ is given. Therefore, the relationship x ^ ∈X, y ^ ∈Y (x ^) ⊆Y holds.

Next, let ＾ be a vector representation of a set of system parameters. Here, w _d is the d-th element of the vector w ^, and at the same time, is the d-th system parameter. That is, w ^ = (w ₁ ,..., W _N ) and d = {1,. . . , N} holds. However, the number of system parameters is N, and w ^ is an N-dimensional vector. At this time, a system that returns an output y ^ when an input x ^ is given can be expressed as follows.

  However, Φ (x ^, y ^; w ^) is a function for determining a score when converting from x ^ to y ^, and is simply referred to as a score function here. That is, among all the outputs y ^ that may be obtained when x ^ is given, y ^ having the highest conversion score is adopted as the output. In other words, w ^ is a system parameter that controls which output is selected, and can be said to be a factor that determines the performance of the entire system. Therefore, how to obtain the system parameter w ^ with high accuracy is the greatest requirement in the construction phase. The term “accurate” as used herein means obtaining w ^ that can perform as many correct outputs as possible for every input. Note that “′” added in front of the symbol indicates that the symbol is an estimated value.

Next, the construction phase will be described. In fact, it is very difficult to find the best parameter w for every possible input. That is due to the fact that it is virtually difficult to enumerate all possible inputs. Therefore, in the field of pattern recognition, w ^ is determined based on actual data. First, the teacher data is represented by D. The teacher data is composed of a set of pairs of input x ^ and output y ^. That is, D = {(x ^ _i , y ^ _i )} ^N _{i = 1} . At this time, x _i is the i-th input data in the teacher data, and y _i is the output data corresponding to the i-th input.
Learning system parameters can be obtained by solving the following optimization problem.

  At this time, L (Φ (), w ^, D) is called a risk function or a loss function, and is a function that returns a value indicating how much correct output can be obtained with respect to the input in the teacher data. When the current system parameter w is actually used to make a discrimination using the above equation (1), if more mistakes are made, a function having a larger value is used. Ω (w ^) is generally called a regularization term, and in a situation where there is only a limited number of teacher data, system parameters are over-adapted to teacher data so that it can be correctly identified even for data that does not appear in the teacher data. This is a term that gives a penalty. For example, it is often used that a parameter is restricted so as not to take an extremely large value by imposing a constraint such that the L2 norm of the parameter becomes as small as possible. Finally, 'w ^ obtained by the above equation (2) can be said to be a set of parameters that can best identify the teacher data under the condition of the regularization term.

  The above is a mathematical definition of the execution phase and the construction phase of the information processing system targeted by the present invention.

Acquisition of system parameters based on the above equation (2) is called supervised learning in pattern recognition. At this time, the learned system parameter 'w ^ is represented by a real value. Therefore, at the end of the construction phase, the value of “w” is written to a file or the like, and in the execution phase, the written file is read and a score function is calculated to obtain an output.
In other words, the larger the number of parameters, the larger the file size required to hold that information.

  The file size is the same as the memory occupancy during execution. Memory occupancy can be a significant problem in computing environments with limited resources, such as portable terminals. In addition, even when executing on a general computer, it is desirable that the memory occupancy is as small as possible from the viewpoint of not affecting other programs as much as possible when simultaneously executing multiple on a recent multi-core computer. . In order to solve this problem, a method has been used so far in which the parameter is made zero as much as possible by using the L1 regularization term (see, for example, Non-Patent Documents 1 to 3). This method is based on a methodology in which, when a parameter becomes zero, items related to the parameter can be deleted, so that the model size can be reduced.

Galen Andrew and Jianfeng Gao, `` Scalable Training of L1-regularized Log-linear Models '', In Zoubin Ghahra-mani, editor, Proceedings of the 24th Annual International Conference on Machine Learning (ICML 2007) Omnipress, 2007, p.33-40 John Duchi and Yoram Singer, “Efficient Online and Batch Learning Using Forward Backward Splitting”, Journal of Machine Learning Research, 10 (2009), p.2899-2934 Lin Xiao, “Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization”, Journal of Machine Learning Research, 11 (2010), p.2543-2596

  However, in the conventional method, learning is performed so that the parameter value becomes zero as much as possible. However, there is a problem that the values other than the zero value remain as before. Here, setting the value to zero is a trade-off between the effect of setting the value to zero and the system performance because there is a risk of reducing the accuracy of the system unless the item related to the system parameter is truly meaningless. It becomes a relationship. In other words, it is desirable to reduce the model size by increasing the number of zeros as much as possible. However, if the number of zeros is excessively large, there is a possibility that the system performance will be insufficient.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a system parameter learning apparatus, method, and program capable of reducing the storage capacity for storing system parameters.

  In order to achieve the above object, a system parameter learning device according to the present invention performs predetermined information processing on input data using a score function for determining a score of output data with respect to input data, and outputs data. A system parameter learning device for learning a plurality of system parameters set in an information processing system that outputs a plurality of input data and a pair of correct output data for each of the plurality of input data Based on a teacher data input unit that inputs certain teacher data, the teacher data input by the teacher data input unit, and the score function, the values of each of the plurality of system parameters are determined in advance. The plurality of system parameters satisfying the constraints included in the set of discrete values and optimized It is configured to include a learning unit for learning the.

  A system parameter learning method according to the present invention includes a teacher data input unit and a learning unit, and performs predetermined information processing on input data using a score function for determining a score of output data with respect to input data. A system parameter learning method in a system parameter learning apparatus for learning a plurality of system parameters, set in an information processing system that performs output data output, wherein each of a plurality of input data is Based on the step of inputting teacher data that is a pair with each of a plurality of correct output data for a plurality of input data, the teacher data input by the teacher data input unit by the learning unit, and the score function And each of the plurality of system parameters has a predetermined finite value. Satisfy the constraints in the set of discrete values, and, and a step of learning a plurality of system parameters optimized.

The learning unit according to the present invention can learn the optimized N system parameters' w ^ according to the following equation.

However, L (Φ (), w ^, D) , the output data obtained when performing the predetermined processing using the system parameter w ^ with respect to the input data x _i of the training data D is, This is a risk function that returns a value indicating how wrong the correct output data y _i of the teacher data D is, Ω (w ^) is a regularization term, and S ^N is the N system parameters. A Cartesian product set of the finite number of discrete values for each.

In addition, the learning unit according to the present invention can learn the optimized N system parameters' w ^ according to the following equation.

Where α ^ is a Lagrange undetermined multiplier, ρ is a predetermined tuning parameter, and u ^ is an auxiliary parameter.

  Further, based on the plurality of system parameters learned by the learning unit according to the present invention, a group is configured with system parameters having the same value, an index number is assigned to each group, and for each group, the group Is further converted to a system parameter of the index number assigned to the group, and further includes a duplicate parameter compression unit for storing the system parameter of the index number.

The plurality of system parameters according to the present invention are weights of a plurality of feature extraction functions f _d (x ^, y ^) for the input data x ^ and output data y ^, For each group, each of the system parameters constituting the group is converted to the system parameter of the index number assigned to the group, and the system parameter of the index number is stored, as shown in the following equation: The score function can be defined and stored.


Here, Φ (x ^, y ^; w ^) is a score function, and _vk is a system parameter of the index number k assigned to the group.

  The information processing apparatus according to the present invention, based on the input unit that receives input data, the score function, and the system parameter of the index number of each group stored by the system parameter learning apparatus, On the other hand, an information processing unit that performs the predetermined information processing and outputs output data is included.

  An information processing method according to the present invention is an information processing method in an information processing apparatus including an input unit and an information processing unit, the step of receiving input data by the input unit, and the score function by the information processing unit. And outputting the output data by performing the predetermined information processing on the input data based on the system parameter of the index number of each group stored by the system parameter learning device. Including.

  The first program according to the present invention is a program for causing a computer to function as each part of the system parameter learning device.

  The second program according to the present invention is a program for causing a computer to function as each unit of the information processing apparatus.

  As described above, according to the system parameter learning apparatus, method, and program of the present invention, based on the teacher data, each value of the plurality of system parameters is converted into a set of a predetermined finite number of discrete values. By learning a plurality of optimized system parameters that satisfy the included constraints, the storage capacity for storing the system parameters can be reduced.

  In addition, according to the information processing apparatus, method, and program of the present invention, predetermined information processing is performed on input data based on the system parameter of each group index number stored by the system parameter learning apparatus. By outputting the output data, it is possible to reduce the resources required at the time of execution.

It is a figure which shows the example of the problem to which embodiment of this invention is applied. It is a conceptual diagram for demonstrating an example of the definition of a feature and a system. It is the schematic which shows the structure of the system parameter learning apparatus which concerns on embodiment of this invention. It is a 1st conceptual diagram for demonstrating an example of a feature extraction function. It is a 2nd conceptual diagram for demonstrating an example of a feature extraction function. It is a conceptual diagram for demonstrating the learning process of a system parameter value based on teacher data. It is the schematic which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the system parameter learning process routine in the system parameter learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the content of the information processing routine in the information processing apparatus which concerns on embodiment of this invention. It is a figure which shows the experimental result in the case of using this Embodiment. It is the 1st explanatory view for explaining an outline of conventional technology. It is the 2nd explanatory view for explaining an outline of conventional technology. It is the 3rd explanatory view for explaining an outline of conventional technology.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Overview>
The embodiment according to the present invention constructs an information processing system that selects output data corresponding to input data based on a predetermined system operation definition for certain input data and returns the output data. The present invention relates to a technical field, and belongs to a technology field in which pre-designed system parameters are automatically acquired by a computer, generally called a pattern recognition or machine learning.

  Examples of the target information processing system include an information processing system that performs speech recognition, machine translation, character recognition, object recognition, and DNA structure prediction.

  In the embodiment according to the present invention, the necessary resource amount (file size, memory exclusive amount, etc.) is essentially based on the general idea that it is better to be smaller in any calculation environment. Tackle the challenge of compressing the model size.

  In the embodiment according to the present invention, unlike the prior art, if a plurality of system parameters have the same value after system parameter learning, the amount of information that must be actually retained using this duplicate information The principle that it is possible to reduce is used.

  For example, when the number of system parameters is 5, the set of system parameters finally obtained by the above equation (2) is' w ^ = (0.3, 0.3, -0.6, -0.6, 1.0). Then, in 'w ^, 0.3 and -0.6 appear twice, so the first and second system parameters are combined, and the third and fourth system parameters are combined. , 'W ^ = (0.3, -0.6, 1.0), the same result can be obtained if there is at least three pieces of information. Thus, it can be seen that the more overlapping values, the smaller the amount of information about equivalent information can be retained.

  That is, in the present embodiment, when learning the system parameter of the above formula (2), a predetermined constraint is added so that each value of the plurality of system parameters takes the same value, and the system parameter By performing learning, each value of the resulting system parameter is made to overlap.

  Then, the system parameters having the same value are combined into one system parameter obtained, thereby reducing the amount of information to be retained. Finally, the information processing system is actually executed using the reduced system parameters.

  In summary, the present embodiment is roughly composed of the following three processes.

1. As a first process, a system parameter acquisition process in which the system parameter value finally obtained at the time of normal system parameter learning becomes the same. As a second process, a system parameter compression process for minimizing the amount of information that should be held by holding duplicate values of the obtained system parameters together 3. As a third process, a process for operating the information processing system in the execution phase using the system parameters compressed to the minimum

More specifically, in the first process, the system parameter learning problem of the above equation (2) is replaced with the following problem.
In this embodiment, the optimized N system parameters' w ^ are learned according to the following equation (3).

However, L (Φ (), w ^, D) is output data 'y _i obtained when predetermined information processing is performed on the input data x _i of the teacher data D using the system parameter w ^. , Is a risk function that returns a value indicating how wrong the correct answer data y _i is, Ω (w ^) is a regularization term, and S ^N is a finite number for each of the N system parameters. Is a Cartesian product set of discrete values.

The difference from the above equation (2) is simply that the constraint term w ^ εS ^N is increased. This restriction means that the solution is allowed only when the system parameter w is an element of an arbitrary discrete set ^SN . That is, the optimization problem equivalent to the above equation (2) is solved, but the solution needs to satisfy the constraints. This constraint is constituted by a Cartesian product set ^SN of a set S composed of a finite number of discrete values in order to design the system parameter values to be duplicated as much as possible. However, for efficient calculation, the set S of discrete values must always include zero and must always include values that are symmetric about zero (for example, 1 and -1, 5 and -5). And

As a second process, system parameters having the same value are combined into one. This is a post-processing position of the construction phase. For example, assume that w _i = w _j = w _k . That is, when the i, j, k-th system parameters (elements of the system parameter vector) have the same value, this corresponds to a process of deleting w _i , w _j , w _k and newly setting them to w _l . However, the newly added w _l is the same value as w _i and the like, and the indexes i, j, and k are equivalent to assuming that the index l is newly reassigned. In this way, even if the same value is held, it is redundant information, so that the same information can be obtained by referring to the index of the original value that points to the newly allocated index. By doing so, it is possible to obtain a set of system parameters, which are system parameters in the same format as before, but with a reduced amount of duplicate system parameter values.

  In the third process, the system is actually operated using the obtained system parameters. In this process, the system parameter of the same format as in the conventional system is obtained in the second process, so that the process of the above formula (1) equivalent to the conventional process can be used.

  In the following, the problem of natural language processing specific expression extraction and dependency analysis as shown in FIG. 1 will be described assuming that the embodiment of the present invention is applied. These problems are called structure prediction problems, and can be regarded as problems that receive as an input what has been converted to a graph structure or the like and output what is also represented by the graph structure.

  In the construction and execution phase described below, as shown in FIG. 2, first, the definition of the features that characterize each input and output is given manually. Here, since the feature is assumed to be defined as a function, it is equivalent to defining a set of feature extraction functions. Further, as shown in FIG. 2, the system operation definition such as what kind of output data is output when given input data is given manually. This is automatically determined according to the definition of the problem to be solved, here, the specific expression extraction and dependency analysis.

<System configuration of system parameter learning device>
A system parameter learning apparatus 100 according to an embodiment of the present invention is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a system parameter learning processing routine described later, and is functionally Is configured as follows. As shown in FIG. 3, the system parameter learning device 100 includes a teacher data input unit 1, a calculation unit 2, and a system parameter storage unit 4.

  The teacher data input unit 1 receives input of teacher data. Here, as shown in FIG. 1, the teacher data is a plurality of pairs of predetermined input data and correct output data for the input data. The teacher data input unit 1 accepts input of a parameter ρ used in the learning unit 3 described later. The parameter ρ is given in advance by hand.

  The calculation unit 2 includes a teacher database 20, a learning unit 3, and a duplicate parameter compression unit 40.

  Teacher data received by the teacher data input unit 1 is stored in the teacher database 20.

  The learning unit 3 learns a plurality of system parameters based on teacher data stored in the teacher database 20 and a predetermined score function. Specifically, the learning unit 3 satisfies, based on the teacher data and the score function, each value of the plurality of system parameters satisfies a constraint included in a predetermined finite set of discrete values, and Learn multiple optimized system parameters. The learning unit 3 includes an initialization unit 30, a system parameter update unit 32, an auxiliary parameter update unit 34, an undetermined multiplier update unit 36, and a convergence determination unit 38.

The learning algorithm in the learning unit 3 learns system parameters w ^ = (w ₁ , w ₂ ,... W _N ) that minimize the objective function. Specifically, the correct answer data and the supervised learning algorithm are determined manually and given as the setting of the construction phase. Hereinafter, an objective function for learning the system parameter w will be described first.

Here, the feature extraction function f for extracting features is a function defined by a combination of input data x ^ and output data y ^. Each feature extraction function is a function of the form f _d (x ^, y ^) and is a function that returns an arbitrary real value. Here, like w ^, a set of feature extraction functions is represented by a vector notation f ^ (x ^, y ^). At this time, f _d (x ^, y ^) represents the d-th element of f ^ (x ^, y ^). Examples of features extracted by the feature extraction function are shown in FIG. 4 and FIG.

  If the number of feature extraction functions is designed to be the number N of system parameters, the score function () can be defined as a linear function as shown in the following equation (4).

In other words, the system parameter w ^ _d is the weight of the feature extraction function f _d , and if the value is large, the feature extraction function f _d more influences the selection of the output data of the system. This corresponds to weighting such that data y is not selected more. In other words, the system parameter here is a value that determines the weight of the feature. The learning unit 3 learns system parameters using a supervised learning algorithm as shown in FIG.

  The objective function is expressed by the above equation (3). However, the above equation (3) is basically a discrete optimization problem because the solution has discrete constraints, and it is very difficult to obtain a solution strictly. Become a system. However, methods based on dual decomposition (eg, Reference 1 (Stephen Boyd, Neal Parikh, Eric Chu, BorjaPeleato, and Jonathan Eckstein, “Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers”, 2011, Foundations and Trends in (See Machine Learning) to obtain an efficient solution. First, the constraint of the above equation (3) is decomposed based on dual decomposition.

  Here, u ^ is an auxiliary parameter. This means that the conventional optimization problem and the constraint are separated by using the equality constraint w ^ = u ^. Next, the constraint is substituted into the objective function by using the extended Lagrangian relaxation of Reference Document 1 above.

  α ^ is a Lagrange multiplier. According to the above reference 1, the objective function shown in the above equation (6) is optimized by the processing in the initialization unit 30, the system parameter update unit 32, the auxiliary parameter update unit 34, the undetermined multiplier update unit 36, and the convergence determination unit 38. In this embodiment, system parameters with overlapping values are obtained.

  As the risk function, basically any function may be used, but here, the risk function is limited to a convex function. In the present embodiment, a hinge loss function shown in the following equation (7) is used as the risk function.

  Here, E (y ^, 'y ^) is a function indicating how much y ^ differs from' y ^. The larger the difference between y ^ and 'y ^, the larger E (y ^,' y ^) is greater than 0. If y ^ and 'y ^ are the same, E (y ^,' y ^) Becomes 0.

  Also, like the risk function, the regularization term is also a convex function. For example, for the regularization term, the L1 norm regularization term shown in the following equation (8) is used.

  Since the risk function and the objective function are convex functions, the optimization problem of the above equation (7) is basically convex optimization, and there is always a global optimum solution.

  The initialization unit 30 sets all three parameters w ^, u ^, and α ^ used for optimization to zero. Since the calculation is repeated, the variable for managing the number of repetitions is t, and t = 0.

The system parameter update unit 32 includes three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized by the initialization unit 30, or three types of parameters w updated last time in each process described later. Using {circumflex over ⁽ u ⁾ }, {circumflex over ⁽ u ⁾ } ^(t) , and {circumflex over ⁽ α ⁾ } ^(t) , the system parameters w ^ ^{(t )} are updated to w ^ ^{(t + 1) as described} below.
At the time of the iterative calculation t, the optimal solution of w ^ when u ^ ^(t) and α ^ ^(t) are fixed is the objective function O (w ^, u ^ ^(t) , α ^ ^(t) ; D ) To minimize w).

  If only the terms related to w ^ are taken out according to the definition, the following equation is obtained.

This optimization problem can be regarded as an optimization problem in which the L2 norm regularization term is added to the above equation (2). This problem is always a convex optimization problem if the above equation (2) is a convex optimization problem, and can be easily solved using a method for solving the above equation (2).
Therefore, the system parameter update unit 32 includes the three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized by the initialization unit 30, or the three types of parameters updated last time in each process described later. Using the parameters w ^, u ^ ^(t) , α ^ ^(t) , the system parameters w ^ ^{(t )} are updated to w ^ ^{(t + 1)} according to the above equation (10).

The auxiliary parameter update unit 34 is updated by the system parameter update unit 32 ＾ ^{(t + 1)} , the parameter α ^ ^(t) initialized by the initialization unit 30, or the parameter α ^ ^(t) updated last time. As described below, the auxiliary parameters u ^ ^{(t )} are updated to u ^ ^{(t + 1)} .
When w ^ ^{(t + 1)} and α ^ ^(t) are fixed, the optimal solution of u ^ is constrained to the objective function O (w ^ ^{(t + 1)} , u ^, α ^ ^(t) ; D) u ^ ∈S ^N Is the solution to the added optimization problem. First, if only terms related to u ^ are collected, the following objective function is obtained.

  First, considering the case where there is no restriction, the gradient of the objective function O (w ^, u ^, α ^; D) with respect to u ^ is a zero vector. From this relationship, the following relational expression is obtained.

Next, considering the constraints, from the above equation (11), the optimal solution u ^ = w ^ ^{(t + 1)} + α ^ ^(t) when there is no constraint is the point belonging to the nearest ^SN at L2 distance. Recognize. Therefore, a process for finding a point belonging to the nearest ^SN from u ^ = w ^ ^{(t + 1)} + α ^ ^(t) may be performed.

Since S ^N is a Cartesian product set of S, since points that are arranged in a grid parallel to each axis are included in S ^N , the closest point in the L2 distance is simply the closest S for each parameter. To the problem of finding points included in. Thus, each parameter can be processed independently and can be obtained very efficiently.
As described above, the auxiliary parameter updating unit 34 is updated by the system parameter updating unit 32, w ^ ^{(t + 1)} , the parameter α ^ ^(t) initialized by the initialization unit 30, or the parameter α ^ updated last time. ^(T) is used to calculate u ^ according to the above equation (13), the closest point to the calculated u ^ is found from the points belonging to ^SN , and u is calculated from the auxiliary parameter u ^ ^(t). ^ Update to ^{(t + 1)} .

The undetermined multiplier updating unit 36 includes w ^ ^{(t + 1)} updated by the system parameter updating unit 32, u ^ ^{(t + 1)} updated by the auxiliary parameter updating unit 34, and the parameter α initialized by the initialization unit 30. Using La ^(t) or the parameter α ^ ^(t) updated last time, the Lagrange undetermined multiplier α ^ is updated as described below.
The direction of the optimum value when w ^ ^{(t + 1)} and u ^ ^{(t + 1)} are fixed is the gradient direction with respect to α ^ of the objective function O (w ^ ^{(t + 1)} , u ^ ^{(t + 1)} , α ^; D). is there.

From this relationship, the following update formula is obtained.

As described above, the undetermined multiplier update unit 36 is initialized by the initialization unit 30 with w ^ ^{(t + 1)} updated by the system parameter update unit 32, u ^ ^{(t + 1)} updated by the auxiliary parameter update unit 34. parameter alpha ^{^ (t),} or by using the parameters alpha ^{^} and ^(t), which was last updated, according to the above (15), to update the Lagrange multiplier alpha ^{^ (t)} alpha ^{^} to ^{(t + 1).}

The convergence determination unit 38 determines whether or not a predetermined condition is satisfied, and repeats the update process by the system parameter update unit 32, the auxiliary parameter update unit 34, and the undetermined multiplier update unit 36 until the condition is satisfied. . Specifically, the convergence determination unit 38 determines whether or not the system parameter w ^ obtained by each of the above processes is an optimal value. More specifically, two small positive real numbers ε ₁ and ε ₂ are given, and it is determined that convergence has occurred when the following expression (16) is satisfied (see the above-mentioned Reference 1).

  When the convergence is not determined by the convergence determination by the convergence determination unit 38, t = t + 1 is returned to the system parameter update unit 32. If it is determined that it has converged, the process proceeds to the duplicate parameter compression unit 40 described later.

Based on the plurality of system parameters learned by the learning unit 3, the duplicate parameter compression unit 40 forms a group with system parameters having the same value, assigns an index number to each group, and sets the group for each group. Each of the constituting system parameters is converted into a system parameter having an index number assigned to the group, and the system parameter having the index number is stored in a system parameter storage unit 4 to be described later. Specifically, as defined in the above equation (4), since the score function is assumed to have a linear format, groups are defined by system parameters having the same values such that w _i = w _j =,. To construct. Assuming that K groups are created, an index number from 1 to K is assigned to each group, and a set of K groups is constructed such that {g ′ _k } ^K _{k = 1} . Here, when w _i = w _j = w _l , g ′ _k is considered as a set of system parameter original numbers, and g ′ _k = {i, j, l} is defined. That is, it means that the conversion from the original system parameter number to the index number of the group having the same value as the system parameter is temporarily stored. Also, let v _{k be} the value of the k-th group, that is, when w _i = w _j = w _l and g ′ _k = {i, j, l}, w _i = w _j = w _l = v _k It is.

Further, the duplicate parameter compression unit 40 applies the conversion of the index number to the feature function f in the same manner. At this time, g _k is newly defined by a simple linear combination of the original feature extraction functions as in the following equation (17).

  Then, the score function of the above equation (4) is

This relationship holds. Here, the plurality of system parameters w ^ are weights of the plurality of feature extraction functions f _d (x ^, y ^) for the input data x ^ and the output data y ^. That is, the original linear function Σ ^N _{d = 1} w _d f _d ^ (x ^, y ^) can be converted equivalently to the new linear function Σ ^K _{k = 1} v _k g _k (x ^, y ^). . However, since K is the number of groups, it can be assumed that the number is much smaller than N. Moreover, since g _k is the sum of the feature functions belonging to the group, it can be easily calculated in advance.
As described above, for each group, the duplicate parameter compression unit 40 converts each system parameter constituting the group into a system parameter of the index number assigned to the group, and converts the system parameter of the index number to In addition to saving, a score function is defined as shown in equation (17) above.

Therefore, the model size can be greatly reduced by merging the overlapping parts from the system parameter set obtained by the system parameter updating unit 32 and having many overlappings and compressing them into an equivalent but useless format. As an additional process, when w _d = 0, f _d contributes nothing to the output selection, so that the definition of the feature function itself can be deleted. This processing is also performed here.

The system parameter storage unit 4 stores the system parameter v _k compressed by the duplicate parameter compression unit 40 and the newly defined score function Φ (x ^, y ^; w ^).

<System configuration of information processing apparatus>
The information processing apparatus 200 performs predetermined information processing on unknown input data using the compressed system parameters obtained by the system parameter learning apparatus 100 described above. The processing results are completely the same when the system parameters are compressed and when the system parameters are not compressed. Therefore, by performing the compression, it is possible to enjoy only the merit that the resources required at the time of execution can be reduced by the amount that the size of the model (the size of the system parameter itself) can be greatly reduced.
FIG. 7 is a block diagram showing an information processing apparatus 200 according to the embodiment of the present invention. The information processing apparatus 200 is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing an information processing routine described later, and is functionally configured as follows. .

  As shown in FIG. 7, the information processing apparatus 200 according to the present embodiment includes an input unit 5, a system parameter storage unit 6, an information processing unit 7, and an output unit 8.

  The input unit 5 accepts input data x ^.

The system parameter storage unit 6 stores a system parameter v _k compressed by the system parameter learning device and a newly defined score function g _k .

The information processing unit 7 is input based on the system parameter v _k of the index number k stored in the system parameter storage unit 6 and the newly defined score function Φ (x ^, y ^; w ^). Predetermined information processing is performed on the input data x ^ received by the unit 5. Specifically, the information processing unit 7 receives the input data x ^ received by the input unit 5, the system parameter v _k of the index number k stored in the system parameter storage unit 6, and a newly defined score function Based on Φ (x ^, y ^; w ^), output data y ^ having the maximum score function Φ (x ^, y ^; w ^) is calculated using a predetermined optimization method.

  The output unit 8 outputs the output data y ^ calculated by the information processing unit 7 as a result.

<Operation of system parameter learning device>
Next, the operation of the system parameter learning apparatus 100 according to the present embodiment will be described. First, when teacher data and a parameter ρ are input to the system parameter learning device 100, the input teacher data is stored in the teacher database 20 by the system parameter learning device 100.

  Then, the system parameter learning apparatus 100 executes the system parameter learning process routine shown in FIG.

  First, in step S100, the initialization unit 30 sets all three types of optimization variables w ^, u ^, α ^ used for optimization to 0 and initializes them.

  In step S102, the initialization unit 30 initializes the variable t for managing the number of repetitions by setting t = 0.

In step S104, the three types of parameters w ^, u ^ ^(t) , α ^ ^(t) initialized in step S100 by the system parameter update unit 32, or the three types last updated in each step described later. System parameters w ^ ^(t) to w ^ ^{(t + 1)} according to the above equation (10) using the parameters w ^, u ^ ^(t) , α ^ ^(t) .

In step S106, the auxiliary parameter update unit 34 updates w ^ ^{(t + 1)} updated in step S104, the parameter α ^ ^(t) initialized in step S100, or updated last time in step S108 described later. Using the parameter α ^ ^(t) , u ^ is calculated according to the above equation (13), the point closest to the calculated u ^ is found from the points belonging to ^SN , and the auxiliary parameter u ^ ^{(t )} To u ^ ^{(t + 1)} .

In step S108, the undetermined multiplier updating unit 36 updates w ^ ^{(t + 1)} updated in step S104, u ^ ^{(t + 1)} updated in step S106, and the parameter α ^ initialized in step S100. ^(T) or using the parameter α ^ ^(t) updated last time in this step S108, the Lagrange undetermined multiplier α ^ ^{(t )} is updated to α ^ ^{(t + 1)} according to the above equation (15).

In step S110, the convergence determining unit 38, and updated in step ^{S104 w ^ (t + 1)} , and ^{w ^} was last updated ^{(t), u ^} updated in step S106 and ^{(t + 1),} the last Based on the updated u ^ ^(t) , it is determined whether or not the convergence condition shown in the above equation (16) is satisfied. If it is determined that it has not converged, the variable t for managing repetition is incremented in step S112, the process proceeds to step S104, and the processes in steps S104 to S108 are repeated. When it determines with having converged, it transfers to step S114.

  In step S114, the duplicate parameter compression unit 40 forms a group with system parameters having the same value based on the value of the system parameter w ^ finally obtained by the update process in step S104. Each of the system parameters constituting the group is converted into the system parameter of the index number assigned to the group, the system parameter of the index number is stored in the system parameter storage unit 4, and the above equation (18) As shown in FIG. 5, a new score function is defined and stored in the system parameter storage unit 4, and the system parameter learning processing routine is terminated.

<Operation of information processing device>
Next, the operation of the information processing apparatus 200 according to this embodiment will be described. First, each of the system parameters v _k stored in the system parameter storage unit 4 of the system parameter learning device 100 and the newly defined score function Φ (x ^, y ^; w ^) Is stored in the system parameter storage unit 6. When the target input data x ^ is input to the information processing apparatus 200, the information processing apparatus 200 executes an execution processing routine shown in FIG.

  First, in step S200, the input data 5 is received by the input unit 5.

Next, in step S202, each of the system parameters v _k stored in the system parameter storage unit 6 and the newly defined score function Φ (x ^, y ^; Read.

In step S204, the information processing unit 7, the input data x ^ accepted in step S200, the read system parameters at step S202 _v respectively and the score function of _{k Φ (x ^, y ^} ; w ^) and the Based on this, output data y ^ having the maximum score function Φ (x ^, y ^; w ^) is calculated using a predetermined optimization method.

  In step S206, the output unit 8 outputs the output data y ^ calculated in step S204 as a result and ends the information processing routine.

<Experimental result>
Next, experimental results of the present embodiment are shown. In fact, in the text processing problem, since the discrete features such as words are used as so-called features that characterize the input and output at the time of identification, the total number of features is used from several million to several billions. It can often happen. FIG. 10 shows the effect of using the method described in this embodiment in the syntax analysis of natural language processing and the proper expression extraction problem.

  The horizontal axis in FIG. 10 represents the number of groups, and the vertical axis represents the analysis accuracy of the system. As can be seen from this figure, when the method of the present embodiment is used, an analysis accuracy equivalent to the conventional one can be obtained with a very small number of groups, such as 2 to 8.

  As described above, according to the system parameter learning device according to the embodiment of the present invention, based on the teacher data, each value of the plurality of system parameters is converted into a set of a predetermined finite number of discrete values. By learning multiple optimized system parameters that satisfy the included constraints, system parameters can be learned so that their values overlap, and by compressing the learned system parameters, system parameters Can be reduced in storage capacity.

  Further, according to the information processing apparatus according to the embodiment of the present invention, the input data is subjected to predetermined information processing based on the compressed system parameters learned by the system parameter learning apparatus, and the output data Since the model size (the size of the system parameter itself) can be greatly reduced, the resources required at the time of execution can be reduced.

  In addition, it is possible to greatly reduce the size of the system parameter itself while maintaining almost the same accuracy as before. Specifically, in the case of a model with 100 million system parameters, since 100,000,000 × 8 bytes = 800,000,000 bytes, a capacity of 800 MB is required. However, by using this embodiment, the unique value of the system parameter can be reduced to a very small value such as about 8. In this case, 8 × 8 bytes = 64 bytes, and in terms of calculation, the compression can be performed to about 12.5 million.

  Further, as a difference from the method shown in the prior art, in the prior art, there is no concept such as “to have the same value”, and thus such a large model size cannot be reduced. Conversely, the concept of “take the same value” in the present embodiment includes the concept of taking zero as much as possible, which is a measure of the prior art. This is because, in the present embodiment, the idea of taking the same value includes the idea of taking the same value as much as possible even when the value becomes zero. In other words, the present embodiment can be considered as a superordinate concept including the prior art. In addition, the effect is not only to increase zero as in the conventional technology, but also to significantly reduce the amount of required information for non-zero values, so that the system performance is greatly reduced compared to the conventional technology. Without significant size reduction.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the system parameter storage units 4 and 6 and the teacher database 20 may be provided outside and connected to the system parameter learning device 100 and the information processing device 200 via a network.

  In the above embodiment, the case where the system parameter learning device 100 and the information processing device 200 are configured as separate devices has been described as an example. However, the system parameter learning device 100 and the information processing device 200 are configured as one device. It may be configured.

  The system parameter learning device 100 and the information processing device 200 described above have a computer system inside. If the “computer system” uses a WWW system, a homepage providing environment (or display environment) is used. ).

  For example, in the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 1 Teacher data input part 2 Calculation part 3 Learning part 4, 6 System parameter memory | storage part 5 Input part 7 Information processing part 8 Output part 20 Teacher database 30 Initialization part 32 System parameter update part 34 Auxiliary parameter update part 36 Undecided multiplier update part 38 Convergence determining unit 40 Duplicate parameter compressing unit 100 System parameter learning device 200 Information processing device

Claims (10)

Learning a plurality of system parameters set in an information processing system that performs predetermined information processing on input data and outputs the output data, using a score function for determining the score of the output data with respect to the input data A system parameter learning device,
A teacher data input unit that inputs teacher data that is a pair of each of a plurality of input data and each of a plurality of correct output data for the plurality of input data;
Based on the teacher data input by the teacher data input unit and the score function, the values of the plurality of system parameters satisfy a constraint included in a predetermined finite set of discrete values. And a learning unit that learns the plurality of optimized system parameters. The system parameter learning device according to claim 1, wherein the learning unit learns the optimized N system parameters' w ^ according to the following expression.

However, L (Φ (), w ^, D) , the output data obtained when performing the predetermined processing using the system parameter w ^ with respect to the input data x _i of the training data D is, This is a risk function that returns a value indicating how wrong the correct output data y _i of the teacher data D is, Ω (w ^) is a regularization term, and S ^N is the N system parameters. A Cartesian product set of the finite number of discrete values for each. 3. The system parameter learning apparatus according to claim 2, wherein the learning unit learns the optimized N system parameters' w ^ according to the following expression.

Where α ^ is a Lagrange undetermined multiplier, ρ is a predetermined tuning parameter, and u ^ is an auxiliary parameter.   Based on the plurality of system parameters learned by the learning unit, a group is configured with system parameters having the same value, an index number is assigned to each group, and for each group, a system parameter that configures the group The system parameter according to any one of claims 1 to 3, further comprising a duplicate parameter compression unit that converts each into a system parameter of the index number assigned to the group and stores the system parameter of the index number. Learning device. The plurality of system parameters are weights of a plurality of feature extraction functions f _d (x ^, y ^) for input data x ^ and output data y ^,
The duplicate parameter compression unit converts, for each group, each system parameter constituting the group into a system parameter of the index number assigned to the group, and stores the system parameter of the index number, and The system parameter learning apparatus according to claim 4, wherein the score function is defined and stored as shown in the equation:

Here, Φ (x ^, y ^; w ^) is a score function, and _vk is a system parameter of the index number k assigned to the group. An input unit for receiving input data;
6. Based on the score function and the system parameter of the index number of each group stored by the system parameter learning device according to claim 4 or 5, the input information is subjected to the predetermined information processing and output. An information processing unit for outputting data;
An information processing apparatus including: An information processing system that includes a teacher data input unit and a learning unit, performs a predetermined information process on the input data using a score function for determining a score of the output data with respect to the input data, and outputs the output data A system parameter learning method in a system parameter learning device that learns a plurality of system parameters set in
Inputting teacher data which is a pair of each of a plurality of input data and each of a plurality of correct output data for the plurality of input data by the teacher data input unit;
Based on the teacher data input by the teacher data input unit and the score function by the learning unit, each value of the plurality of system parameters is set to a predetermined finite set of discrete values. Learning the plurality of system parameters that satisfy and satisfy the constraints included therein, and a system parameter learning method. An information processing method in an information processing apparatus including an input unit and an information processing unit,
Receiving input data by the input unit;
Based on the score function and the system parameter of the index number of each group stored by the system parameter learning device according to claim 4 or 5, the information processing unit performs the predetermined function on the input data. Performing information processing and outputting output data;
An information processing method including:   The program for functioning a computer as each part of the system parameter learning apparatus of any one of Claims 1-5.   The program for functioning a computer as each part of the information processing apparatus of Claim 6.