TWI752380B - Parameter iteration method of artificial intelligence training - Google Patents

Abstract

A parameter iteration method of artificial intelligence training includes: a setting a step, providing a training set, and setting a numerical range of the training parameters; an initializing step, selecting a three initial setting values from the numerical range of the training parameters, and calculating an accuracy values of each initial setting value according to the training set, setting the initial setting value with highest accuracy valuesas a first core value, and setting a first parameter range based on the first core value; the parameter optimization step, selecting three first iteration values in the first parameter range, calculating the accuracy value of each first training set value, comparing the accuracy value of each of the first training set values, setting the value with the highest accuracy value as the second core value, and setting the second parameter range based on the second core value; and the determining step, if the accuracy value of the second core value is higher than 0.9, setting the second core value as the training parameter standard value; if not higher than 0.9, repeats the parameter optimization step until the accuracy of a test core value is higher than 0.9, and then setting the test core value as the training parameter standard value.

Description

Iterative method of artificial intelligence training parameters

The present application relates to the field of artificial intelligence, in particular to an iterative method for training parameters of artificial intelligence.

With the advancement of science and technology, the application of artificial intelligence is getting larger and larger. For example, defect detection, face recognition, medical judgment, etc. can gradually see the application of artificial intelligence. Before general artificial intelligence actually enters the application field, it usually needs to be trained with data. The training method of data can be algorithms such as neural network and convolutional neural network (CNN).

Currently, CNN's deep learning is the most common image discrimination learning and training method. Since the current training parameters are usually set by random numbers and the amount of input data is huge, the calculation process may generate a large amount of calculation data. The burden of memory and computing resources is relatively large, resulting in long training time and poor efficiency.

Here, an iterative method for artificial intelligence training parameters is provided. The artificial intelligence training parameter iteration method includes a setting step, an initialization step, a parameter optimization step, and a judgment step. The setting step is to provide a training set and set the numerical range of at least two training parameters. The initialization step is to randomly select at least three initial setting values from the numerical range of the training parameters, and calculate the accuracy rate of each of the three initial setting values according to the training set, and the highest accuracy rate among the at least three initial setting values is the first core value. The parameter coordinate value of the first core value sets the first parameter range for the physical center. The parameter optimization step is to select at least three first iteration values in the first parameter range, and calculate the accuracy rate of each first iteration value according to the training set, and compare the accuracy rates of at least three first iteration values, with the highest accuracy rate is the second core value, and the second parameter range is set according to the parameter coordinate value of the second core value for the physical center. The judging step is to judge whether the accuracy of the second core value is higher than 0.9, if it is higher than 0.9, stop, and use the second core value as the standard value of the training parameter. If it is not higher than 0.9, the second core value and the second parameter range are used to replace the first core value and the first parameter range, and the parameter optimization steps are repeated until the accuracy of the test core value is higher than 0.9. The parameter coordinates are the standard values of the training parameters.

In some embodiments, the at least two parameters include a batch learning quantity and a learning rate, the range of the batch learning quantity is 0.5 to 1.5, the range of the learning rate is 0.5 to 1.5, and the first parameter range is the batch learning quantity and the learning rate. On a coordinate system whose rate is the horizontal axis and the vertical axis respectively, a circle with the first core value as the center of the circle. In more detail, in some embodiments, the learning amount ranges from 0.7 to 1.3 and the learning rate ranges from 0.7 to 1.3.

More specifically, in some embodiments, in the initialization step, the two GPUs respectively perform the selection of the batch learning quantity and the learning rate of the initial set value, and the calculation of the accuracy rate.

In some embodiments, the at least two parameters further include momentum, and the range of the momentum is 0 to 1. Here, the first parameter range is the number of batch learning, the learning rate, and the momentum, which are the x-axis, the y-axis, and the z-axis, respectively. A sphere with the first core value as the center of the sphere on a coordinate system. In more detail, the range of momentum is 0.3 to 0.8.

In some embodiments, the at least two parameters further include normalization, and the normalization range is 0.00001 to 0.001, wherein the first parameter range is the number of batch learning, learning rate, momentum, and normalization are the x-axis, y-axis, On the coordinate system of the z-axis and the w-axis, the first core value is the physical quantity range of the physical center. In more detail, in some embodiments, the normalization ranges from 0.0001 to 0.0005.

In some embodiments, in the initialization step, any two of batch learning size, learning rate, momentum, normalization are selected by the first graphics processor, while batch learning size, learning rate, momentum, normalization The other two are selected by the second graphics processor, and the accuracy is calculated by the third graphics processor.

In some embodiments, the artificial intelligence training parameter iteration method further includes a checking step. The verification step is to provide a test set, and calculate the accuracy rate with the second core value or the test core value with the accuracy rate higher than 0.9 in the judgment step. If the accuracy rate calculated according to the test set is lower than 0.9, the initialization step is performed again.

In summary, through the artificial intelligence training parameter iteration method, the selection of parameter values can be accelerated, and the overall artificial intelligence training process can be accelerated, so that the artificial training can achieve the effect of rapid gradient descent with a smaller number of files and a shorter speed. , which can shorten the training time and greatly improve the training efficiency.

Fig. 1 is a flowchart of an iterative method of artificial intelligence training parameters. As shown in FIG. 1 , the artificial intelligence training parameter iteration method S1 includes a setting step S10 , an initialization step S20 , a parameter optimization step S30 , and a judgment step S40 .

，其中w表示為權重、n為批次學習量(batch size)、η為學習率。 The setting step S10 is to provide a training set and set the numerical range of at least two training parameters. Here, artificial intelligence training is performed with a convolutional neural network (CNN), and a model of stochastic gradient descent is used as a training set, which can be shown as the following equation (1). However, this is only an example rather than a limitation, and the training set here also includes calculation methods such as loss calculation. Equation (1):

, where w is the weight, n is the batch size, and η is the learning rate.

As shown in Equation (1) above, the batch learning amount and learning rate will actually affect the convergence of weights. When the batch learning amount is large, the number of batches can be reduced, and the training time can be reduced, but relatively In general, a larger learning amount will require a larger number of iterations, resulting in a poorer model performance and a larger amount of data generated. If the learning rate is too small, it means that the weight update range is very small, which will make the training very slow; then the learning rate is too large, which may lead to failure to converge. Therefore, the setting step S10 not only provides an appropriate training set, but also needs to set the range of parameters. For example, the set parameters include the number of batches and the learning rate. The range of the number of batches is 0.5 to 1.5, and the range of the learning rate is 0.5 to 1.5. Preferably, the learning amount ranges from 0.7 to 1.3, and the learning rate ranges from 0.7 to 1.3. However, the above is only an example, not a limitation, and the relevant parameters are not only the number of batch learning and the learning rate.

FIG. 2 is a schematic diagram of an initialization step. As shown in FIG. 2 , in the initialization step S20 , at least three initial setting values A1 , A2 , and A3 are randomly selected from the numerical range of the training parameters. Here, with the batch learning amount as the horizontal axis and the learning rate, the coordinate values of the three initial set values A1, A2, and A3 on the coordinate system are randomly selected as (x1, y1), (x2, y2), ( x3, y3) are examples. This is only presented on a plane for convenience, and the actual parameters are not limited to this.

。然而，這僅為示例，也可以預選為特定半徑值作為第一參數範圍R1。 Next, according to the training set, the accuracy rates of the three initial setting values are calculated, and the calculation method of the accuracy rate (Accuracy Rate, ACC) is 1-loss. After calculation, compare the accuracy rates of the three, and take the highest accuracy among the at least three initial setting values A1, A2, and A3 (assuming A1) as the first core value, and use the parameter coordinate value of the first core value (x1 , y1) set the first parameter range R1 for the physical center. Here, the first parameter range R1 is a circle with the first core value (x1, y1) as the center on a coordinate system in which the number of batch learning and the learning rate are respectively the horizontal axis and the vertical axis. Here, the radius of the circle can be

. However, this is only an example, and a specific radius value may also be preselected as the first parameter range R1.

。然而，這僅為示例，也可以預選為特定半徑值作為第一參數範圍R1 FIG. 3 is a schematic diagram of parameter optimization steps. As shown in FIG. 3, the parameter optimization step S30 is to select at least three first iteration values B1, B2, B3 in the first parameter range R1, and calculate the accuracy of each first iteration value B1, B2, B3 according to the training set , and compare the accuracy rates of at least three first iteration values B1, B2, B3, take the one with the highest accuracy rate (assuming B3) as the second core value, and according to the parameter coordinate values (x6, y6) of the second core value as The physical center sets the second parameter range R2. Here, the radius of the circle can be

. However, this is only an example, it is also possible to preselect a specific radius value as the first parameter range R1

The judgment step S40 is to judge whether the accuracy rate of the second core value is higher than 0.9, if it is higher than 0.9, stop, and enter step S45, select the second core value (x6, y6) as the training parameter standard value for training.

。可以依此方式重覆判斷步驟S40及參數優化步驟S30，直到某次的測試核心值的準確率高於0.9，以測試核心值之參數座標為訓練參數標準值。 FIG. 4 is a schematic diagram of parameter optimization steps. As shown in FIG. 4 , if the determination step S40 determines that the accuracy of the second core value is not higher than 0.9, the second core value (x6, y6) and the second parameter range R2 are used to replace the first core value (x1, y1 ) ) and the first parameter range R1, repeat the parameter optimization step S30, select at least three second iteration values C1, C2, C3, and calculate the accuracy of each second iteration value C1, C2, C3 according to the training set, and compare at least three The second training sets the accuracy rates of C1, C2, and C3 values, takes the highest accuracy rate (assuming C1) as the third core value, and sets the physical center according to the parameter coordinate values (x7, y7) of the third core value C1 out the third parameter range R3. Here, the radius of the circle can be

. The determination step S40 and the parameter optimization step S30 can be repeated in this way until the accuracy rate of a certain test core value is higher than 0.9, and the parameter coordinates of the test core value are used as the training parameter standard value.

Referring to FIG. 1 again, the artificial intelligence training parameter iteration method S1 further includes a checking step S50. The verification step S50 is to provide a test set, and the values in the test set and the training set are different. In the verification step S50, the second core value with an accuracy rate higher than 0.9 in the step S40 or the subsequent repeated parameter optimization step S30 will be determined in the verification step S50. The test core value is calculated based on the test set to calculate the accuracy. If the accuracy rate calculated according to the test set is higher than 0.9, the second core value or the test core value is used as the standard value of the training parameter. On the other hand, if the accuracy rate is lower than 0.9, the parameter is discarded, and the initialization step S20 is performed again.

In the aforementioned embodiment, in the initialization step S20, the two graphics processors respectively perform the selection of the batch learning quantity and the learning rate of the initial set value, and the calculation of the accuracy rate. That is, the selection of the initial setting values A1 , A2 , and A3 in FIG. 2 and the calculation of the accuracy of the initial setting values A1 , A2 , and A3 are respectively calculated by two graphics processors. In this way, the computing resources of the graphics processor can be dispersed to achieve faster computing efficiency.

However, Figures 2 to 4 are only examples. The parameters that actually affect the training efficiency are momentum and normalization. If three parameters are set, such as the number of batch learning, learning rate and momentum, it can be understood that the first The parameter range is a sphere with the first core value as the center of the sphere on a coordinate system in which the number of batch learning, the learning rate, and the momentum are the x-axis, y-axis, and z-axis, respectively. And if there are four parameters, batch learning quantity, learning rate, momentum and normalization, the first parameter range is the batch learning quantity, learning rate, momentum, and normalization, respectively x-axis, y-axis, z-axis, w On the coordinate system of the axis, the first core value is the physical quantity range of the physical center. However, it is not easy to show the iterative effect when drawing three-axis and four-axis spaces on a plane, which is not shown here, but those with ordinary knowledge in the art can think about three-axis and four-axis spaces according to Figures 2 to 4. transformation.

In more detail, if momentum and normalization are considered, the range of momentum is 0 to 1, preferably, the range is 0.3 to 0.8. The normalized range is 0.00001 to 0.001, preferably, the range is 0.0001 to 0.0005.

Further, in the initialization step S20, considering momentum and normalization, any two of the batch learning quantity, learning rate, momentum, and normalization can be selected by the first graphics processor, and the other two can be selected by the first graphics processor. The second graphics processor selects, and the third graphics processor performs the calculation of the accuracy. In this way, computing resources are distributed through three graphics processors to achieve faster computing effects and make the gradient drop faster.

Here, the comparative example is compared with the embodiment. The actual training set used is the standard training set provided by google. The comparative example is based on the standard AI framework, using Tesla 40m (2880CUDA, 12GB) GPU, using google tensorflow mode In the embodiment, four GTX 1050i (768CUDA, 4GB) GPUs are used for serial connection. Using the foregoing embodiment, the batch learning quantity, learning rate, momentum, and normalization are respectively 0.7 to 1.3, 0.7 to 1.3, 0.3 to 0.8, 0.001 to 0.005 range for training. The result of the comparative example is 86% accuracy at 14400 seconds; and the result of the example is 98% accuracy at 900 seconds.

To sum up, through the artificial intelligence training parameter iteration method of the present application, the selection of parameters can be quickly optimized, and the effect of gradient descent convergence can be quickly achieved, which helps to improve the efficiency of training, reduce computing resources, Complete your training with less expensive hardware.

Although the technical content of the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Any person who is familiar with the art, makes some changes and modifications without departing from the spirit of the present invention, should be included in the present invention. Therefore, the protection scope of the present invention should be determined by the scope of the appended patent application.

S1 Iterative method of artificial intelligence training parameters S10 Setting steps S20 initialization steps S30 Parameter optimization steps S40 Judgment step S45 selected training parameters S50 Inspection steps A1, A2, A3 Initial setting value B1, B2, B3 first iteration value C1, C2, C3 second iteration value R1 first parameter range R2 Second parameter range R3 third parameter range    

Fig. 1 is a flowchart of an iterative method of artificial intelligence training parameters. FIG. 2 is a schematic diagram of initialization steps. 3 and 4 are schematic diagrams of parameter optimization steps.

S1 Iterative method of artificial intelligence training parameters S10 Setting steps S20 initialization steps S30 Parameter optimization steps S40 Judgment step S45 selected training parameters S50 Inspection steps

Claims (8)

An iterative method for artificial intelligence training parameters, comprising: a setting step, providing a training set, and setting a numerical range of at least two training parameters; an initialization step, randomly selecting at least three initial setting values from the numerical range of the training parameters, and calculate an accuracy rate of each of the initial setting values according to the training set, the highest accuracy rate among the at least three initial setting values is a first core value, and the parameter coordinate value of the first core value is set as the physical center A first parameter range is obtained; in a parameter optimization step, in the first parameter range, at least three first iteration values are selected, and the accuracy rate of each first iteration value is calculated according to the training set, and the at least three third values are compared. For the accuracy rate of an iterative value, the one with the highest accuracy rate is a second core value, and a second parameter range is set according to the parameter coordinate value of the second core value for the physical center; and a judgment step is to judge the first parameter range. Whether the accuracy rate of the two core values is higher than 0.9, if it is higher than 0.9, stop, and use the second core value as a training parameter standard value, if not higher than 0.9, then use the second core value and the second parameter range, replace the first core value and the first parameter range, repeat the parameter optimization step until the accuracy of a test core value is higher than 0.9, and use the parameter coordinate of the test core value as the standard value of the training parameter, wherein the At least two training parameters include a batch learning quantity and a learning rate, the batch learning quantity ranges from 0.5 to 1.5, the learning rate ranges from 0.5 to 1.5, and the first parameter range is the batch learning quantity and The learning rate is on the coordinate system of the horizontal axis and the vertical axis respectively, and the first core value is a circle with the center of the circle, wherein the range of the batch learning quantity is 0.7 to 1.3, and the range of the learning rate is 0.7 to 1.3. The artificial intelligence training parameter iteration method as claimed in claim 1, wherein in the initialization step, the selection of the batch learning quantity and the learning rate of the initial set value and the calculation of the accuracy rate are respectively performed by two graphics processors. The artificial intelligence training parameter iteration method according to claim 1, wherein the at least two parameters further include a momentum, the range of the momentum is 0 to 1, wherein the first parameter range is the number of learning in the batch, the learning rate And the momentum is a sphere with the first core value as the center of the sphere on the coordinate system of the x-axis, the y-axis, and the z-axis respectively. The artificial intelligence training parameter iteration method of claim 3, wherein the momentum is in the range of 0.3 to 0.8. The artificial intelligence training parameter iteration method according to claim 3, wherein the at least two parameters further include a normalization, and the normalization range is 0.00001 to 0.001, wherein the first parameter range is the number of learning in the batch, the The learning rate, the momentum, and the normalization are a range of physical quantities with the first core value as the physical center on the coordinate system of the x-axis, the y-axis, the z-axis, and the w-axis, respectively. The artificial intelligence training parameter iteration method as claimed in claim 5, wherein the normalization range is 0.0001 to 0.0005. The artificial intelligence training parameter iteration method according to claim 5, wherein in the initialization step, any two of the batch learning quantity, the learning rate, the momentum, and the normalization are performed by a first graphics processor. and the other two of the batch learning quantity, the learning rate, the momentum, and the normalization are selected by a second graphics processor, and the accuracy is calculated by a third graphics processor. The artificial intelligence training parameter iteration method according to claim 1, further comprising a checking step, providing a test set, and using the second core value or the test core value with the accuracy rate higher than 0.9 in the judgment step according to the test If the accuracy rate is calculated according to the test set and is lower than 0.9, the initialization step is performed again.

Images