TW202121257A - Parameter iteration method of artificial intelligence training - Google Patents

Abstract

A parameter iteration method of artificial intelligence training includes: a setting a step, providing a training set, and setting a numerical range of the training parameters; an initializing step, selecting a three initial setting values from the numerical range of the training parameters, and calculating an accuracy values of each initial setting value according to the training set, setting the initial setting value with highest accuracy valuesas a first core value, and setting a first parameter range based on the first core value; the parameter optimization step, selecting three first iteration values in the first parameter range, calculating the accuracy value of each first training set value, comparing the accuracy value of each of the first training set values, setting the value with the highest accuracy value as the second core value, and setting the second parameter range based on the second core value; and the determining step, if the accuracy value of the second core value is higher than 0.9, setting the second core value as the training parameter standard value; if not higher than 0.9, repeats the parameter optimization step until the accuracy of a test core value is higher than 0.9, and then setting the test core value as the training parameter standard value.

Description

Artificial intelligence training parameter iteration method

This application relates to the field of artificial intelligence, especially an artificial intelligence training parameter iteration method.

With the advancement of science and technology, the application of artificial intelligence is increasing. For example, defect detection, face recognition, medical judgment, etc. can gradually see the application of artificial intelligence. Before general artificial intelligence actually enters the application field, it usually needs to be trained with data. The training method of the data can be neural network, convolutional neural network (Convolutional Neural Network, CNN) and other algorithms.

At present, CNN's deep learning is the most common method of image discrimination learning and training. Because the current training parameters are usually set by random numbers, and the amount of input data is huge, the calculation process may generate a large amount of calculation data. The burden of memory and computing resources is relatively large, which results in long training time and poor efficiency.

Here, an artificial intelligence training parameter iteration method is provided. The artificial intelligence training parameter iteration method includes a setting step, an initialization step, a parameter optimization step, and a judgment step. The setting step is to provide a training set and set the numerical range of at least two training parameters. The initialization step is to randomly select at least three initial settings from the numerical range of the training parameters, and calculate the accuracy of each of the three initial settings according to the training set. The highest accuracy of the at least three initial settings is the first core value, and The parameter coordinate value of the first core value sets the first parameter range for the physical center. The parameter optimization step is to select at least three first iteration values in the first parameter range, calculate the accuracy of each first iteration value according to the training set, and compare the accuracy of at least three first iteration values, with the highest accuracy Is the second core value, and the second parameter range is set according to the physical center of the parameter coordinate value of the second core value. The judging step is to judge whether the accuracy of the second core value is higher than 0.9, if it is higher than 0.9, stop, and take the second core value as the standard value of the training parameter. If it is not higher than 0.9, replace the first core value and the first parameter range with the second core value and the second parameter range, and repeat the parameter optimization step until the accuracy of the test core value is higher than 0.9 to test the core value The parameter coordinates are the standard values of the training parameters.

In some embodiments, the at least two parameters include the number of batch learning and the learning rate, the range of the number of batch learning is 0.5 to 1.5, the range of the learning rate is 0.5 to 1.5, and the first parameter range is the number of batch learning and the learning rate. The rate is a circle with the first core value on the coordinate system of the horizontal axis and the vertical axis. In more detail, in some embodiments, the range of the number of learning is 0.7 to 1.3, and the range of the learning rate is 0.7 to 1.3.

In more detail, in some embodiments, in the initialization step, two graphics processors respectively perform the selection of the batch learning quantity and the learning rate of the initial setting value, and the calculation of the accuracy rate.

In some embodiments, the at least two parameters further include momentum, and the range of momentum is 0 to 1. Here, the first parameter range is when the number of batch learning, the learning rate, and the momentum are the x-axis, y-axis, and z-axis, respectively. A sphere with the first core value as the center of the sphere on a standard system. In more detail, the range of momentum is 0.3 to 0.8.

In some embodiments, the at least two parameters further include normalization, and the range of normalization is 0.00001 to 0.001, where the first parameter range is the number of batches learned, learning rate, momentum, and normalization are x-axis, y-axis, On the z-axis and w-axis of a standard system, the first core value is the physical quantity range of the physical center. In more detail, in some embodiments, the range of normalization is 0.0001 to 0.0005.

In some embodiments, in the initialization step, any two of the batch learning number, learning rate, momentum, and normalization are selected by the first graphics processor, and the batch learning number, learning rate, momentum, and normalization are selected by the first graphics processor. The other two are selected by the second graphics processor, and the third graphics processor calculates the accuracy.

In some embodiments, the artificial intelligence training parameter iteration method further includes a verification step. The verification step is to provide a test set, and calculate the accuracy rate based on the second core value or test core value whose accuracy rate is higher than 0.9 in the judgment step. If the accuracy rate is lower than 0.9 calculated according to the test set, the initialization step is performed again.

To sum up, through the artificial intelligence training parameter iteration method, the selection of parameter values can be accelerated, and the overall artificial intelligence training process can be accelerated, so as to achieve the effect of a rapid decline in the gradient of manual training with a smaller number of files and a shorter speed. , And can shorten the training time and greatly improve the efficiency of training.

Figure 1 is a flowchart of an iterative method for artificial intelligence training parameters. As shown in FIG. 1, the artificial intelligence training parameter iteration method S1 includes a setting step S10, an initialization step S20, a parameter optimization step S30, and a judgment step S40.

，其中w表示為權重、n為批次學習量(batch size)、η為學習率。The setting step S10 is to provide a training set and set the numerical range of at least two training parameters. Here, a Convolutional Neural Network (CNN) is used for artificial intelligence training, and a model of the stochastic gradient descent method is used as a training set, which can be as shown in the following equation (1). However, this is only an example, not a limitation. The training set here further includes calculation methods such as loss calculation. Equation (1):

, Where w is the weight, n is the batch size, and η is the learning rate.

As shown in the aforementioned equation (1), the batch learning volume and learning rate will actually affect the convergence of the weights. When the batch learning volume is large, the number of batches can be reduced, and the training time can be reduced. Ground, a larger amount of learning will require a larger number of iterations, and make the performance of the model worse, and the amount of data generated is also larger. The learning rate is too small, which means that the weight update range is very small, which will make the training very slow; then the learning rate is too large, which may lead to failure to converge. Therefore, in addition to providing an appropriate training set, the setting step S10 also needs to set the range of parameters. For example, the set parameters include the number of batch learning and the learning rate. The range of the number of batch learning is 0.5 to 1.5, and the range of the learning rate is 0.5 to 1.5. Preferably, the range of the learning amount is 0.7 to 1.3, and the range of the learning rate is 0.7 to 1.3. However, the above is only an example, not a limitation, and the relevant parameters are not only the number of batch learning and the learning rate.

Fig. 2 is a schematic diagram of the initialization step. As shown in Fig. 2, the initialization step S20 is to randomly select at least three initial setting values A1, A2, and A3 from the numerical range of the training parameter. Here, the following uses the batch learning amount as the horizontal axis and the learning rate to randomly select the three initial settings A1, A2, and A3. The coordinate values on the coordinate system are (x1, y1), (x2, y2), ( x3, y3) are examples. This is only for the convenience of presentation on the plane, and the actual parameters are not limited to this.

。然而，這僅為示例，也可以預選為特定半徑值作為第一參數範圍R1。Then, the accuracy of each three initial setting values is calculated according to the training set, and the accuracy rate (Accuracy Rate, ACC) is calculated as 1-loss. After calculation, compare the accuracy of the three, and take the most accurate of at least three initial settings A1, A2, and A3 (assuming A1) as the first core value, and use the parameter coordinate value of the first core value (x1 , y1) Set the first parameter range R1 for the physical center. Here, the first parameter range R1 is a circle with the first core value (x1, y1) as the center on a coordinate system where the number of batch learning and the learning rate are the horizontal axis and the vertical axis, respectively. Here, the radius of the circle can be

. However, this is only an example, and a specific radius value can also be pre-selected as the first parameter range R1.

。然而，這僅為示例，也可以預選為特定半徑值作為第一參數範圍R1Figure 3 is a schematic diagram of the parameter optimization step. As shown in Figure 3, the parameter optimization step S30 is to select at least three first iteration values B1, B2, B3 in the first parameter range R1, and calculate the accuracy of each first iteration value B1, B2, B3 according to the training set , And compare the accuracy rates of at least three first iteration values B1, B2, B3, take the highest accuracy rate (assuming B3) as the second core value, and based on the parameter coordinate value (x6, y6) of the second core value as The physical center sets a second parameter range R2. Here, the radius of the circle can be

. However, this is only an example, and a specific radius value can also be pre-selected as the first parameter range R1

The judging step S40 is to judge whether the accuracy of the second core value is higher than 0.9, if it is higher than 0.9, stop, go to step S45, and select the second core value (x6, y6) as the training parameter standard value for training.

。可以依此方式重覆判斷步驟S40及參數優化步驟S30，直到某次的測試核心值的準確率高於0.9，以測試核心值之參數座標為訓練參數標準值。Figure 4 is a schematic diagram of the parameter optimization step. As shown in Fig. 4, if it is determined in step S40 that the accuracy of the second core value is not higher than 0.9, the second core value (x6, y6) and the second parameter range R2 are used to replace the first core value (x1, y1). ) And the first parameter range R1, repeat the parameter optimization step S30, select at least three second iteration values C1, C2, C3, and calculate the accuracy of each second iteration value C1, C2, C3 according to the training set, and compare at least three The second training sets the accuracy of the C1, C2, and C3 values. The highest accuracy (assuming C1) is the third core value, and the third core value C1's parameter coordinate value (x7, y7) is set as the physical center. Out of the third parameter range R3. Here, the radius of the circle can be

. The judgment step S40 and the parameter optimization step S30 can be repeated in this way until the accuracy of the tested core value is higher than 0.9 at a certain time, and the parameter coordinates of the tested core value are used as the training parameter standard value.

Referring to FIG. 1 again, the artificial intelligence training parameter iteration method S1 further includes a verification step S50. The verification step S50 is to provide a test set. The values in the test set and the training set are not the same. In the verification step S50, the second core value whose accuracy rate is higher than 0.9 in step S40 will be judged, or it will be obtained by repeating the parameter optimization step S30. The core value of the test calculates the accuracy rate based on the test set. If the accuracy rate calculated based on the test set is higher than 0.9, the second core value or the test core value is used as the standard value of the training parameter. If the accuracy rate is lower than 0.9, the parameter is discarded, and the initialization step S20 is performed again.

In the foregoing embodiment, in the initialization step S20, the two graphics processors respectively perform the selection of the batch learning number and the learning rate of the initial setting value, and the calculation of the accuracy rate. That is, the selection of the initial setting values A1, A2, A3 in FIG. 2 and the calculation of the accuracy of the initial setting values A1, A2, A3 are respectively calculated by two graphics processors. In this way, the computing resources of the graphics processor can be dispersed to achieve faster computing efficiency.

However, Figures 2 to 4 are only examples. The parameters that actually affect training efficiency are momentum and regularization. If three parameters are set, such as the number of batches, learning rate, and momentum, it can be understood that the first The parameter range is a sphere with the first core value as the center of the sphere on a standard system where the number of batch learning, learning rate, and momentum are x-axis, y-axis, and z-axis respectively. And if there are four parameters, the number of batch learning, learning rate, momentum and normalization, the first parameter range is the number of batch learning, learning rate, momentum, and normalization are x-axis, y-axis, z-axis, w The axis coordinate system takes the first core value as the physical quantity range of the physical center. However, drawing three-axis and four-axis space on a plane is not easy to show an iterative effect. It is not shown here. Those with ordinary knowledge in the field can think about three-axis and four-axis space based on Figures 2 to 4 The transformation.

In more detail, if momentum and normalization are considered, the range of momentum is 0 to 1, preferably, the range is 0.3 to 0.8. The range of normalization is 0.00001 to 0.001, preferably, the range is 0.0001 to 0.0005.

Further, in the initialization step S20, considering momentum and normalization, any two of the batch learning quantity, learning rate, momentum, and normalization can be selected by the first graphics processor, and the other two can be selected by the first graphics processor. The second graphics processor selects, and the third graphics processor calculates the accuracy. In this way, the computing resources are distributed through the three graphics processors to achieve faster computing effects and make the gradient drop faster.

Here, the comparative example is compared with the embodiment. The actual training set used is the standard training set provided by google. The comparative example is the standard AI framework, using Tesla 40m (2880CUDA, 12GB) GPU, and using google tensorflow mode For training; and the embodiment uses four GTX 1050i (768CUDA, 4GB) GPUs for serial connection. Using the foregoing embodiment, the number of batch learning, learning rate, momentum, and regularization are respectively 0.7 to 1.3 and 0.7 to 1.3, 0.3 to 0.8, 0.001 to 0.005 range for training. The result of the comparative example is an accuracy of 86% at 14400 seconds; the result of the example is an accuracy of 98% at 900 seconds.

In summary, through the artificial intelligence training parameter iteration method of this application, the selection of parameters can be quickly optimized, and the effect of gradient descent convergence can be quickly achieved, which helps to improve training efficiency, reduce computing resources, and Complete training with cheaper hardware costs.

Although the technical content of the present invention has been disclosed in the preferred embodiments as above, it is not intended to limit the present invention. Anyone who is familiar with this technique and makes some changes and modifications without departing from the spirit of the present invention should be covered by the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of the attached patent application.

S1: Artificial intelligence training parameter iteration method S10: Setting procedure S20: Initialization steps S30: Parameter optimization steps S40: Judgment step S45: selected training parameters S50: Inspection steps A1, A2, A3: Initial setting value B1, B2, B3: the first iteration value C1, C2, C3: second iteration value R1: The first parameter range R2: The second parameter range R3: The third parameter range

Figure 1 is a flowchart of an iterative method for artificial intelligence training parameters. Figure 2 is a schematic diagram of the initialization step. Figures 3 and 4 are schematic diagrams of parameter optimization steps.

S1: Artificial intelligence training parameter iteration method

S10: Setting procedure

S20: Initialization steps

S30: Parameter optimization steps

S40: Judgment step

S45: selected training parameters

S50: Inspection steps

Claims (10)

An artificial intelligence training parameter iteration method, including: A setting step, providing a training set, and setting a value range of at least two training parameters; In an initialization step, at least three initial setting values are randomly selected from the numerical range of the training parameter, and an accuracy rate of each initial setting value is calculated according to the training set, and the highest accuracy rate among the at least three initial setting values is one A first core value, and a first parameter range is set with the physical center of the parameter coordinate value of the first core value; A parameter optimization step, selecting at least three first iteration values in the first parameter range, calculating the accuracy rate of each first iteration value according to the training set, and comparing the accuracy rates of the at least three first iteration values , Taking the highest accuracy rate as a second core value, and setting a second parameter range according to the physical center of the parameter coordinate value of the second core value; and A judgment step is to judge whether the accuracy of the second core value is higher than 0.9, if it is higher than 0.9, stop, take the second core value as a training parameter standard value, if it is not higher than 0.9, use the second core value The core value and the second parameter range replace the first core value and the first parameter range, and the parameter optimization step is repeated until the accuracy of a test core value is higher than 0.9, and the parameter coordinates of the test core value are the Standard values of training parameters. The artificial intelligence training parameter iteration method according to claim 1, wherein the at least two parameters include a batch learning quantity and a learning rate, the batch learning quantity ranges from 0.5 to 1.5, and the learning rate ranges from 0.5 to 1.5. The first parameter range is a circle with the center of the first core value on a standard system with the number of batches of learning and the learning rate being the horizontal axis and the vertical axis, respectively. The artificial intelligence training parameter iteration method according to claim 2, wherein the batch learning quantity ranges from 0.7 to 1.3, and the learning rate ranges from 0.7 to 1.3. The artificial intelligence training parameter iteration method according to claim 3, wherein in the initialization step, two graphics processors respectively perform the selection of the batch learning quantity and the learning rate of the initial setting value and the calculation of the accuracy rate. The artificial intelligence training parameter iteration method according to claim 2, wherein the at least two parameters further include a momentum, the momentum ranges from 0 to 1, and the first parameter range is the number of learning in the batch, the learning rate And the momentum is a sphere with the first core value as the center of the sphere on a standard system of x-axis, y-axis, and z-axis. The artificial intelligence training parameter iteration method according to claim 5, wherein the range of the momentum is 0.3 to 0.8. The artificial intelligence training parameter iteration method according to claim 5, wherein the at least two parameters further include a normalization, and the normalization ranges from 0.00001 to 0.001, and the first parameter range is the number of learning in the batch, the The learning rate, the momentum, and the normalization are respectively a range of physical quantities on a standard system of x-axis, y-axis, z-axis, and w-axis with the first core value as the physical center. The artificial intelligence training parameter iteration method described in claim 7, wherein the normalization range is 0.0001 to 0.0005. The artificial intelligence training parameter iteration method according to claim 7, wherein in the initialization step, any two of the batch learning quantity, the learning rate, the momentum, and the normalization are performed by a first graphics processor Select, and the other two of the batch learning quantity, the learning rate, the momentum, and the normalization are selected by a second graphics processor, and a third graphics processor performs the calculation of the accuracy. The artificial intelligence training parameter iteration method described in claim 1, further comprising a verification step, providing a test set, and using the second core value or the test core value with an accuracy rate higher than 0.9 in the judgment step according to the test The accuracy rate is calculated based on the test set. If the accuracy rate is calculated to be lower than 0.9 based on the test set, the initialization step is performed again.

Images