CN114298277B - A distributed deep learning training method and system based on layer sparsification - Google Patents

Abstract

The application discloses a distributed deep learning training method and system based on layer sparsification, which belong to the technical field of distributed training communication sparsification and comprise the following steps: obtaining a normalized window center list according to the convergence characteristic of the neural network model; obtaining a layer transmission list by using a layer sparsification method and a normalized window center list; performing distributed deep learning training based on layer sparsification according to the layer transmission list to obtain weight updating parameters, and completing the distributed deep learning training based on layer sparsity; the application solves the problem that the existing training frame is only thinned in the network layer, effectively improves the thinning degree and reduces the traffic.

Description

A distributed deep learning training method and system based on layer sparsification

Technical field

The invention belongs to the technical field of distributed training communication sparsification, and in particular relates to a distributed deep learning training method and system based on layer sparsification.

Background technique

In recent years, with the continuous development of deep learning, models have become larger and more complex, and training these complex large models on a single machine is very time-consuming. In order to reduce training time, a distributed training method is proposed to accelerate model training. Specifically, distributed training mainly includes two methods: model parallelism and data parallelism. Among them, model parallelism is difficult to accelerate due to the splitting of model parameters to different computing nodes, coupled with the imbalance of parameter sizes at different levels and high computing dependence between nodes. Data parallelism splits the training data set while maintaining the complete model at each computing node. In each iteration, each computing node uses different training data to calculate local gradients, and then transfers and exchanges them.

When each computing node performs local gradient transfer, the expansion of distribution is limited due to the large amount of parameters and limited network bandwidth. In order to solve this bottleneck, two different methods of reducing communication volume, sparsification and quantization, are proposed. Among them, the sparsification method aims to reduce the number of elements transmitted in each iteration, set the vast majority of elements to zero, and only transmit the most valuable gradients to update parameters, thereby ensuring the convergence of training.

In a basic synchronous distributed training framework SSGD (Synchronous Stochastic Gradient Descent) without communication compression, each computing node must wait for all nodes to complete the transmission of all parameters in the current iteration. Without compression, excessive communication volume The communication load caused has become its biggest bottleneck. In practical applications, when computing resources are limited, it can only be applied to the training of some smaller models; in a distributed depth that performs deep communication compression within the network layer In the learning and training framework DGC (Deep Gradient Compression), many existing auxiliary technologies have been added to overcome the loss problem caused by gradient sparseness within the network layer and greatly reduce the gradient communication volume between nodes. However, DGC every Secondary communication still needs to pass all network layers of the model, and for networks with deeper model depth, bottlenecks still exist.

Contents of the invention

In view of the above-mentioned deficiencies in the prior art, the present invention applies the convergence characteristics of the neural network model to distributed training, and provides a distributed deep learning training method and system based on layer sparsification, which solves the problem that the existing training framework only The problem of sparsity within the network layer effectively improves the degree of sparsity and reduces communication volume.

In order to achieve the above-mentioned object of the invention, the technical solutions adopted by the present invention are:

The present invention provides a distributed deep learning training method based on layer sparsification, which includes the following steps:

S1. According to the convergence characteristics of the neural network model, obtain the normalized window center list;

S2. Use the layer sparsification method and the normalized window center list to obtain the layer transmission list;

S3. Perform distributed deep learning training based on layer sparsification according to the layer transmission list, obtain weight update parameters, and complete distributed deep learning training based on layer sparsification.

The beneficial effects of the present invention are: the present invention provides a distributed deep learning training method based on layer sparseness. According to the neural network training process, the key learning opportunities of different layers are different, and the most needed learning at the moment is selected. The network layer performs communication synchronization, and other layers stay local for accumulation. When transmitting gradients between nodes, the gradients of some layers are selected for inter-node communication, which effectively reduces the communication volume.

Further, the step S1 includes the following steps:

S11. Set all layers of the neural network as a continuous sequence of layers, and set the total number of training times of the neural network;

S12. Set a dynamic window according to the convergence characteristics of the neural network model. The dynamic window traverses the continuous sequence of layers from back to front as the number of neural network training times increases, and sets the overall number of traversals of the dynamic window;

S13. Based on the total number of training times of the neural network and the number of overall traversals of the dynamic window, calculate the number of training times of the neural network and the remaining number of neural network training times during a single traversal of the neural network in the dynamic window;

S14. According to the number of training times of the neural network during a single traversal of the neural network in the dynamic window, the normalized step size of the movement during the dynamic window traversal is obtained;

S15. Based on the normalized step size, iterate the number of overall traversals of the dynamic window and the number of training times of the neural network in the process of a single traversal of the dynamic window through the neural network model, and obtain a list of the entire cycle window centers;

S16. Determine whether the remaining number of neural network training times is zero. If so, normalize the entire period window center list as the normalized window center list. Otherwise, proceed to step S17;

S17. Based on the normalized step size moved during dynamic window traversal, iterate the training times of the remaining neural network to obtain the remaining window center list;

S18. Add the remaining window center list to the end of the entire period window center list as the normalized window center list.

The beneficial effects of adopting the above further solution are: according to the convergence characteristics of the neural network model, a dynamic window is established. In the early stage of training, the network layer at the bottom of the model is mainly transmitted. As the training cycle increases, the key layers of transmission also move up. Each layer The importance of gradients is not the same. It provides the basis for layer sparseness in distributed training and effectively prevents the selection probability of some layers from being too low through multiple traversals.

Further, the specific steps of step S2 are as follows:

S21. Obtain the warm-up period and normalized window center list of the neural network;

S22. According to the layer sparsification method, normalize the sequence numbers of the network layers in the neural network to obtain a list of normalized layer sequence numbers;

S23. As the current training period is in the warm-up period, transmit all parameters of all layers of the neural network, otherwise proceed to step S24;

S24. According to the normalized window center list and the normalized layer serial number list, obtain the dynamic window list and the sampling list within the current training cycle window;

S25. According to the normalized layer serial number list and the dynamic window list, obtain the sampling list outside the window of the current training cycle;

S26. Merge the sampling list within the current training cycle window and the sampling list outside the current training cycle window to obtain a layer transmission list.

The beneficial effects of adopting the above further solution are: in the early stage of neural network training, the dynamic window is at the bottom of the model. As the neural network training progresses, the dynamic window gradually slides forward to the top layer. By setting the preset ratio and sampling ratio of the network layer inside the dynamic window, The preset proportion of network layer sampling outside the dynamic window reduces the amount of transmitted data, and when the model is deep enough, the advantages are obvious.

Further, the step S24 includes the following steps:

S241. Obtain the window center of the current training cycle according to the normalized window center list;

S242. Use the window center and the normalization layer sequence number list of the current training cycle as the expectation and independent variable lists respectively, and calculate the standard normal distribution list;

S243. Select the sequence number of the network layer corresponding to the preset amount at the head of the standard normal distribution list to obtain a dynamic window list;

S244. Obtain the sampling list within the current training cycle window by randomly and uniformly sampling the preset proportion k _in of the dynamic window list.

The beneficial effects of adopting the above further solution are: by randomly and uniformly sampling the dynamic window list, the amount of transmitted data is effectively reduced, and when the model is deep enough, the advantages are obvious.

Further, the step S25 includes the following steps:

S251. Obtain the dynamic window external list according to the normalized layer serial number list and the dynamic window list;

S252. Obtain the sampling list outside the window of the current training cycle by randomly and uniformly sampling the preset proportion k _out of the dynamic window external list.

The beneficial effect of adopting the above further solution is that by randomly and uniformly sampling the neural network layer outside the dynamic window list, the amount of transmitted data is effectively reduced, and when the model is deep enough, the advantages are obvious.

Further, the step S3 includes the following steps:

S31. Obtain the neural network layer gradient list through neural network sample feedforward and feedback calculations;

S32. Traverse each layer in the neural network layer gradient list layer by layer, and determine whether each layer is in the layer transmission list. If so, obtain several selected layers and enter step S33. Otherwise, obtain several local accumulated gradients;

S33. Determine whether each selected layer has intra-layer compression. If so, obtain a number of selected layers with intra-layer compression and enter step S34. Otherwise, obtain a number of selected layer transmission gradients without intra-layer compression;

S34. Perform intra-layer sparsification, inter-node communication and decompression synchronization in sequence within each compressed selected layer with intra-layer compression, and obtain a number of selected layer transmission gradients with intra-layer compression;

S35. Globally average each local accumulated gradient, each selected layer transmission gradient without intra-layer compression, or each selected layer transmission gradient with intra-layer compression, to obtain a complete gradient;

S36. According to the complete gradient, obtain the weight update parameters and complete the distributed deep learning training based on layer sparsification.

The beneficial effects of adopting the above further scheme are: according to the results of judging whether each layer in the neural network is in the layer transmission list and whether there is intra-layer compression, the transmission gradient is obtained through local accumulation, direct transmission and inter-node communication respectively, and through global average Implement gradient fusion to obtain complete gradients, and then obtain weight update parameters based on the complete gradients to complete distributed deep learning training based on layer sparsification.

The present invention also provides a system for a distributed deep learning training method based on layer sparsification, including:

The normalized window center list acquisition module is used to obtain the normalized window center list based on the convergence characteristics of the neural network model;

The layer transmission list acquisition module is used to obtain the layer transmission list using the layer sparsification method and the normalized window center list;

The distributed deep learning training module based on layer sparsification performs distributed deep learning training based on layer sparsification according to the layer transmission list, obtains weight update parameters, and completes distributed deep learning training based on layer sparsification.

The beneficial effects of the present invention are: the system of the distributed deep learning training method based on layer sparsification provided by the present invention is a system corresponding to the distributed deep learning training method based on layer sparse provided by the present invention, and is used to implement the distributed deep learning training method based on layer sparse. Sparse distributed deep learning training method.

Description of drawings

Figure 1 is a flow chart of the steps of a distributed deep learning training method based on layer sparsification in an embodiment of the present invention.

Figure 2 is a schematic diagram of the window center moving with the training cycle in the embodiment of the present invention.

Figure 3 is a schematic diagram of dynamic window movement in an embodiment of the present invention.

Figure 4 is a schematic diagram of obtaining a dynamic window list based on a standard normal distribution list in an embodiment of the present invention.

Figure 5 is a schematic diagram of the all_reduce loop transmission method in the embodiment of the present invention.

Figure 6 is a schematic diagram of the time-consuming situation of training the Resnet110 model on the Cifar10 and Cifar100 data sets using the DGC and LS-DGC frameworks respectively in the embodiment of the present invention.

Figure 7 is a schematic diagram of the time-consuming situation of training the Resnet18 model in the Cifar10 data set using the SSGD and LS-SSGD frameworks respectively, and the time-consuming situation of the Resnet50 model using the SSGD and LS-SSGD frameworks respectively in training the Cifar100 data set in the embodiment of the present invention.

Figure 8 is a schematic diagram of the convergence of the Resnet18 model on Cifar10 using SSGD and LS-SSGD frameworks respectively, and the convergence of the Resnet50 model on Cifar100 using SSGD and LS-SSGD frameworks respectively in the embodiment of the present invention.

Figure 9 is a block diagram of a system of a distributed deep learning training method based on layer sparsification in an embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the technical field, as long as various changes These changes are obvious within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions and creations utilizing the concept of the invention are protected.

Based on the conclusions on the convergence characteristics of the neural network model derived from the dynamics of deep representation learning, this solution proposes a distributed deep learning training method and system based on layer sparsification. During the training process of the neural network, the key learning timing of different layers is different. This provides feasible support for this solution to sparse the gradient from the perspective of the network layer, and communicates and synchronizes the network layer that currently needs to learn most. , while other layers remain local for accumulation. Consider that the network layer at the bottom of the model is mainly transferred in the early stage of training. As the training cycle increases, the transfer focus layer also moves up, which also provides support for the layer sparseness of distributed training. , the present invention therefore provides a method and system for effectively reducing communication traffic.

Example 1

As shown in Figure 1, in one embodiment of the present invention, the present invention provides a distributed deep learning training method based on layer sparsification, which includes the following steps:

S1. According to the convergence characteristics of the neural network model, obtain the normalized window center list;

According to the convergence characteristics of the neural network model, a dynamic window is established. In the early stage of training, the network layer at the bottom of the model is mainly transmitted. As the training cycle increases, the key layer of transmission also moves up. The importance of the gradient of each layer is not the same. Provides a basis for layer sparsification in distributed training;

S2. Use the layer sparsification method and the normalized window center list to obtain the layer transmission list;

In the early stage of neural network training, the dynamic window is at the bottom of the model. As the neural network training progresses, the dynamic window gradually slides forward to the top layer. By setting the preset sampling ratio of the network layer inside the dynamic window and the preset sampling ratio of the network layer outside the dynamic window , reducing the amount of data to be transmitted, and when the model is deep enough, the advantages are obvious;

S3. Perform distributed deep learning training based on layer sparsification according to the layer transmission list, obtain weight update parameters, and complete distributed deep learning training based on layer sparsification.

According to the results of judging whether each layer in the neural network is in the layer transmission list and whether there is intra-layer compression, the transmission gradient is obtained through local accumulation, direct transmission and inter-node communication, and the gradient fusion is achieved through global averaging to obtain the complete gradient. Complete gradient, obtain weight update parameters, and complete distributed deep learning training based on layer sparsification.

The beneficial effects of the present invention are: the present invention provides a distributed deep learning training method based on layer sparseness. According to the neural network training process, the key learning opportunities of different layers are different, and the most needed learning at the moment is selected. The network layer performs communication synchronization, and other layers stay local for accumulation. When transmitting gradients between nodes, the gradients of some layers are selected for inter-node communication, which effectively reduces the communication volume.

Example 2

Regarding step S1 of Embodiment 1, it includes the following sub-steps S11 to S18:

S11. Set all layers of the neural network as a continuous sequence of layers, and set the total number of training times of the neural network;

S12. Set a dynamic window according to the convergence characteristics of the neural network model. The dynamic window traverses the continuous sequence of layers from back to front as the number of neural network training times increases, and sets the overall number of traversals of the dynamic window;

S13. Based on the total number of training times of the neural network and the number of overall traversals of the dynamic window, calculate the number of training times of the neural network and the remaining number of neural network training times during a single traversal of the neural network in the dynamic window;

S14. According to the number of training times of the neural network during a single traversal of the neural network in the dynamic window, the normalized step size of the movement during the dynamic window traversal is obtained;

S15. Based on the normalized step size, iterate the number of overall traversals of the dynamic window and the number of training times of the neural network in the process of a single traversal of the dynamic window through the neural network model, and obtain a list of the entire cycle window centers;

S16. Determine whether the remaining number of neural network training times is zero. If so, normalize the entire period window center list as the normalized window center list. Otherwise, proceed to step S17;

S17. Based on the normalized step size moved during dynamic window traversal, iterate the training times of the remaining neural network to obtain the remaining window center list;

S18. Add the remaining window center list to the end of the entire period window center list as the normalized window center list;

As shown in Figure 2, in this embodiment, the neural network model is set to have 300 layers and is trained for 103 epochs. At the same time, the dynamic window performs 4 overall traversals and 1 partial traversal of the model (the remainder of 103/4); then The total number of training times is 103 times, and the overall number of traversals is 4 times. Then the number of neural network training times in a single traversal of the neural network is 25 times, the remaining neural network training times are 3 times, and the normalized step size of dynamic window movement is 1 /24 (counting from 0); considering that not all times are exactly integer traversals, similar iterations are performed when there are remaining neural network training times to generate the remaining window centers, and then the normalized window center list is obtained by adding the tail .

Example 3

Regarding step S2 in Embodiment 1, it includes the following sub-steps S21 to S26:

S21. Obtain the warm-up period and normalized window center list of the neural network;

Usually the warm-up period is set to the first 5 epochs of neural network training, and sparsification is not performed during the warm-up period to prevent the neural network from going in the wrong direction;

S22. According to the layer sparsification method, normalize the sequence numbers of the network layers in the neural network to obtain a list of normalized layer sequence numbers;

S23. As the current training period is in the warm-up period, transmit all parameters of all layers of the neural network, otherwise proceed to step S24;

S24. According to the normalized window center list and the normalized layer serial number list, obtain the dynamic window list and the sampling list within the current training cycle window;

The step S24 includes the following steps:

S241. Obtain the window center of the current training cycle according to the normalized window center list;

S242. Use the window center and the normalization layer sequence number list of the current training cycle as the expectation and independent variable lists respectively, and calculate the standard normal distribution list;

S243. Select the sequence number of the network layer corresponding to the preset amount at the head of the standard normal distribution list to obtain a dynamic window list;

S244. Obtain the sampling list within the current training cycle window by randomly and uniformly sampling the preset proportion k _in of the dynamic window list;

S25. According to the normalized layer serial number list and the dynamic window list, obtain the sampling list outside the window of the current training cycle;

The step S25 includes the following steps:

S251. Obtain the dynamic window external list according to the normalized layer serial number list and the dynamic window list;

S252. Obtain the sampling list outside the window of the current training cycle by randomly and uniformly sampling the preset proportion k _out of the dynamic window external list;

S26. Merge the sampling list within the current training cycle window and the sampling list outside the current training cycle window to obtain the layer transmission list;

As shown in Figure 3, in this embodiment, the neural network is set to have 20 layers. In the early stage of training, the window is at the bottom of the model. As the training progresses, the window gradually slides forward to the top layer, and a dynamic window list is set to randomly and uniformly sample pre-sets. Suppose the proportion k _in =50%, and the dynamic window external list random uniform sampling preset proportion k _out =12.5%, then the overall model compression rate k =20%, further reducing the amount of transmitted data. When the model is deep enough At this time, the advantages of this method are obvious; no sparsification is performed during the warm-up period to prevent the model from going in the wrong direction; according to the normalized window center list, the window center of the current training period is obtained; the sum of the window centers of the current training period is normalized The layer serial number list is used as the expectation and independent variable list respectively, and the standard normal distribution list is calculated; then the layer serial number corresponding to the top-20% in the standard normal distribution list is selected as the dynamic window list;

As shown in Figure 4, the vertex of the standard normal distribution list is the window center, and only the top-20% is intercepted. The corresponding abscissa is the dynamic window list corresponding to the current window center, that is, the network layer sequence. In the dynamic window list Perform random uniform sampling in the current training cycle window to obtain the sampling list within the current training cycle window; remove the top-20% dynamic window and perform random uniform sampling to obtain the sampling list outside the current training cycle window, and merge the sampling list within the current training cycle window with the current training Sample the list outside the period window to obtain the layer transmission list.

Example 4

Regarding step S3 in Embodiment 1, it includes the following sub-steps S31 to S36:

S31. Obtain the neural network layer gradient list through neural network sample feedforward and feedback calculations;

S32. Traverse each layer in the neural network layer gradient list layer by layer, and determine whether each layer is in the layer transmission list. If so, obtain several selected layers and enter step S33. Otherwise, obtain several local accumulated gradients;

S33. Determine whether each selected layer has intra-layer compression. If so, obtain a number of selected layers with intra-layer compression and enter step S34. Otherwise, obtain a number of selected layer transmission gradients without intra-layer compression;

S34. Perform intra-layer sparsification, inter-node communication and decompression synchronization in sequence within each compressed selected layer with intra-layer compression, and obtain a number of selected layer transmission gradients with intra-layer compression;

S35. Globally average each local accumulated gradient, each selected layer transmission gradient without intra-layer compression, or each selected layer transmission gradient with intra-layer compression, to obtain a complete gradient;

S36. According to the complete gradient, obtain the weight update parameters and complete the distributed deep learning training based on layer sparsification;

In this embodiment, when the selected layer has intra-layer compression, layer sparsification is performed by the same method in step S2 of the distributed deep learning training method based on layer sparsification before intra-layer sparsification and inter-node communication, and the desired layer is selected. Communicating gradient layers to reduce the sparsification overhead of neural network layers that do not communicate between nodes. During the warm-up period, all layers are transmitted without layer sparsification to prevent the training from going in the wrong direction. After the degree of sparsification within the layer is stable, the layer sparsification strategy is called to reduce the communication volume between nodes, as shown in Figure 5 , When communicating between nodes within the selected layer for compression within each layer, the all_reduce cyclic transmission method is used. First, cyclic transmission between nodes is performed. After 4 rounds of transmission, each node has the gradient information of other nodes. ; Then perform averaging to obtain the average gradient. After decompression, a number of intra-layer compression selected layer transmission gradients are obtained.

Example 5

In a practical example of the present invention, this solution was tested on two image classification data sets, including the simple Cifair10 data set and the complex Cifair100 data set, to prove the distributed deep learning based on layer sparsification proposed by this solution. Effectiveness of the training method; where Cifar10 consists of 50,000 training images and 10,000 validation images of 10 classes, while Cifar100 contains 100 categories, each category has 500 training images and 100 test images. All experiments are based on the Pytorch distributed training framework and are run on a machine equipped with four GeForce RTX 3090 graphics cards.

First, a layer sparse distributed framework (LS-DGC) training experiment based on deep gradient compression DGC was conducted, and the warm-up period was still set to 4 cycles in 164 training cycles. In addition, since DGC has already performed up to 99% compression within the layer, it should not be too low when performing layer sparsification; this solution uses 20% of the model sequence as the sliding window size, and selects all inner layers of the window k _in =100%, setting k _out =20% of the layer outside the window to prevent external gradient expiration, then the communication volume of the DGC algorithm based on layer sparsification is reduced to about 36% of the original algorithm;

As shown in Figure 6, in the two trainings using the Resnet110 model, the LS-DGC framework is lower than the DGC framework. Since DGC has done a large degree of intra-layer compression, the time has been greatly shortened, and when combined with the DGC-based After layer sparsification algorithm, the time consumption is further improved; in addition, in terms of time proportion within the cycle, this scheme comparatively analyzes two data sets, and the time changes of the two methods in the whole cycle, compression and communication, decompression and synchronization stages. , the results are shown in Table 1:

Table 1

According to Table 1, in terms of time consumption, the average time consumption of compression communication and decompression synchronization has dropped significantly, and the proportion within the cycle has also dropped by about half, further alleviating the bottleneck problem of communication between nodes. In addition, since DGC is based on layer-based pipeline communication, when LS-DGC performs layer sparsification, some layers no longer communicate, and the frequency of communication between nodes is also greatly reduced;

Compared with the baseline DGC framework, LS-DGC further reduces the communication volume between nodes to alleviate communication bottlenecks, which will inevitably affect the model convergence speed to a certain extent. The accuracy of LS-DGC is slightly lower while keeping the number of training cycles unchanged. The decrease is due to the fact that the model has not converged yet. Increase the number of training cycles appropriately (the total time consuming remains unchanged). When the model fully converges, its accuracy will be further improved and even surpass the results of the baseline method, as shown in Table 2:

Table 2

method Cifar10 accuracy Cifar100 accuracy DGC(baseline) 93.55% 72.04% LS-DGC 93.08% 71.74% LS-DGC(more) 94.15%(↑) 72.55%(↑)

Secondly, a layer sparsification distributed framework (LS-SSGD) training experiment based on the random fast descent method SSGD is performed. In order to verify the universality of layer sparsification, the present invention also conducts experiments on SSGD without intra-layer compression. On SSGD with intra-layer compression, we reduce the layer selection probability within the window to k _in =50%, and at the same time reduce the probability outside the window to k _out =10%, thus reducing the overall communication volume of SSGD to about 18% of the original algorithm. ; Since there is no compression within the layer and the gradient loss is small, using Resnet18 to train the Cifar10 data set and Resnet50 to train the Cifar100 data set can also converge well;

As shown in Figure 7, in terms of time, similarly, the LS-SSGD framework is still less time-consuming than the training of SSGD, and the compression ratio within the stage, communication synchronization and the entire cycle time are both reduced, of which the communication synchronization time is reduced by 50%. Above, as shown in Table 3:

table 3

As shown in Figure 8, in terms of accuracy, compared with the baseline SSGD results, LS-SSGD can surpass the SSGD results in the same training cycle. The LS-SSGD framework is better than the SSGD framework throughout the training process. This solution It is believed that this is because in the absence of intra-layer compression, layer-based sparsification can play a role similar to Dropout, improving the generalization ability of the model, thereby achieving better performance.

The beneficial effects of this solution are: applying the convergence characteristics of the neural network model to distributed training, and proposing a distributed training framework for layer sparsification of the neural network, which solves the problem of previous training frameworks being only sparse within the network layer. Further improve the degree of sparsification; combined with the existing intra-layer deep sparsification framework DGC and no intra-layer sparsification framework SSGD, the distributed deep learning frameworks LS-DGC and LS- based on layer sparsification proposed in this scheme were implemented through experiments. SSGD; experiments were conducted on multiple classification models and multiple image data sets, and analysis and comparison were conducted in terms of overall time-consuming, communication volume, communication proportion, accuracy, etc., which fully proved the effectiveness and advancement of our method.

Example 6

As shown in Figure 9, this solution also provides a system for distributed deep learning training methods based on layer sparsification, including:

The normalized window center list acquisition module is used to obtain the normalized window center list based on the convergence characteristics of the neural network model;

The layer transmission list acquisition module is used to obtain the layer transmission list using the layer sparsification method and the normalized window center list;

The distributed deep learning training module based on layer sparsification performs distributed deep learning training based on layer sparsification according to the layer transmission list, obtains weight update parameters, and completes distributed deep learning training based on layer sparsification.

The system of the distributed deep learning training method based on layer sparseness provided by the embodiment can execute the technical solution shown in the above method embodiment based on the distributed deep learning training method based on layer sparseness. Its implementation principle and beneficial effects are similar. Here No longer.

In the embodiment of the present invention, the application can divide functional units according to the distributed deep learning training method based on layer sparsification. For example, each function can be divided into functional units, or two or more functions can be integrated. in a processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional unit. It should be noted that the division of units in the present invention is schematic and is only a logical division. In actual implementation, there may be other division methods.

In the embodiment of the present invention, the system of the distributed deep learning training method based on layer sparsification includes the hardware structure and/or corresponding hardware structure for executing each function in order to realize the principles and beneficial effects of the distributed deep learning training method based on layer sparsification. Software modules. Those skilled in the art should easily realize that, in conjunction with the schematic units and algorithm steps described in the disclosed embodiments of the present invention, the present invention can be implemented in the form of hardware and/or a combination of hardware and computer software. A certain function can be implemented in hardware. Whether it is executed in a computer software-driven manner depends on the specific application and design constraints of the technical solution. Different methods can be used to implement the described functions for each specific application, but this implementation should not be considered beyond the scope of this application. range.

Claims (5)

1. A distributed deep learning training method based on layer sparsification, which is characterized by including the following steps: S1. According to the convergence characteristics of the neural network model, obtain the normalized window center list; S2. Use the layer sparsification method and the normalized window center list to obtain the layer transmission list; The specific steps of step S2 are as follows: S21. Obtain the warm-up period and normalized window center list of the neural network; S22. According to the layer sparsification method, normalize the sequence numbers of the network layers in the neural network to obtain a list of normalized layer sequence numbers; S23. As the current training period is in the warm-up period, transmit all parameters of all layers of the neural network, otherwise proceed to step S24; S24. According to the normalized window center list and the normalized layer serial number list, obtain the dynamic window list and the sampling list within the current training cycle window; S25. According to the normalized layer serial number list and the dynamic window list, obtain the sampling list outside the window of the current training cycle; S26. Merge the sampling list within the current training cycle window and the sampling list outside the current training cycle window to obtain the layer transmission list; S3. Perform distributed deep learning training based on layer sparsification according to the layer transmission list, obtain weight update parameters, and complete distributed deep learning training based on layer sparsification; The step S3 includes the following steps: S31. Obtain the neural network layer gradient list through neural network sample feedforward and feedback calculations; S32. Traverse each layer in the neural network layer gradient list layer by layer, and determine whether each layer is in the layer transmission list. If so, obtain several selected layers and enter step S33. Otherwise, obtain several local accumulated gradients; S33. Determine whether each selected layer has intra-layer compression. If so, obtain a number of selected layers with intra-layer compression and enter step S34. Otherwise, obtain a number of selected layer transmission gradients without intra-layer compression; S34. Perform intra-layer sparsification, inter-node communication and decompression synchronization in sequence within each compressed selected layer with intra-layer compression, and obtain a number of selected layer transmission gradients with intra-layer compression; S35. Globally average each local accumulated gradient, each selected layer transmission gradient without intra-layer compression, or each selected layer transmission gradient with intra-layer compression, to obtain a complete gradient; S36. According to the complete gradient, obtain the weight update parameters and complete the distributed deep learning training based on layer sparsification. 2. The distributed deep learning training method based on layer sparsification according to claim 1, characterized in that the step S1 includes the following steps: S11. Set all layers of the neural network as a continuous sequence of layers, and set the total number of training times of the neural network; S12. Set a dynamic window according to the convergence characteristics of the neural network model. The dynamic window traverses the continuous sequence of layers from back to front as the number of neural network training times increases, and sets the overall number of traversals of the dynamic window; S13. Based on the total number of training times of the neural network and the number of overall traversals of the dynamic window, calculate the number of training times of the neural network and the remaining number of neural network training times during a single traversal of the neural network in the dynamic window; S14. According to the number of training times of the neural network during a single traversal of the neural network in the dynamic window, the normalized step size of the movement during the dynamic window traversal is obtained; S15. Based on the normalized step size, iterate the number of overall traversals of the dynamic window and the number of training times of the neural network in the process of a single traversal of the dynamic window through the neural network model, and obtain a list of the entire cycle window centers; S16. Determine whether the remaining number of neural network training times is zero. If so, normalize the entire period window center list as the normalized window center list. Otherwise, proceed to step S17; S17. Based on the normalized step size moved during dynamic window traversal, iterate the training times of the remaining neural network to obtain the remaining window center list; S18. Add the remaining window center list to the end of the entire period window center list as the normalized window center list. 3. The distributed deep learning training method based on layer sparsification according to claim 1, characterized in that the step S24 includes the following steps: S241. Obtain the window center of the current training cycle according to the normalized window center list; S242. Use the window center and the normalization layer sequence number list of the current training cycle as the expectation and independent variable lists respectively, and calculate the standard normal distribution list; S243. Select the sequence number of the network layer corresponding to the preset amount at the head of the standard normal distribution list to obtain a dynamic window list; S244. Obtain the sampling list within the current training cycle window by randomly and uniformly sampling the preset proportion k _in of the dynamic window list. 4. The distributed deep learning training method based on layer sparsification according to claim 1, characterized in that the step S25 includes the following steps: S251. Obtain the dynamic window external list according to the normalized layer serial number list and the dynamic window list; S252. Obtain the sampling list outside the window of the current training cycle by randomly and uniformly sampling the preset proportion k _out of the dynamic window external list. 5. A system of distributed deep learning training method based on layer sparsification, which is characterized by including: The normalized window center list acquisition module is used to obtain the normalized window center list based on the convergence characteristics of the neural network model; The layer transmission list acquisition module is used to obtain the layer transmission list using the layer sparsification method and the normalized window center list, which is specifically: A1. Obtain the warm-up period and normalized window center list of the neural network; A2. According to the layer sparsification method, normalize the sequence numbers of the network layers in the neural network to obtain a list of normalized layer sequence numbers; A3. As the current training period is in the warm-up period, transmit all parameters of all layers of the neural network, otherwise proceed to step A4; A4. According to the normalized window center list and the normalized layer sequence number list, obtain the dynamic window list and the sampling list within the current training cycle window; A5. According to the normalized layer serial number list and the dynamic window list, obtain the sampling list outside the window of the current training cycle; A6. Merge the sampling list within the current training cycle window and the sampling list outside the current training cycle window to obtain the layer transmission list; The distributed deep learning training module based on layer sparsification performs distributed deep learning training based on layer sparsification according to the layer transmission list, obtains weight update parameters, and completes distributed deep learning training based on layer sparsification. The details are: B1. Obtain the neural network layer gradient list through neural network sample feedforward and feedback calculations; B2. Traverse each layer in the neural network layer gradient list layer by layer, and determine whether each layer is in the layer transmission list. If so, get a number of selected layers and enter step B3. Otherwise, get a number of local accumulated gradients; B3. Determine whether each selected layer has intra-layer compression. If so, obtain a number of selected layers with intra-layer compression and enter step B4. Otherwise, obtain a number of selected layer transmission gradients without intra-layer compression; B4. Perform intra-layer sparsification, inter-node communication and decompression synchronization in sequence within each compressed selected layer with intra-layer compression, and obtain several selected layer transmission gradients with intra-layer compression; B5. Globally average each local accumulated gradient, each selected layer transmission gradient without intra-layer compression, or each selected layer transmission gradient with intra-layer compression, to obtain the complete gradient; B6. Based on the complete gradient, obtain the weight update parameters and complete distributed deep learning training based on layer sparsification.