CN111858058A - SGD load balancing method, device and storage medium based on parallel computing - Google Patents

Abstract

The invention discloses an SGD load balancing method based on parallel computing, which comprises the following steps: realizing distributed parallel gpu calculation based on a design mode combining model parallel and data parallel; and a semaphore mechanism is adopted to realize synchronous communication between the main node and the sub-nodes, and the optimizer in the sub-container updates the weight by adopting a random gradient descent algorithm. The main node constructs a minimum spanning tree by taking the error in the control table of the child nodes as the weight, finds out the key nodes in the graph nodes, eliminates the nodes without nodes in sequence and redistributes the hardware resources of the nodes. The method realizes that a plurality of model copies simultaneously process different subsets of training samples, periodically carries out interactive combination on the model copies, and optimizes a distributed algorithm. The invention provides a new framework thought to realize the strategy of load balancing calculation, improves the model development efficiency and reduces the development cost, and the algorithm has better adaptability to the data scale and simultaneously realizes the asynchronous communication among the dynamic management sub-containers.

Description

SGD load balancing method, device and storage medium based on parallel computing

technical field

The present invention relates to the field of machine learning, and in particular, to an SGD load balancing method, device and storage medium based on parallel computing.

Background technique

At present, people have realized the huge advantages of artificial intelligence in many fields. Machine learning is an important part of artificial intelligence. It helps people make decisions by modeling and training massive amounts of data.

However, with the rise of big data, the scale of data is getting larger and larger, and the storage and computing power in stand-alone mode can no longer meet the requirements of massive data. Distributed machine learning emerges as the times require. Using distributed machine learning to speed up model convergence has become a mainstream approach in the industry. Currently, there are two common approaches to distributed machine learning: model parallelism and data parallelism.

However, the current parallel computing is limited by the barrel effect, and it is often necessary to wait for the slowest node to complete the calculation before proceeding to the next step. It realizes the simultaneous processing of different subsets of training samples for multiple model copies, and periodically merges the results of each model copy to provide computing efficiency under large-scale data, which requires high technical difficulty.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide an SGD load balancing method, device and storage medium based on parallel computing, using a combination of model-based parallel mode and data parallel mode. Compared with the prior art, the present invention effectively realizes that multiple model copies process different subsets of training samples at the same time, periodically merges the results of each model copy interactively, and optimizes the distributed algorithm.

The purpose of this invention is to realize through the following technical solutions:

The SGD load balancing method based on parallel computing includes the following steps:

Step 1: Build a parallel GPU computing architecture, build a one-way connected graph based on a combination of model-parallel mode and data-parallel mode, and periodically circulate models between graph nodes, so that the model covers the data set and creates a graph for the graph. Nodes preferentially allocate hardware devices;

Step 2: Dynamically manage the node hardware resources, use the semaphore mechanism to realize synchronous communication between the master node and the child nodes, and update the weights by the stochastic gradient descent algorithm in the optimizer in the child container.

Specifically, building a parallel GPU computing architecture in step 1 specifically includes the following sub-steps:

S101, configure a management node Manager, create N containers and deploy them on different machines, denoted as node Node, create a node control table on child nodes, record node ID, node data set and current batch error;

S102, establishing connections between child nodes to form a one-way connected graph, building a neural network in the child nodes, and setting a period of time slice T;

S103: Divide the data samples into N equally, send them to the nodes in sequence, use the SGD algorithm to train on different nodes, each data sample obtains a local gradient value through forward propagation and back propagation, and updates the gradient ;

S104, traverse according to the level of the graph in each training cycle, record the unbiased estimator of the model error, and record the error value in the node control table.

Specifically, the traversal process of the graph in the sub-step S104 specifically includes: encapsulating parameters such as weights and biases output by the upper-layer node into a NN object for transmission; after the current node receives the NN object transmitted by the upper-layer node, The NN object is used as the hidden layer for training; if the current node has multiple upper-layer nodes, the NN objects from the upper-layer nodes are merged, and the mean value of the NN objects is obtained as the hidden layer for training.

Specifically, the process of dynamically managing node hardware resources in step 2 specifically includes the following sub-steps:

S201, in each cycle, query the node control table through the master node, use the error in the node control table as the weight to construct a minimum spanning tree, and sort the weights in the minimum spanning tree;

S202, when the training model is about to converge, the master node sorts the nodes according to the weights according to the minimum spanning tree of each cycle in the node control table, and sends a synchronization signal to the key nodes;

S203, the master node reclaims the tasks of the nodes that have not received the synchronization signal in the one-way Unicom diagram in order, and allocates the hardware resources of the node to the adjacent key nodes to speed up the calculation speed of the adjacent key nodes until all nodes are completed. Complete training tasks.

A computing device includes a memory in which computer-executable instructions are stored; and a processor for implementing the steps of the above load balancing method when executing the computer program.

A computer-readable storage medium stores a computer program on the computer-readable storage medium, and when the computer program is executed by a processor, implements the steps of the above load balancing method.

Beneficial effects of the present invention: The present invention proposes a new architectural idea to realize the strategy of load balancing calculation, which improves the model development efficiency and reduces the development cost, so that the algorithm has better adaptability to the data scale, and at the same time realizes the Asynchronous communication between subcontainers is dynamically managed.

Description of drawings

FIG. 1 is a flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of a parallel computing architecture of the present invention.

FIG. 3 is a schematic diagram of the present invention using a semaphore mechanism to dynamically manage node hardware resources.

Detailed ways

In order to have a clearer understanding of the technical features, objects and effects of the present invention, the specific embodiments of the present invention will now be described with reference to the accompanying drawings.

In this embodiment, as shown in FIG. 1 , the SGD load balancing method based on parallel computing mainly includes the following steps:

Step 1: Build a parallel GPU computing architecture, build a one-way connected graph based on a combination of model-parallel mode and data-parallel mode, and periodically circulate models between graph nodes, so that the model covers the data set and creates a graph for the graph. Nodes preferentially allocate hardware devices;

Step 2: Dynamically manage the node hardware resources, use the semaphore mechanism to realize synchronous communication between the master node and the child nodes, and update the weights by the stochastic gradient descent algorithm in the optimizer in the child container.

In this embodiment, as shown in FIG. 2 , the present invention provides a schematic structural diagram of the SGD load balancing method based on parallel computing. The specific implementation process includes: firstly configuring a management node Manager, and then creating N containers and deploying them on different machines , denoted as node Node, create a node control table on child nodes to record node ID, node data set, and current batch error. Establish connections between child nodes to form a one-way connected graph (the graph nodes are GPU hardware devices), build a neural network in the child nodes, and set a period of time slice T. The data samples are equally divided into N parts, which are sent to the nodes in sequence, and the SGD algorithm is used to train on different nodes. Each data sample obtains a local gradient value through forward propagation and back propagation, and the gradient is updated. In each training cycle, the unbiased estimator of the model error is traversed and recorded according to the level of the graph, and the error value is recorded in the node control table. Among them, in the process of graph traversal, the weights and offsets between adjacent nodes need to be transmitted between adjacent nodes. Because the neural network is complex and has many parameters, the parameters are encapsulated into a NN object for transmission, and the node receives the upper-layer node. After the incoming NN object, the NN object is used as a hidden layer for training. If the node has multiple upper-layer nodes, the NN objects from the upper-layer nodes are merged, and the mean value of the NN objects is obtained as the hidden layer for training. Periodically circulate the model so that the model runs on all data.

Based on the architecture described in step 1, after training for a period of time, the error of some nodes will decrease very slowly, and it will take a very long training time to achieve convergence, which will greatly affect the training efficiency, and will also generate a large number of invalid calculations, causing hardware failure. waste of resources. Therefore, the present invention introduces a semaphore mechanism to realize the synchronous communication between the master node and the child nodes, and manages the dynamic management of the node hardware resources.

In this embodiment, FIG. 3 is a schematic diagram of the present invention using a semaphore mechanism to implement dynamic management of node hardware resources. The specific implementation process includes: in each cycle, the master node queries the node control table, and uses the error in the node control table. Build a minimum spanning tree as weights, and sort the weights in the minimum spanning tree. After a certain period of training (when the model is about to converge), the master node sorts the nodes according to the weights according to the minimum spanning tree of each cycle in the node control table, and sends a synchronization signal to the key nodes. Then the master node reclaims the tasks of the nodes that have not received the synchronization signal in order, and allocates the hardware resources of the node to the adjacent key nodes to speed up the calculation speed of the adjacent nodes and improve the efficiency of the entire model.

The architectural idea adopted by the present invention can effectively reduce the Loss value, improve the development efficiency of the model, reduce the development cost, and have better adaptability to the data scale.

In addition, the present invention also provides a computing device and a computer-readable storage medium. Wherein, a computing device includes a memory, in which computer-executable instructions are stored; and a processor for implementing all implementation processes and steps of the load balancing method in the embodiment when executing the computer program. A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements all the methods and steps of the above load balancing method.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments. The above-mentioned embodiments and descriptions only illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will also have Various changes and modifications fall within the scope of the claimed invention. The claimed scope of the present invention is defined by the appended claims and their equivalents.

Claims (6)

1. The SGD load balancing method based on parallel computing is characterized by comprising the following steps of:

step 1: constructing a parallel gpu computing architecture, constructing a one-way communication graph by adopting a mode of combining a model parallel mode and a data parallel mode, periodically carrying out model circulation among graph nodes, enabling a model to cover a data set, and preferentially distributing hardware equipment for the graph nodes;

step 2: and dynamically managing node hardware resources, realizing synchronous communication between the main node and the sub-nodes by adopting a semaphore mechanism, and updating the weight by adopting a random gradient descent algorithm in the optimizer in the sub-container.

2. The SGD load balancing method based on parallel computing according to claim 1, wherein the building of the parallel gpu computing architecture in step 1 specifically includes the following sub-steps:

s101, configuring a management Node Manager, creating N containers to be deployed on different machines, marking as Node nodes, creating a Node control table on a child Node, and recording a Node ID, a Node data set and a current batch error;

s102, establishing connection among the sub-nodes to form a one-way connection graph, building a neural network in the sub-nodes, and setting a time slice T of one period;

s103, evenly dividing the data samples into N parts, sending the N parts into nodes in sequence, training the nodes on different nodes by using an SGD algorithm, obtaining a local gradient value by each part of the data samples through forward propagation and backward propagation, and updating the gradient; and S104, traversing according to the hierarchy of the graph in each training period, recording the unbiased estimation quantity of the model error, and recording the error value in the node control table.

3. The SGD load balancing method according to claim 2, wherein the traversal process of the graph in the sub-step S104 specifically includes: packing parameters such as weight and bias output by an upper node into an NN object for transmission; after the current node receives the NN object transmitted by the upper node, training the NN object as a hidden layer; and if the current node has a plurality of upper nodes, merging NN objects transmitted from the upper nodes, and solving the mean value of the NN objects as a hidden layer for training.

4. The SGD load balancing method based on parallel computing according to claim 1, wherein the step 2 of dynamically managing hardware resources of nodes specifically comprises the following sub-steps:

s201, in each period, inquiring a node control table through a main node, constructing a minimum spanning tree by taking an error in the node control table as a weight, and sequencing the weights in the minimum spanning tree;

s202, when the training model is to be converged, the main node sorts the nodes according to the minimum spanning tree of each period in the node control table and the weight, and sends a synchronization signal to the key node;

and S203, the main node sequentially recovers the tasks of the nodes which do not receive the synchronous signals in the unidirectional communication graph, distributes the hardware resources of the nodes to the adjacent key nodes, and accelerates the calculation speed of the adjacent key nodes until all the nodes finish the training tasks.

5. A computing device, comprising

A memory having computer-executable instructions stored therein;

a processor for implementing the steps of the load balancing method according to any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the load balancing method according to any one of claims 1 to 4.

Images