CN104580503A - Efficient dynamic load balancing system and method for processing large-scale data - Google Patents

Abstract

The invention discloses a system and method for processing large-scale data with high-efficiency dynamic load balancing, which belongs to the technical field of processing large-scale data, and its structure includes a central control system, a computing cluster system, a storage system, and a high-speed network; nodes in the central control system Adopt CPU and GPU hybrid heterogeneous architecture; nodes in the computing cluster system adopt CPU and GPU hybrid heterogeneous architecture or nodes adopt CPU architecture; storage system is divided into shared storage and local storage, shared storage nodes adopt CPU architecture, and local storage is used for storage The data of the nodes of the central control system or the nodes of the computing cluster system; the high-speed network is used to connect the nodes in the central control system, the nodes in the computing cluster system, and the nodes in the shared storage to form a centralized processing large-scale data system. The invention solves the problem that the current server computing system cannot process large-scale data due to insufficient network bandwidth and small memory capacity.

Description

A system and method for processing large-scale data with efficient dynamic load balancing

technical field

The invention relates to the technical field of processing large-scale data, in particular to a system and method for processing large-scale data with high-efficiency dynamic load balancing.

Background technique

In the current social data explosion of human beings, there are more and more information data, and people have higher and higher requirements for information data processing capabilities. Not only oil exploration, weather forecast, aerospace defense, scientific research, etc. require high-performance computing, but also financial, The demand for high-performance computing in a wider range of fields such as government informatization, education, enterprises, and online games is growing rapidly.

Computing speed is particularly important for high-performance computing. High-performance computing is developing toward multi-core and many-core, and heterogeneous parallelism is used to improve application computing speed. At present, CPU+GPU is a very mature heterogeneous collaborative computing mode, which is suitable for highly parallel computing applications or algorithms. . However, due to the relatively large scale of computing data in some applications, the way of adding hardware devices to a single server due to limitations of network bandwidth and system memory has been unable to meet the current needs. It is necessary to further design a method based on the existing limited hardware. A method capable of processing large-scale data under the device.

Contents of the invention

The technical task of the present invention is to provide a system and method for processing large-scale data with high-efficiency dynamic load balancing; realize dynamic load balancing CPU+GPU hybrid heterogeneous cluster system, make full use of the performance of the equipment, and greatly improve the efficiency of the entire system , and solve the problem that the current server computing system cannot process large-scale data due to insufficient network bandwidth and small memory capacity.

Technical task of the present invention is realized in the following manner,

A system for processing large-scale data with high-efficiency dynamic load balancing, which is a hybrid heterogeneous cluster system of CPU and GPU, including a central control system, a computing cluster system, a storage system, and a high-speed network; architecture; the nodes in the computing cluster system adopt a hybrid heterogeneous architecture of CPU and GPU or adopt CPU architecture; the storage system is divided into shared storage and local storage, the nodes in the shared storage adopt CPU architecture, and the local storage is set in the nodes of the central control system and the computing cluster In each node of the system, the shared storage is divided into main storage and backup storage. The main storage and backup storage are used as redundant storage to store the same computing data. The local storage is used for the nodes of the central control system where the storage is located or the computing cluster system The data of the nodes; the high-speed network is used to connect the nodes in the central control system, the nodes in the computing cluster system, and the nodes in the shared storage to form a centralized system for processing large-scale data.

A system for processing large-scale data with high-efficiency dynamic load balancing. The nodes in the central control system control the nodes in the computing cluster system and the nodes in the storage system.

A system for processing large-scale data with high-efficiency dynamic load balancing. There is one node in the central control system, at least one node in the shared storage system, and at least two nodes in the computing cluster system.

A method for processing large-scale data with high-efficiency dynamic load balancing, using any one of the above-mentioned large-scale data processing systems to process large-scale data, including the following steps:

(1) The nodes in the central control system are connected to the nodes in all computing cluster systems through a high-speed network, the nodes in the central control system control the nodes in each computing cluster system, and the nodes in the central control system dynamically assign computing tasks to the computing cluster systems Node, the node in the central control system receives the return result of the node in the computing cluster system;

(2) The nodes in the computing cluster system and the nodes in the shared storage are interconnected through a high-speed network, and the nodes in the central control system and the nodes in the shared storage are interconnected through a high-speed network; Nodes in the cluster system send computing task data;

(3) The nodes in the computing cluster system are responsible for computing tasks. There are multiple GPU processors of the same model in the nodes of the computing cluster system for computing; it can improve the parallelism of computing and the computing power of a single node. At the same time, GPUs of the same model Ease of division of computational tasks;

(4) The local storage in the nodes of the central control system or the nodes of the computing cluster system is used to cache the necessary local data;

(5) The nodes in the shared storage store the calculation data and calculation result data required by the nodes in the computing cluster system, and send the calculation data to the nodes in the computing cluster system through the high-speed network; at the same time, the nodes in the shared storage use the storage of primary storage and backup storage way to ensure data security.

A method for processing large-scale data with high-efficiency dynamic load balancing. The nodes in the central control system collect the computing capability information of all nodes in the computing cluster system. The nodes in the central control system dynamically divide the computing data and order the nodes in the shared storage. The calculation data is sent to the selected nodes in the computing cluster system; the nodes in the shared storage first divide the calculation data in units of data blocks according to the commands of the nodes in the central control system, and then dynamically send different numbers of data blocks to the corresponding computing nodes. Nodes in the cluster system; the nodes in the computing cluster system receive the calculation data sent by the nodes in the shared storage, and transmit the calculation result data to the nodes in the central control system, and the nodes in the central control system process the received calculation results and store them in the Nodes in shared storage.

The nodes in the computing cluster system calculate the current data block while receiving the next data block, and send the last calculated data block at the same time.

A method for processing large-scale data with high-efficiency dynamic load balancing. The workflow of the system for processing large-scale data is:

①. The nodes in the central control system are responsible for collecting the number of GPU cards of the nodes in each computing cluster system. According to the different card numbers of the nodes in each computing cluster system, the computing capability information of the nodes in each computing cluster system is generated, and the computing capability information is Send to the nodes in the shared storage; the computing capability information includes the number of node GPU cards in each computing cluster system, the communication capability of the high-speed network, and the computing capability of the GPU card;

②. According to the computing capability information sent by the nodes in the central control system, the nodes in the shared storage first divide the data into suitable basic data blocks that can be sent, and then allocate the corresponding number of computing data blocks to the nodes in each computing cluster system, and then Dynamically send data blocks to nodes in the computing cluster system;

③. While the nodes in the computing cluster system receive data for calculation, if the transmission data is fast and the calculation is not completed, the calculation data can be temporarily stored in the local storage. If there is no data transmission, it will be obtained from the local storage. If the local If not, wait;

④. When the nodes in the computing cluster system complete the calculation of data blocks, the calculation results can be sent to the nodes in the central control system. If the transmission is busy, the data can be temporarily stored in the local storage, and then sent when the network is idle To the nodes in the central control system;

⑤. The nodes in the central control system will receive the calculation results of the nodes in each computing cluster system, perform necessary processing operations, and then send them to the nodes in the shared storage. During the calculation period, the nodes in the central control system will regularly collect them in the computing cluster system The necessary information of the node is cached to the local storage and stored in the shared storage node.

An efficient dynamic load balancing system and method for processing large-scale data of the present invention has the following advantages:

1. According to the amount of data tasks to be processed, clusters are deployed on the existing hardware equipment conditions, and the dynamic load balancing method is self-adapting to the system platform, which can realize the reliability and efficiency of the system;

2. This load balancing method can be adaptive to the cluster system, which can be a complex hybrid cluster system composed of one or more subsystems of pure CPU cluster subsystem and CPU+GPU heterogeneous cluster subsystem;

3. Realize dynamic load balancing, realize load balancing between nodes in each computing cluster system in the cluster system, high utilization rate of system equipment, different computing equipment, such as CPU, GPU can achieve calculation balancing, do not wait for each other, the computing equipment in the system does not In the idle state, the entire cluster system will operate efficiently;

4. In the case of existing hardware conditions such as insufficient memory capacity and network bandwidth, adopting the method of data block transmission and calculation asynchronous processing can effectively process large-scale data;

5. Running this system will achieve high performance. This load balancing method will dynamically divide different computing tasks to nodes in different computing cluster systems according to the computing capabilities of the nodes in different computing cluster systems and the computing capabilities of different computing devices in the nodes, so as to realize The non-fixedness of computing task division improves the efficiency of dynamic load balancing;

6. According to the characteristics of the application algorithm and the different computing capabilities of different computing devices such as CPU and GPU, the computing tasks dynamically acquired by the computing devices should be set to be different;

7. Extending the present invention to multiple servers can handle relatively large-scale data, and make computing load balancing between nodes and computing devices in the cluster system, thereby maximizing the use of the performance of existing devices, Improve the efficiency of the overall operation of the system and greatly shorten the running time of the program.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings.

Accompanying drawing 1 is a schematic structural block diagram of a system for processing large-scale data of efficient dynamic load balancing;

Attached Figure 2 is a schematic block diagram of the communication of each node in an efficient dynamic load balancing system for processing large-scale data.

Detailed ways

A system and method for processing large-scale data with high-efficiency dynamic load balancing of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Example 1:

A system for processing large-scale data with high-efficiency dynamic load balancing of the present invention is a hybrid heterogeneous cluster system of CPU and GPU, including a central control system, a computing cluster system, a storage system, and a high-speed network; the nodes in the central control system use CPU and GPU GPU hybrid heterogeneous architecture; nodes in the computing cluster system adopt CPU and GPU hybrid heterogeneous architecture or CPU architecture; storage system is divided into shared storage and local storage, shared storage nodes adopt CPU architecture, and local storage is set on the nodes of the central control system And in each node of the computing cluster system, the shared storage is divided into main storage and backup storage. The main storage and backup storage are used as redundant storage to store the same computing data. The local storage is used to store the nodes of the central control system or the Compute the data of the nodes of the cluster system; the high-speed network is used to connect the nodes in the central control system, the nodes in the computing cluster system, and the nodes in the shared storage to form a centralized system for processing large-scale data.

The nodes in the central control system control the nodes in the computing cluster system and the nodes in the storage system.

There is one node in the central control system, one node in the shared storage, and two nodes in the computing cluster system.

Example 2:

A system for processing large-scale data with high-efficiency dynamic load balancing of the present invention is a hybrid heterogeneous cluster system of CPU and GPU, including a central control system, a computing cluster system, a storage system, and a high-speed network; the nodes in the central control system use CPU and GPU GPU hybrid heterogeneous architecture; nodes in the computing cluster system adopt CPU and GPU hybrid heterogeneous architecture or CPU architecture; storage system is divided into shared storage and local storage, shared storage nodes adopt CPU architecture, and local storage is set on the nodes of the central control system And in each node of the computing cluster system, the shared storage is divided into main storage and backup storage. The main storage and backup storage are used as redundant storage to store the same computing data. The local storage is used to store the nodes of the central control system or the Compute the data of the nodes of the cluster system; the high-speed network is used to connect the nodes in the central control system, the nodes in the computing cluster system, and the nodes in the shared storage to form a centralized system for processing large-scale data.

The nodes in the central control system control the nodes in the computing cluster system and the nodes in the storage system.

There is one node in the central control system, two nodes in the shared storage system, and five nodes in the computing cluster system.

Example 3:

A method for processing large-scale data with high-efficiency dynamic load balancing of the present invention uses any one of the above-mentioned systems for processing large-scale data to process large-scale data, including the following steps:

(1) The nodes in the central control system are connected to the nodes in all computing cluster systems through a high-speed network, the nodes in the central control system control the nodes in each computing cluster system, and the nodes in the central control system dynamically assign computing tasks to the computing cluster systems Node, the node in the central control system receives the return result of the node in the computing cluster system;

(2) The nodes in the computing cluster system and the nodes in the shared storage are interconnected through a high-speed network, and the nodes in the central control system and the nodes in the shared storage are interconnected through a high-speed network; Nodes in the cluster system send computing task data;

(3) The nodes in the computing cluster system are responsible for computing tasks. There are multiple GPU processors of the same model in the nodes of the computing cluster system for computing; it can improve the parallelism of computing and the computing power of a single node. At the same time, GPUs of the same model Ease of division of computational tasks;

(4) The local storage in the nodes of the central control system or the nodes of the computing cluster system is used to cache the necessary local data;

(5) The nodes in the shared storage store the calculation data and calculation result data required by the nodes in the computing cluster system, and send the calculation data to the nodes in the computing cluster system through the high-speed network; at the same time, the nodes in the shared storage use the storage of primary storage and backup storage way to ensure data security.

The nodes in the central control system collect the computing capability information of all nodes in the computing cluster system, and the nodes in the central control system dynamically divide the computing data, and order the nodes in the shared storage to send the computing data to the selected nodes in the computing cluster system; the shared storage According to the command of the nodes in the central control system, the middle node first divides the calculation data in units of data blocks, and then dynamically sends different numbers of data blocks to the corresponding nodes in the computing cluster system; the nodes in the computing cluster system receive the data in the shared storage The calculation data sent by the nodes, and the calculation result data are transmitted to the nodes in the central control system, and the nodes in the central control system process the received calculation results uniformly and store them in the shared storage nodes.

The nodes in the computing cluster system calculate the current data block while receiving the next data block, and send the last calculated data block at the same time.

A centralized control and storage method is adopted, and the large data is divided into blocks, which are dynamically distributed from the nodes in the shared storage to the nodes in each computing cluster system; in the asynchronous way of computing and communication, the nodes in the central control system It is necessary to divide the data into appropriate data blocks according to the communication capabilities of the network and the computing capabilities of the nodes in the computing cluster system, so that the calculation and transmission can cover each other to achieve optimal performance. The asynchronous method of transmission and calculation not only shortens the calculation time, but also can be applied to the calculation of large-scale data for the small memory and low bandwidth hardware devices of the system due to the block calculation.

Example 4:

A method for processing large-scale data with high-efficiency dynamic load balancing of the present invention, the workflow of the system for processing large-scale data is:

①. The nodes in the central control system are responsible for collecting the number of GPU cards of the nodes in each computing cluster system. According to the different card numbers of the nodes in each computing cluster system, the computing capability information of the nodes in each computing cluster system is generated, and the computing capability information is Send to the nodes in the shared storage; the computing capability information includes the number of node GPU cards in each computing cluster system, the communication capability of the high-speed network, and the computing capability of the GPU card;

②. According to the computing capability information sent by the nodes in the central control system, the nodes in the shared storage first divide the data into suitable basic data blocks that can be sent, and then allocate the corresponding number of computing data blocks to the nodes in each computing cluster system, and then Dynamically send data blocks to nodes in the computing cluster system;

③. While the nodes in the computing cluster system receive data for calculation, if the transmission data is fast and the calculation is not completed, the calculation data can be temporarily stored in the local storage. If there is no data transmission, it will be obtained from the local storage. If the local If not, wait;

④. When the nodes in the computing cluster system complete the calculation of data blocks, the calculation results can be sent to the nodes in the central control system. If the transmission is busy, the data can be temporarily stored in the local storage, and then sent when the network is idle To the nodes in the central control system;

⑤. The nodes in the central control system will receive the calculation results of the nodes in each computing cluster system, perform necessary processing operations, and then send them to the nodes in the shared storage. During the calculation period, the nodes in the central control system will regularly collect them in the computing cluster system The necessary information of the node is cached to the local storage and stored in the shared storage node.

In a system for processing large-scale data, calculation and communication adopt a fully asynchronous method, that is, each data block can be sent to the network after calculation is completed. It ensures that the calculation and transmission are hidden and shortens the calculation time of the entire system. At the same time, the calculation data is divided into blocks, which can make the system well applied to the situation where the network bandwidth is low and the storage space is insufficient but can handle large data. Timing backup plays a fault-tolerant function, which effectively prevents system crashes caused by interruption of nodes in the system.

Through the above specific implementation manners, those skilled in the technical field can easily realize the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. On the basis of the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features, so as to realize different technical solutions.

Claims (7)

1. a system for the process large-scale data of high-efficiency dynamic load balancing, is characterized in that, for CPU and GPU mixes Heterogeneous Cluster Environment, comprising central control system, computing cluster system, storage system, express network; Central control system interior joint adopts CPU and GPU to mix isomery framework; Computing cluster system interior joint adopts CPU and GPU mix isomery framework or adopt CPU architecture; Storage system divides shared storage and local storage, share and store interior joint employing CPU architecture, local storage is arranged in the node of central control system and each node of computing cluster system, shared storage is divided into primary storage and back-up storage, primary storage and back-up storage as redundant storage, store identical calculated data, the local data stored for the node of central control system or the node of computing cluster system storing place; Express network is used for central control system interior joint, computing cluster system interior joint, the shared interior joint that stores to be connected to each other, and forms the system of centralized process large-scale data.

2. the system of the process large-scale data of a kind of high-efficiency dynamic load balancing according to claim 1, is characterized in that central control system interior joint controlling calculation group system interior joint, storage system interior joint.

3. the system of the process large-scale data of a kind of high-efficiency dynamic load balancing according to claim 1, it is characterized in that central control system interior joint is provided with 1, share storage interior joint and be provided with at least 1, computing cluster system interior joint is provided with at least 2.

4. a method for the process large-scale data of high-efficiency dynamic load balancing, is characterized in that adopting the system of any one process large-scale data in claim 1-3, processes, comprise the steps: large-scale data

(1), central control system interior joint is interconnected by express network and all computing cluster system interior joint, central control system interior joint controls each computing cluster system interior joint, the dynamic Distribution Calculation task of central control system interior joint is to computing cluster system interior joint, and central control system interior joint receives returning results of computing cluster system interior joint;

(2), computing cluster system interior joint stored interior joint and interconnected by express network with sharing, and central control system interior joint is stored interior joint interconnected by express network with shared; Share and store interior joint according to the order of central control system interior joint to computing cluster system interior joint transmission calculation task data;

(3), computing cluster system interior joint is responsible for calculation task, has the GPU processor of multiple same model to calculate in computing cluster system interior joint;

(4) this locality, in the node of central control system or the node of computing cluster system stores, for the local necessary data of buffer memory;

(5) calculated data, required for shared storage interior joint storage computing cluster system interior joint and calculation result data, send calculated data by express network to computing cluster system interior joint; Share simultaneously and store the storage mode that interior joint adopts primary storage and back-up storage.

5. the method for the process large-scale data of a kind of high-efficiency dynamic load balancing according to claim 4, it is characterized in that central control system interior joint collects the computing capability information of all computing cluster system interior joint, calculated data divides by central control system interior joint dynamically, and calculated data is sent to the computing cluster system interior joint chosen by the shared interior joint that stores of order; Share storage interior joint first calculated data to be divided in units of data block according to the order of central control system interior joint, then the data block of varying number is sent to dynamically corresponding computing cluster system interior joint; Computing cluster system interior joint receives the calculated data shared and store interior joint and send, and calculation result data is transferred to central control system interior joint, central control system interior joint stores interior joint by being stored into share after unified for the result of calculation received process.

6. the method for the process large-scale data of a kind of high-efficiency dynamic load balancing according to claim 5, it is characterized in that computing cluster system interior joint is while the next data block of reception, calculate current data block, send a data block completed as calculated simultaneously.

7. the method for the process large-scale data of a kind of high-efficiency dynamic load balancing according to claim 4, is characterized in that the workflow of the system processing large-scale data is:

1., central control system interior joint is responsible for the quantity of the GPU card collecting each computing cluster system interior joint, the card quantity different according to each computing cluster system interior joint, generate the computing capability information of each computing cluster system interior joint, this computing capability information is sent to and shares storage interior joint; Computing capability information comprises the quantity of each computing cluster system interior joint GPU card, the communication capacity of express network, the computing capability of GPU card;

2. the computing capability information storing interior joint and send according to central control system interior joint, is shared, first data are divided into the suitable basic data block sent, then be the calculated data block that each computing cluster system interior joint distributes respective amount, then data block sent to dynamically computing cluster system interior joint;

3., computing cluster system interior joint reception data carry out calculating simultaneously, if transmit the very fast of data and calculating does not complete, calculated data can be stored into temporarily in local storage, if when there is no transfer of data, then obtain from this locality stores, if this locality also not, does not need to wait for;

4. the result of calculating can be sent to central control system interior joint while, computing cluster system interior joint completes calculated data block, if transmit busy, can first by data temporary storage in this locality store in, wait for network idle time send it to central control system interior joint again;

5., the result of calculation of each computing cluster system interior joint that will receive of central control system interior joint, carry out necessary process operation, then send to shared storage interior joint, store at the information cache of necessity of the collection computing cluster system interior joint of computing interval central control system interior joint timing to local and be stored into shared storage interior joint.