CN105302526A - Data processing system and method - Google Patents

Abstract

The invention provides a data processing system. The system comprises a master node and a plurality of slave nodes, wherein the master node is used for batch reading of to-be-processed data; the master node is also used for sending the to-be-processed data to each slave node after reading each time, updating a network according to a weight returned by each slave node, sending updated network information parameters to each slave node, and reading a next batch of to-be-processed data; and the slave nodes are used for performing forward-backward calculation on the received data sent by the main node to obtain weights and returning the weights to the master node. According to the scheme, a master-slave calculation mode is adopted, so that the time for processing a deep learning application is shortened and the calculation efficiency is improved.

Description

A data processing system and method

technical field

The invention relates to the field of computers, in particular to a data processing method and system.

Background technique

In 2006, Geoffrey Hinton, a professor at the University of Toronto in Canada and a leader in the field of machine learning, and his student Ruslan Salakhutdinov published an article in the top academic journal "Science", which opened a wave of deep learning in academia and industry. Since 2006, deep learning has continued to gain momentum in academia. Stanford University, New York University, University of Montreal, Canada, etc. have become important centers for deep learning research. In 2010, the DARPA program of the U.S. Department of Defense funded deep learning projects for the first time, and the participants included Stanford University, New York University and NEC American Research Institute. An important basis for supporting deep learning is that the brain nervous system does have a rich hierarchical structure. One of the most famous examples is the Hubel-Wiesel model, which won the Nobel Prize in Medicine and Physiology for revealing the mechanism of the optic nerve. In addition to the perspective of bionics, the current theoretical research on deep learning is still in its infancy, but it has shown great energy in the field of application. Since 2011, speech recognition researchers at Microsoft Research and Google have successively used DNN technology to reduce the error rate of speech recognition by 20% to 30%, which is the biggest breakthrough in the field of speech recognition for more than ten years. In 2012, DNN technology achieved amazing results in the field of image recognition, reducing the error rate from 26% to 15% in the ImageNet evaluation. In this year, DNN was also applied to the DrugActivity prediction problem of pharmaceutical companies, and achieved the best results in the world. This important achievement was reported by the New York Times.

Now Google, Microsoft, Baidu and other well-known high-tech companies with big data are scrambling to invest resources and occupy the technical commanding heights of deep learning, precisely because they have seen that in the era of big data, more complex and powerful deep models can profoundly Reveal the complex and rich information carried in massive data, and make more accurate predictions for future or unknown events.

At present, deep learning applications include speech recognition, image recognition, natural language processing, and search advertisement CTR estimation. It is a good choice to design a deep learning system.

Invention content:

The invention provides a data processing system and method, which improves calculation efficiency.

In order to solve the above technical problems, the present invention provides a data processing system, the system includes: a master node, a plurality of slave nodes;

The master node is used to read the data to be processed in batches; it is also used to distribute the data to be processed to each slave node after reading each time, update the network according to the weight returned by each slave node, and update the After the last network information parameter is sent to each described slave node, read the next batch of data to be processed;

The slave node is used to perform forward and backward calculations on the received data distributed by the master node to obtain weights and return them to the master node.

Preferably,

The master node includes two CPUs and a GPU;

The slave node includes two CPUs and two GPUs;

The master node and the slave nodes adopt a hybrid cluster system mode of CPU and GPU heterogeneous architecture.

Preferably,

The system also includes parallel distributed Luster storage:

The master node is used to read the data to be processed in batches specifically refers to:

The master node reads data in parallel from the Luster storage.

Preferably,

The Luster storage supports multi-process or multi-thread parallel reading and writing.

Preferably,

The remote direct data access (RDMA) method is used to receive/send data between the master node and the slave nodes.

Preferably,

The slave node is configured with one IB network card, and the master node and each slave node are interconnected through an IB network;

The master node and the CPU and GPU in each node pass the PCIE3.0 standard.

Preferably,

The number of said slave nodes is not greater than 8.

The present invention also provides a method of data processing, which is applied to the system according to any one of claims 1 to 7, said method comprising:

Step S1, the master node reads the data to be processed and distributes it to each slave node;

Step S2, the master node receives the weights returned by the slave nodes;

Step S3, the master node updates the network according to the weights returned by the slave nodes, and sends the updated network information parameters to the slave nodes.

Step S4, after the master node sends the updated network, it checks whether there is still data to be processed, and if so, returns to S1.

Preferably,

After the step S1, before the step S2, the method also includes:

Step S11, the GPUs of the slave nodes perform forward and backward calculations on the received data distributed by the master node according to the network information parameters to obtain weights;

The S2 includes:

The GPU of the master node receives the weights sent by the slave nodes.

Preferably,

Before the step S1, the method also includes:

Step S0, the master node reads data in parallel from the parallel distributed Luster storage.

The above scheme adopts the master-slave computing mode, which reduces the processing time of deep learning applications and improves computing efficiency.

Description of drawings

Fig. 1 is the structural representation of the data processing system in embodiment one;

Fig. 2 is the flowchart of the data processing method in embodiment one;

Fig. 3 is the structural representation of the data processing system in the second embodiment;

Fig. 4 is the schematic diagram of the connection between the Luster storage and the master node in the second embodiment;

FIG. 5 is a schematic diagram of the logical relationship of the data processing system in the second embodiment.

detailed description

In order to make the purpose, technical solution and advantages of the application clearer, the embodiments of the application will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

Embodiment one

As shown in Figure 1, the present invention provides a kind of data processing system, and described system comprises: a master node 11, a plurality of slave nodes 12;

The master node 11 is used to read the data to be processed in batches; it is also used to distribute the data to be processed to each slave node after each reading, and update the network according to the weight returned by each slave node, and the After the updated network information parameters are sent to each of the slave nodes, the next batch of data to be processed is read;

The slave node 12 is used to perform forward and backward calculations on the received data distributed by the master node to obtain weights and return them to the master node.

In this embodiment, the number of slave nodes may be set to be no greater than 8. The master node 11 includes two CPUs 121 and one GPU 122; the slave node 12 includes two CPUs 121 and two GPUs 122; the master node and each slave node adopt a mixed cluster system mode of CPU and GPU heterogeneous architecture.

Preferably,

The system also includes a parallel distributed Luster storage 13: the Luster storage 13 supports multi-process or multi-thread parallel reading and writing. Master nodes 11 read data in parallel from Luster storage.

The remote direct data access RDMA method is used to receive/send data between the master node and each slave node.

Preferably,

The slave node 12 is configured with one IB network card, and the master node 11 and each slave node 12 are interconnected through the IB network; the master node 11 and the CPU and GPU in each node are connected through the PCIE3.0 standard.

As shown in Figure 2, the present invention also provides a data processing method, which is applied to the system according to any one of claims 1 to 7, the method comprising:

Step S1, the master node reads the data to be processed and distributes it to each slave node;

Step S2, the master node receives the weight returned by each slave node;

Specifically, the GPU of the master node receives the weights sent by each slave node

Step S3, the master node updates the network according to the weight returned by each slave node, and sends the updated network information parameters to each slave node.

Step S4, after the master node sends the updated network, it checks whether there is still data to be processed, and if so, returns to S1.

Preferably,

After the step S1, before the step S2, the method also includes:

Step S11, the GPU of each slave node performs forward and backward calculation on the received data distributed by the master node according to the network information parameters to obtain the weight;

Preferably,

Before the step S1, the method also includes:

Step S0, the master node reads data in parallel from the parallel distributed Luster storage.

Embodiment two

The technical solution of the present invention will be further described below in conjunction with specific scenarios.

As shown in Figure 3, the data processing system of this embodiment can run deep learning caffe applications, using Cifar-10 data testing, specifically can be implemented using the following architecture:

1. The data processing system can adopt the hybrid cluster system mode of CPU+GPU heterogeneous architecture; and adopt the master-slave mode, and the computing nodes of the whole system are divided into 1 master node and 8 slave nodes. According to the characteristics of deep learning application algorithms, parameter update calculations, data reading and distribution, and network update calculations are completed by the master node; time-consuming forward and backward calculations are completed by the slave nodes.

The master node and the slave node in this embodiment are further described in detail below:

a) Master node

The master node is for CPU and GPU collaborative computing, CPU and GPU communication adopts PCIE3.0 standard, 2 CPUs, 1 NvidiaK40GPU, GPU supports PCIE3.0 standard, and the number of master nodes is 1. The master node is equipped with two IB network cards, and the master node is interconnected with the storage and other slave nodes through the IB network.

b) slave node

The slave node is for CPU and GPU collaborative computing. The communication between CPU and GPU adopts the PCIE3.0 standard. There are 2 CPUs and 2 NvidiaK40GPUs. The GPUs support the PCIE3.0 standard. Both GPUs are plugged into the CPU0 slot. The number of slave nodes is 8. The slave node is equipped with an IB network card, and the slave node and the master node are interconnected through the IB network.

2. As shown in Figure 4, in the technical solution of this embodiment, parallel distributed Luster storage needs to be designed to support multi-process or multi-thread parallel read and write, with high parallel read and write bandwidth and low latency. Luster storage and the master node Interconnected through the IB network.

3. Network design. This data processing system adopts 56Gb/sIB high-speed network of Mellanox Company to realize parallel storage, high-speed interconnection of master nodes and slave nodes.

As shown in Figure 5, the working logic relationship of each component of the system is designed as follows:

(1) The master node reads Cifar-10 data in parallel from parallel Luster storage;

(2) The master node distributes the data to 8 slave nodes;

(3) The two GPUs of each slave node start to perform forward and backward calculations, and directly transmit the calculated weights to the master node GPU through RDMA;

(4) After the master node receives the new weight, it calculates on the GPU, updates the network, and then sends the new network RDMA to the slave node;

The above steps are executed iteratively until all data processing is completed, and the logical relationship is shown in FIG. 3 .

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention. Those skilled in the art can understand that all or part of the steps in the above method can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, and the like. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Correspondingly, each module/module in the above embodiments can be implemented in the form of hardware, or can be implemented in the form of software function modules. The form is realized. This application is not limited to any specific form of combination of hardware and software.

Claims (10)

1. A data processing system, characterized in that the system comprises: a master node, a plurality of slave nodes; The master node is used to read the data to be processed in batches; it is also used to distribute the data to be processed to each slave node after reading each time, update the network according to the weight returned by each slave node, and update the After the last network information parameter is sent to each described slave node, read the next batch of data to be processed; The slave node is used to perform forward and backward calculations on the received data distributed by the master node to obtain weights and return them to the master node. 2. The system of claim 1, wherein: The master node includes two CPUs and a GPU; The slave node includes two CPUs and two GPUs; The master node and the slave nodes adopt a hybrid cluster system mode of CPU and GPU heterogeneous architecture. 3. The system according to claim 2, further comprising parallel distributed Luster storage: The master node is used to read the data to be processed in batches specifically refers to: The master node reads data in parallel from the Luster storage. 4. The system of claim 3, wherein: The Luster storage supports multi-process or multi-thread parallel reading and writing. 5. The system of claim 4, wherein: The remote direct data access (RDMA) method is used to receive/send data between the master node and the slave nodes. 6. The system according to any one of claims 1 to 5, characterized in that: The slave node is configured with one IB network card, and the master node and each slave node are interconnected through an IB network; The master node and the CPU and GPU in each node pass the PCIE3.0 standard. 7. The system according to any one of claims 1 to 5, characterized in that: The number of said slave nodes is not greater than 8. 8. A method for data processing, applied to the system according to any one of claims 1 to 7, characterized in that the method comprises: Step S1, the master node reads the data to be processed and distributes it to each slave node; Step S2, the master node receives the weights returned by the slave nodes; Step S3, the master node updates the network according to the weights returned by the slave nodes, and sends the updated network information parameters to the slave nodes. Step S4, after the master node sends the updated network, it checks whether there is still data to be processed, and if so, returns to S1. 9. The method of claim 8, wherein: After the step S1, before the step S2, the method also includes: Step S11, the GPUs of the slave nodes perform forward and backward calculations on the received data distributed by the master node according to the network information parameters to obtain weights; The S2 includes: The GPU of the master node receives the weights sent by the slave nodes. 10. The method according to any one of claims 8 to 9, characterized in that: Before the step S1, the method also includes: Step S0, the master node reads data in parallel from the parallel distributed Luster storage.