CN116089477B - Distributed Training Method and System - Google Patents

Abstract

The present application provides a distributed training method and system. The method includes: creating a cache process corresponding to the first data set in multiple cache nodes of the first cluster; creating a first cache process corresponding to the first computing task among multiple first computing nodes in the first cluster; Calculate the task process, and set the first cache service as the input of the first calculation task, the first computing node belongs to the computing node group; according to the first cache service, the first computing task, and the first computing node, determine whether to expand; if it is determined to expand , create a cache process of the first cache service in the first computing node, so that in the process of training the first computing task, the first computing node reads data from the cache process in the first computing node to complete the first calculation Training of the first computing task process on the node. In this way, the data required for training can be read locally during the training of computing tasks, and the training speed is improved.

Description

Distributed Training Method and System

technical field

The present application relates to the field of terminal equipment, and in particular to a distributed training method and system.

Background technique

At present, a large number of machine learning tasks are trained in the cloud-native container environment, which facilitates efficient use of computing resources. The data sets required for training are usually stored in remote storage services such as obs, nfs, etc., and the training tasks on computing nodes need to read remote data sets.

The training time of the training task includes the reading time and computing time of the training data. When the computing time of the training task is less than the remote data reading time, the data reading speed limits the task training speed, resulting in a slower training speed.

Contents of the invention

In order to solve the above technical problems, this application provides a distributed training method and system, which can automatically expand the cache process corresponding to the data set required by the computing task training to the computing node where the computing task is located, so that the computing task training process The data required for training can be read locally to improve the training speed.

In a first aspect, the present application provides a distributed training method, the method includes: creating a cache process of the first cache service corresponding to the first data set in multiple cache nodes of the first cluster, the first data set is located in the first A data set in a remote database outside the cluster; creating a first computing task process corresponding to the first computing task in multiple first computing nodes of the first cluster, and setting the first cache service as the input of the first computing task , the first computing node belongs to the computing node group; according to the first cache service, the first computing task, and the first computing node, determine whether to expand the capacity; if it is determined to expand the capacity, create a cache process of the first cache service in the first computing node to In the process of training the first computing task, the first computing node reads data from the cache process in the first computing node to complete the training of the first computing task process on the first computing node. In this way, the cache process corresponding to the data set required for computing task training can be automatically expanded to the computing node where the computing task is located, so that the data required for training can be read locally during the computing task training process, and the training speed is improved.

According to the first aspect, according to the first cache service, the first computing task, and the first computing node, determine whether to expand capacity, including: the acquired first feature data corresponding to the first cache service, and the second feature data corresponding to the first computing task . Calculate the third characteristic data corresponding to the node group; extract the first characteristic vector from the first characteristic data, extract the second characteristic vector from the second characteristic data, and extract the third characteristic vector from the third characteristic data; according to the first The eigenvector, the second eigenvector, and the third eigenvector are used to obtain the combined eigenvector; the combined eigenvector is input into the trained capacity expansion decision model, and the capacity expansion decision model outputs the decision result of whether to expand capacity.

According to the first aspect, the capacity expansion decision model is a classification model.

According to the first aspect, the first feature data includes statistical information, cache setting information and cache application information of the first data set.

According to the first aspect, the statistical information of the first data set includes the total file size, the total number of files, and the file format of the first data set; the cache setting information of the first data set includes cache capacity, cache media, and the number of cache processes; the first The cached application information of the data set includes the number of cached computing tasks to which the first data set is applied, and historical information of the cached computing tasks to which the first data set is applied.

According to the first aspect, the second characteristic data includes any one or more of the following data: task priority, user information, central processing unit CPU resources applied for, graphics processing unit GPU resources applied for, memory resources applied for, used Input data information, corresponding algorithm type, and historical execution information.

According to the first aspect, the third feature data includes any one or more of the following data: free CPU information, GPU information, memory information, and solid-state disk information that can be allocated by each computing node, and CPU information that has been allocated by each computing node , GPU information, memory information, SSD information, and the network topology of each computing node.

According to the first aspect, creating a caching process of the first caching service corresponding to the first data set among multiple caching nodes of the first cluster includes: receiving a first caching service creation request; obtaining the data volume of the first data set; if The data volume of the first data set is less than the data volume threshold, set the cache capacity of the cache process of the first cache service equal to the data volume of the first data set; set the cache initialization tag and cache for the first cache service resource corresponding to the first data set service label; send a first instruction to the first cluster, the first instruction carries the first cache service resource; according to the first instruction, create the first cache node corresponding to the first data set in the multiple cache nodes with the cache initialization label in the first cluster A caching process of the caching service; the data in the first data set is loaded into the caching process.

According to the first aspect, determining whether to expand capacity according to the first cache service, the first computing task, and the first computing node includes: if the available storage resource of the first computing node is greater than the data volume of the first data set, determining to expand the capacity.

According to the first aspect, determining whether to expand capacity according to the first cache service, the first computing task, and the first computing node includes: acquiring the priority of the first computing task; if the priority of the first computing task is higher than a preset level, Confirm expansion.

According to the first aspect, determining whether to expand capacity according to the first cache service, the first computing task, and the first computing node includes: obtaining the historical training speed of the algorithm of the first computing task; if the historical training speed of the algorithm of the first computing task If it is less than the preset speed value, it is determined to expand the capacity.

According to the first aspect, each cache process stores all data of the first data set.

In a second aspect, the present application provides a distributed training system. The system includes a control node and a first cluster, wherein: the control node is configured to: create the first data set corresponding to the first data set among multiple cache nodes in the first cluster. A caching process of a caching service, the first data set is a data set located in a remote database outside the first cluster; a first computing task process corresponding to the first computing task is created in multiple first computing nodes of the first cluster , and set the first cache service as the input of the first computing task, the first computing node belongs to the computing node group; according to the first cache service, the first computing task, and the first computing node, determine whether to expand the capacity; Create a cache process of the first cache service in a computing node; the first computing node in the first cluster is used to read data from the cache process in the first computing node during the training of the first computing task to complete Training of the first computing task process on the first computing node.

In a third aspect, the present application provides an electronic device, including: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes any one of the first aspects distributed training method.

In a fourth aspect, the present application provides a computer-readable storage medium, including a computer program. When the computer program runs on an electronic device, the electronic device executes the distributed training method of any one of the foregoing first aspects.

Description of drawings

FIG. 1 is an exemplary diagram showing an example of a system structure related to distributed training;

FIG. 2 is an exemplary diagram showing an example of the deployment of a distributed cache service related to distributed training;

Fig. 3 is an exemplary diagram showing a process of creating a cache service;

FIG. 4 is an exemplary diagram showing an application process of a cache service;

Fig. 5 is an exemplary diagram showing a cache service expansion process;

FIG. 6 is a schematic diagram of the process of obtaining the result of capacity expansion from the input data required by the capacity expansion decision-making model;

Fig. 7 is an exemplary diagram showing the expansion situation and reading method of different data sets.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.

The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than describing a specific order of the target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

FIG. 1 is an exemplary diagram showing a system structure related to distributed training. Referring to FIG. 1 , in this embodiment, a system related to distributed training may include a cluster 1, a remote server, and a control node. Wherein, the management service is deployed in the control node, and the management service is an application on the control node device. Wherein, there are data sets stored in the remote server. For the convenience of description, these data sets stored in the remote server are referred to as remote data sets in this paper.

Among them, cluster 1 is located in the cloud native environment, that is, the k8s machine environment. Both the remote server and the control node are k8s external machines.

It should be noted that although only one data set is illustrated in the remote server in FIG. 1 , it should be understood that each remote server can store multiple data sets, not limited to one.

Similarly, although only one remote server is illustrated in FIG. 1 , it should be understood that there may be multiple remote servers in a system related to distributed training.

It should be noted that the control node may be located in the cluster 1 or outside the cluster 1.

It can be understood that the management service can be deployed on multiple nodes, and each node on which the management service is deployed can serve as a control node. Management services can follow the microservices architecture.

Please continue to refer to FIG. 1 , cluster 1 includes a computing node group, a cache initialization node group, and other node groups. Wherein, the computing node group may include multiple computing nodes, and the cache initialization node group may include multiple cache nodes. These node groups are pre-divided by the administrator. Among them, the nodes in the computing node group are called computing nodes, and each computing node is set with a GPU label; the nodes in the cache initialization node group are called cache nodes, and each cache node is set with a cpu label, and the cpu is called cache initialization Label. Therefore, according to whether the label of the node is GPU or cpu, it can be distinguished whether it is a computing node or a cache node. It should be noted that the cpu label indicates that the node with this label belongs to the cache initialization node group, and the label GPU indicates that the node with this label belongs to the computing node group. At this time, cpu and GPU are only label names. In other embodiments, the nodes in the cache initialization node group may also use labels other than cpu, and the nodes in the computing node group may also use labels other than GPU.

Wherein, each computing node is provided with a cache client daemon process. The cache client daemon is used to detect that the node is set with the cache client label client-id, and create a cache client on the node with the cache client label client-id set.

Wherein, other node groups are provided with api interface services, controllers, schedulers and databases. Devices outside cluster 1, such as control nodes, can call controllers, schedulers, etc. in the cloud native environment (that is, k8s machine environment) through API interface services to control nodes in cluster 1 to perform corresponding operations. Therefore, the management service in the control node can control the nodes in the cluster 1 by sending instructions or information to the API interface services in other node groups in the cluster 1 .

FIG. 2 is an exemplary diagram showing an example of deployment of a distributed cache service related to distributed training. The process of distributed training will be described below with reference to FIG. 1 , FIG. 2 and subsequent FIG. 3 , FIG. 4 and FIG. 5 .

First, the creation process of the distributed cache service is described.

Please refer to Figure 1, where process 1 represents creating a cache service. Process 1 includes the following steps:

1.1. The management service in the control node sends an instruction 1 for creating a cache service resource to the api interface service. The instruction 1 carries the cache service resource. The cache service resource is set with the cache initialization tag cpu and the cache service tag cache-id.

Wherein, each cache service corresponds to a cache-id, and cache-ids corresponding to different cache services are different. Wherein, the cache initialization label cpu is used to indicate that the cache service is created on the nodes in the cache initialization node group. The scheduler and controller in the cluster determine to create the cache service on the node with the same label (that is, the cache initialization label cpu) according to the cache initialization label cpu. The cache service label cache-id is used to indicate which cache service to create.

1.2, the api interface service calls the controller, scheduler, etc. to create a cache process worker for the cache service in multiple cache nodes in the cache initialization node group according to instruction 1.

It should be noted that although FIG. 1 only shows a cache node that has created a cache process worker for a cache service, it should be understood that, in an application, cache process workers may be created in multiple cache nodes.

Among them, which cache nodes in the cache initialization node group to create the cache process worker is determined by the schedulers in other node groups according to the preset scheduling policy. After the cache process worker is created in the cache node, the controller sets a cache-id label for the cache node that has created the cache process worker, to indicate that the node has created a cache service with the label cache-id. A cache node without a cache process worker has the cache initialization tag cpu but no cache-id tag.

1.3, preload the original data of dataset 1 in the remote server into the cache process worker of the cache node.

In this way, data set 1 is cached in the cache process worker of the cache node. During the process of training computing tasks, the computing nodes in cluster 1 do not need to read data from the remote server, but read data from the cache process worker of the cache node. read data.

For the detailed process of 1.1 to 1.3, please refer to the process shown in Figure 3.

Fig. 3 is an exemplary diagram showing a process of creating a cache service. Referring to Fig. 3, in this embodiment, the cache service creation process may include the following steps:

S301. The management service in the control node receives a cache service creation request.

In the application, the user can send a cache service creation request to the management service by clicking the option to create a cache service in the management service.

S302. The management service acquires relevant information of the target data set in response to the user's operation of selecting the target data set, where the relevant information of the target data set includes the data volume of the target data set.

In this embodiment, the target data set is the data set 1 in FIG. 1 .

Users can select the target dataset from the current dataset list. Wherein, the data set list may include all data sets in each remote server currently connected to the control node. Moreover, the database of the management service of the control node stores information of all data sets of the remote server, and the information of the data sets includes information such as the data volume of the data sets. In one example, the data set information stored in the database of the management service may be pre-input by the user.

for example. Suppose the data set list includes data set 1, data set 2, data set 3, data set 4, and data set 5 data sets. The data set in the remote server in Figure 2 is data set 1. When the user is in the data set list If dataset 1 is selected, then dataset 1 is the target dataset, and the management service obtains relevant information of dataset 1 from the database, and the relevant information includes the data volume of dataset 1.

S303. The management service judges whether the data volume of the target data set is smaller than the data volume threshold, if yes, execute step S305, otherwise execute step S304.

In one example, the database threshold can be set according to the hard disk storage capacity of the node. For example, the data volume threshold can be set equal to 60% of the storage capacity of the hard disk.

In this embodiment, the management service sets the cache capacity of a single cache process worker in the cache service according to the data volume of the target data set and the data volume threshold.

A cache service can include one or more cache process workers. The number of cache process workers included in the cache service can be set according to the data volume and data volume threshold of the dataset.

S304. Set the cache capacity of a single cache process worker equal to the data volume threshold, disable flexible scheduling, and execute step S306.

When the data volume of the target data set is greater than the data volume threshold, it indicates that the data set is too large. At this time, use multiple cache process workers to cache all the original data of an entire target dataset, set the cache capacity of a single cache process worker equal to the data volume threshold, and the number of cache process workers equal to the data volume of the target dataset divided by the data volume threshold earned quotient. If it is not divisible, the number of cached process workers is obtained by rounding up.

When the data volume of the target dataset is greater than the data volume threshold, elastic scheduling does not need to be enabled to cache the data in the dataset locally on the computing node, so elastic scheduling is disabled. The management service can set a status flag bit for each cache process worker, and the status flag bit is used to indicate whether elastic scheduling is enabled. For example, in one example, if the status flag of the cache process worker is 1, it indicates that the flexible scheduling of the cache process worker is enabled, and the cache process worker can be expanded to the computing node; if the status flag of the cache process worker is 0, indicating that the elastic scheduling of the cache process worker is disabled, and the cache process worker cannot be expanded to compute nodes.

S305. Set the cache capacity of a single cache process worker equal to the data volume of the target data set, and enable elastic scheduling.

When the data volume of the target data set is less than or equal to the data volume threshold, use a cache process worker to cache all the original data of the entire target data set, and set the cache capacity of a single cache process worker equal to the data volume of the target data set. In one example, taking resource utilization and data reading speed into consideration, the number of initialization cache process workers can be set to 2. In this way, it is possible to resist the impact of a possible single point of failure and save storage resources to the greatest extent.

Assuming that the data volume of Dataset 1 in Figure 2 is less than the data volume threshold, then set two cache process workers for Dataset 1, and set the cache capacity of a single cache process worker equal to the data volume of Dataset 1, and start the elastic scheduling cache process worker In this way, the cache process worker of dataset 1 can be flexibly scheduled.

S306. Set an affinity tag for the cache service resource.

The affinity tags in this step include the cache service tag cache-id and the cache initialization tag cpu. The cache service tag cache-id can control the creation of the cache process worker, and the cache initialization tag can control the cache process worker initialization scheduling to the cache initialization node group. . You can set the affinity weight of the initialization label to be lower than that of the cache service label cache-id to ensure that nodes with the cache service label cache-id are dispatched preferentially.

Wherein, steps S301 to S306 are completed in the management service before step 1.1 in FIG. 1 . After step S306, the management service executes step 1.1 in FIG. 1 .

After step S306, the management service generates instruction 1 in step 1.1 in FIG. 1, and then executes step 1.1. In this way, cluster 1 receives instruction 1.

S307. Create a cache process worker.

In this step, the k8s scheduler and controller in cluster 1 create a cache process worker according to instruction 1. Among them, the scheduler is used for node scheduling (that is, to determine which nodes to create cache services), and the controller is used to create a cache process worker on the corresponding node according to the scheduling result (that is, the nodes scheduled by the scheduler).

Please refer to Figure 2, assuming that the result of this creation is that the cache process worker1 and the cache process worker2 are respectively created on the two cache nodes (that is, cache node 1 and cache node 2 in Figure 2).

S308. Start a preloading task, and load the original data of the target data set to the cache process worker.

The management service timing task queries k8s to see if the cache process is created. If completed, k8s starts the data preloading task.

Through the embodiment shown in FIG. 3 , the user can create multiple cache services respectively, and after the creation is completed, these cache services can be displayed in the cache service list. A cache service list may be displayed on the interface for creating a computing task, so that the user may select a cache service from the cache service list when creating a computing task.

Then, the creation process of the computing task (corresponding to the application process of the cache service) is described.

Please continue to refer to FIG. 1 , where process 2 represents creating a computing task. Process 2 may include the following steps:

2.1. The management service in the control node sends an instruction 2 for creating computing task resources to the api interface service, and the instruction 2 carries cache service information and computing task resources.

Wherein, the cache service information may include a cache service tag cache-id.

2.2. Create a computing task process on the computing node.

Among them, which computing nodes in the computing node group to create computing task processes are determined by schedulers in other node groups according to resource conditions of each computing node and a preset scheduling strategy.

2.3. Set the cache client label client-id for the computing node where the computing task process is located.

The role of the client tag is to control client creation. When a node is labeled as a client, the cache client daemon in the management service will create a corresponding cache client on the node.

2.4. The cache client daemon process on the computing node with the cache client label client-id is set. After detecting that the cache client label client-id is set on this computing node, create a cache client on this computing node.

The cache client is used to read data from the cache process worker.

Each cache client corresponds to all cache process workers of a cache service, that is, each cache client can read data from all cache process workers of a cache service.

Please refer to Figure 4 for the procedures corresponding to 2.1 to 2.4.

Fig. 4 is an exemplary diagram of a caching service application process. Referring to FIG. 4, in this embodiment, the cache service application process may include the following steps:

S401. The management service configures the cache service as an input of the calculation task according to the user's operation of selecting the cache service when creating the calculation task.

When the cache service is available, the user can select the cache service as input when creating a computing task.

For example, assuming that there are currently three cache services available: cache service 1, cache service 2, and cache service 3, when the user creates computing task X, all three cache services can be selected. Assuming that the user selects cache service 1, then the management service configures cache service 1 as the input of computing task X.

It is assumed that cache service 1 is the cache service created in the embodiment shown in FIG. 3 .

S402. After the management service submits the computing task to the api interface service, the scheduler schedules the computing task to the computing node, that is, creates a computing task process on the computing node.

The management service sends an instruction 2 to create a computing task resource to the api interface service, which means that the management service submits a computing task to the api interface service.

Please refer to Figure 2, assuming that this calculation task is scheduled to two computing nodes, that is, computing node 1 and computing node 2 in Figure 2 . Computing process 1 in computing node 1 and computing process 2 in computing node 2 run distributed tasks of the computing task. That is, this calculation task is distributed to computing node 1 and computing node 2.

S403. The controller in the cluster 1 regularly detects the usage of the cache service resource, and if there is a computing node using the cache service, label the node with a cache client tag client-id corresponding to the cache service.

The database inside cluster 1 stores the usage of cache resources. The content of the cache resource usage may be: which cache service is used by the computing container. Then, the controllers of other node groups in the cluster 1 mark the cache client tag client-id on the nodes where these computing containers are located according to the detected usage of the cache service resources.

In this way, the computing nodes with computing tasks in Figure 1 are set with the label GPU and the cache client label client-id.

S404. After the client daemon perceives that the computing node has a cache client label, it creates a cache client on the computing node.

S405. The computing task reads the data in the cache process worker corresponding to the cache service through the cache client.

Then, the expansion process of the cache service is described.

Please continue to refer to FIG. 1 , where process 3 represents the capacity expansion process. Process 3 may include the following steps:

3.1, management service decision expansion, set the cache service label cache-id for the computing node, and update the number of corresponding cache process workers.

In this way, in FIG. 1 , the computing nodes with computing tasks and cache services are set with the label GPU, the cache service label cache-id, and the cache client label client-id.

3.2, expand the cache service to the computing node, that is, create a computing task process on the computing node.

3.3. Read data from the cache process worker in the cache node and save it to the cache process worker in the computing node.

Please refer to Figure 5 for the processes corresponding to 3.1 to 3.3.

Fig. 5 is an exemplary diagram showing a caching service expansion process. Please refer to FIG. 5. In this embodiment, the caching service expansion process may include the following steps:

S501. When elastic scheduling is enabled, the management service detects that there is no local cache process worker on the computing node using the cache service, and triggers an elastic scheduling task. The computing task in the computing node is computing task a.

In this paper, computing tasks may also be referred to as training tasks, and these tasks are tasks for training using data in a dataset.

S502. Obtain input data required by the capacity expansion decision model, input the input data into the capacity expansion decision model, and obtain a decision result of whether to expand capacity for computing task a.

The input data required for the capacity expansion decision-making model can be determined according to the specific model. The expansion decision model can infer whether to expand the current capacity based on historical data.

The capacity expansion decision model can be a binary classification model, and the model can be trained by collecting data from different scenarios based on manual experience to form a training data set. For example, the capacity expansion decision model may be a decision tree, logistic regression, svm and other models. The capacity expansion decision model is a classification model in the field of machine learning. For example, the capacity expansion decision model can be an LGBM model. Of course, it is not limited to this model, and other models can also be used as the capacity expansion decision model.

Among them, the input data required for the capacity expansion decision-making model may include the following data:

(1) Cache service characteristic data related to computing tasks, including but not limited to:

Statistics of the original dataset: total file size, total number of files, and file format.

Cache setting information: cache capacity, cache media (ram, ssd), number of cache workers; cache setting information can also be called cache node details.

Cache application information: the number of computing tasks cached by the application native data set, and the history information of the cached computing tasks of the application native data set. Caching application information may also be referred to as usage task details.

(2) Computing task characteristic data related to computing tasks, including but not limited to:

Task priority, user information, application for cpu resources, application for gpu resources, application for memory resources, input data information used, corresponding algorithm type, historical execution information.

(3) Computing node group characteristic data related to computing tasks, including but not limited to:

Each computing node can allocate idle cpu, gpu, memory, solid state disk and other information, each computing node has allocated cpu, gpu, memory, solid state disk and other information, and the network topology structure of each computing node. See Figure 6 for the process of obtaining the result of capacity expansion from the input data required by the capacity expansion decision model. FIG. 6 is a schematic diagram illustrating a process of obtaining a result of whether to expand capacity from the input data required by the capacity expansion decision-making model.

Please refer to Figure 6, after obtaining the above feature data, extract the cache service feature vector from the cache service feature data, extract the computing task feature vector from the computing task feature data, and extract the computing node group feature vector from the computing node group feature data;

Then, the cache service feature vector, computing task feature vector, and computing node group feature vector are combined into a combined feature vector, and the combined feature vector is input into the trained expansion decision model, and the expansion decision model outputs the decision result of whether to expand.

Among them, the expansion decision model is a trained model.

The training process of the expansion decision model may include the following steps:

Construct the first classification model and set initial parameter values;

Obtain several sets of sample data, each set of sample data includes: combined feature vector samples and corresponding decision result label data;

Several sets of sample data are used to train the first classification model to obtain a trained first classification model, and the trained first classification model is used as a trained expansion decision model.

Wherein, the acquisition process of the combined feature vector samples in the sample data is consistent with the aforementioned process of obtaining the combined feature vector from the input data required by the capacity expansion decision model, and will not be repeated here.

Wherein, the process of using several sets of sample data to train the first classification model to obtain the trained first classification model may be:

Determining the first classification model obtained after training the previous set of sample data as the initial classification model corresponding to this set of sample data;

Input the combined feature vector samples in this group of sample data into the initial classification model to obtain the decision result output by the initial classification model, which is recorded as the output decision result;

According to the difference between the output decision result and the decision result label data in this group of sample data, adjust the parameter value in the initial classification model, and use the classification model after adjusting the parameter value this time as the first one obtained after training this group of sample data. classification model;

Judging whether the convergence condition of the training is met, if so, stop the training, and use the first classification model obtained after the training of this set of sample data as the trained expansion decision model; otherwise, continue to execute the training of the next set of sample data.

Wherein, the first classification model corresponding to the first group of sample data is the constructed first classification model with initial parameter values.

Of course, the above is only an exemplary description of the training methods, and is not intended to limit this embodiment, and this embodiment may not be limited to the training methods listed above.

In this embodiment, decision accuracy can be improved by using a classification model in the field of machine learning to decide whether to expand the cache process worker to a computing node.

Of course, in addition to using a classification model to determine whether to expand, other methods may also be used to determine whether to expand. This embodiment does not limit how to determine whether to expand. For example, the expansion decision-making method based on preset rules.

In an example, the preset rule may be: if the available storage resource of the computing node is greater than the data volume of the dataset, expand the capacity; otherwise, if the available storage resource of the computing node is less than or equal to the data volume of the dataset, no expansion will be performed.

In an example, the preset rule may be: expand the capacity if the priority of the computing task is higher than the preset level; otherwise, do not expand the capacity if the priority of the computing task is lower than or equal to the preset level.

In an example, the preset rule can be: if the historical training speed of the algorithm of the computing task is less than the preset speed value, expand the capacity; otherwise, if the historical training speed of the algorithm of the computing task is greater than or equal to the preset speed value, the capacity will not be expanded .

S503. Determine whether the decision result is expansion, if yes, execute step S504, otherwise execute step S507.

S504. Apply the cache service tag cache-id to all computing nodes related to the computing task a, count the number n of all nodes with the cache service tag cache-id, set the number of cache process workers to be expanded to be equal to the number n of nodes, and update cloud native resources.

The way to update cloud-native resources is to submit a request to the cloud-native service to update the number of cache process workers.

The process of steps S501 to S504 corresponds to the combined process of deciding whether to expand capacity and step 3.1 in FIG. 1 .

S505. The scheduler creates a cache process worker on the computing node with the cache service label cache-id according to the cache service label cache-id.

Step S505 corresponds to step 3.2 in FIG. 1 .

Please refer to Figure 2. As a result of this creation, a cache process worker3 and a cache process worker4 are created on two computing nodes (that is, computing node 1 and computing node 2 in Figure 2).

S506. The cache client of the computing process in the computing node reads data from the caching process worker of the caching node, and caches the read data in the local caching process worker of the computing node, and ends.

Step S506 corresponds to step 3.3 in FIG. 1 .

In this way, subsequent computing nodes can directly read dataset data from the local cache process worker. Taking Figure 2 as an example, the calculation process 1 can directly read data from the local cache process worker3 during the training process. Similarly, during the training process, computing process 2 can directly read data from the local cache process worker4. In this way, by elastically scheduling the cache service to the computing nodes, the computing tasks in the computing nodes can directly read data from the local during the distributed training process, which improves the data reading speed, thereby improving the distributed training of computing tasks. speed.

S507. The cache client of the calculation process in the calculation node reads data from the cache process worker of the cache node, and ends.

The following is a comparison to illustrate the difference in the speed at which computing tasks read data between expansion and no expansion.

Fig. 7 is an exemplary diagram showing the expansion situation and reading method of different data sets. Please refer to Figure 7. In Dataset 1, Dataset 2, Dataset 3, and Dataset 4, Dataset 1 expands the cache service worker to the computing node. Therefore, the computing tasks using Dataset 1 can reach the computing node to read data locally. The effect, the training speed is fast. Dataset 2 and Dataset 3 are not expanded, and the data of other nodes in the cluster is read. Therefore, the training speed of computing tasks using Dataset 2 and Dataset 3 is slower.

It can be seen that after expansion, the training speed is fast because the computing task can read data locally on the computing node.

The distributed training method of this embodiment can automatically expand the cache process corresponding to the data set required by the computing task training to the computing node where the computing task is located, so that the data required for training can be read locally during the computing task training process , to increase the training speed.

Especially when the computing node memory resources are insufficient and disk resources need to be used to cache data, this embodiment can adaptively expand the cache process without changing the cloud-native scheduler.

In addition, the distributed training method of this embodiment can read the data required for training locally on the expanded computing node without accessing the remote storage service, and can also relieve the pressure on the remote storage service, avoiding a large number of computing tasks to read the remote storage service The problem that causes the performance of the remote storage service to degrade.

Furthermore, the distributed training method of this embodiment, since the data required for training can be read locally on the expanded computing node, there is no need to access remote storage services, and the occupation of communication bandwidth is reduced, communication resources are saved, and large model When performing distributed training, because the parameter exchange of different nodes requires a large amount of bandwidth resources, reading data remotely will occupy a certain amount of bandwidth resources and reduce the performance of parameter exchange.

This embodiment also provides a distributed training system, the system includes a control node and a first cluster, wherein:

Control node for:

Create a cache process of the first cache service corresponding to the first data set in multiple cache nodes of the first cluster, where the first data set is a data set located in a remote database outside the first cluster;

Create a first computing task process corresponding to the first computing task in a plurality of first computing nodes in the first cluster, and set the first cache service as an input of the first computing task, and the first computing node belongs to the computing node group;

Determine whether to expand capacity according to the first cache service, the first computing task, and the first computing node;

If it is determined to expand the capacity, create a cache process of the first cache service in the first computing node;

The first computing node in the first cluster is used to read data from the cache process in the first computing node during the training process of the first computing node to complete the training of the first computing task process on the first computing node .

For the first cluster, please refer to the aforementioned cluster 1, that is, cluster 1 in FIG. 1 .

The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory is coupled to the processor, the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device described above Execution of the distributed training method.

It can be understood that, in order to realize the above functions, the electronic device includes hardware and/or software modules corresponding to each function. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

This embodiment also provides a computer storage medium, in which computer instructions are stored, and when the computer instructions are run on the electronic device, the electronic device is made to execute the above-mentioned related method steps to realize the distributed training method in the above-mentioned embodiment .

This embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the above-mentioned related steps, so as to realize the distributed training method in the above-mentioned embodiment.

In addition, the embodiment of the present application also provides a device, which may specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein, the memory is used to store instructions executed by a computer, and when the device is running, the processing The device can execute the computer-executed instructions stored in the memory, so that the chip executes the distributed training method in the above method embodiments.

Wherein, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can refer to the corresponding method provided above The beneficial effects in the method will not be repeated here.

Through the description of the above embodiments, those skilled in the art can understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned by different Completion of functional modules means that the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or It may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

A unit described as a separate component may or may not be physically separated, and a component shown as a unit may be one physical unit or multiple physical units, which may be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above contents is within the scope of the present application.

If an integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (random access memory, RAM), magnetic disk or optical disk, and other media capable of storing program codes.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

The steps of the methods or algorithms described in connection with the disclosure of the embodiments of the present application may be implemented in the form of hardware, or may be implemented in the form of a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read only memory (Read Only Memory, ROM), erasable programmable read-only memory ( Erasable Programmable ROM, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically EPROM, EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and storage medium can be located in the ASIC.

Those skilled in the art should be aware that, in the foregoing one or more examples, the functions described in the embodiments of the present application may be implemented by hardware, software, firmware or any combination thereof. When implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of this application, without departing from the purpose of this application and the scope of protection of the claims, many forms can also be made, all of which belong to the protection of this application.

Claims (15)

1. A distributed training method, the method comprising:

Creating a cache process of a first cache service corresponding to a first data set in a plurality of cache nodes of a first cluster, wherein the first data set is a data set in a remote database outside the first cluster;

creating a first computing task process corresponding to a first computing task in a plurality of first computing nodes of the first cluster, and setting the first cache service as input of the first computing task, wherein the first computing nodes belong to a computing node group;

determining whether to expand the capacity according to the first cache service, the first computing task and the first computing node;

if the capacity expansion is determined, a cache process of the first cache service is created in the first computing node, so that in the process of training the first computing task, the first computing node reads data from the cache process in the first computing node to complete training of the first computing task process on the first computing node.

2. The method of claim 1, wherein determining whether to expand based on the first cache service, the first computing task, the first computing node, comprises:

The first characteristic data corresponding to the first cache service, the second characteristic data corresponding to the first computing task and the third characteristic data corresponding to the computing node group are obtained;

extracting a first feature vector from the first feature data, extracting a second feature vector from the second feature data, and extracting a third feature vector from the third feature data;

obtaining a combined feature vector according to the first feature vector, the second feature vector and the third feature vector;

and inputting the combined feature vector into a trained capacity expansion decision model, and outputting a decision result of whether capacity expansion is performed or not by the capacity expansion decision model.

3. The method of claim 2, wherein the capacity expansion decision model is a classification model.

4. The method of claim 2, wherein the first characteristic data comprises statistics, cache setup information, and cache application information for the first data set.

5. The method of claim 4, wherein the statistics of the first data set include a total file size, a total number of files, a file format of the first data set; the cache setting information of the first data set comprises cache capacity, cache media and cache process quantity; the cache application information of the first data set comprises the number of computing tasks of the cache of the first data set and the historical information of the computing tasks of the cache of the first data set.

6. The method of claim 2, wherein the second characteristic data comprises any one or more of the following:

task priority, user information, applied CPU resources, applied GPU resources, applied memory resources, used input data information, corresponding algorithm types and historical execution information.

7. The method of claim 2, wherein the third characteristic data comprises any one or more of the following:

the CPU information, the GPU information, the memory information and the solid state disk information which can be allocated by each computing node are allocated, and the network topology structure of each computing node is located.

8. The method of claim 1, wherein creating a caching process for the first caching service corresponding to the first data set in the plurality of caching nodes of the first cluster comprises:

receiving a first cache service creation request;

acquiring the data volume of a first data set;

if the data volume of the first data set is smaller than a data volume threshold value, setting the cache capacity of a cache process of a first cache service to be equal to the data volume of the first data set;

Setting a cache initialization tag and a cache service tag for a first cache service resource corresponding to the first data set;

sending a first instruction to the first cluster, wherein the first instruction carries the first cache service resource;

according to the first instruction, a cache process of a first cache service corresponding to the first data set is created in a plurality of cache nodes with the cache initialization tag in the first cluster;

and loading the data in the first data set into the caching process.

9. The method of claim 1, wherein determining whether to expand based on the first cache service, the first computing task, the first computing node, comprises:

and if the available storage resources of the first computing node are larger than the data volume of the first data set, determining expansion.

10. The method of claim 1, wherein determining whether to expand based on the first cache service, the first computing task, the first computing node, comprises:

acquiring the priority of the first computing task;

and if the priority of the first computing task is higher than a preset level, determining expansion.

11. The method of claim 1, wherein determining whether to expand based on the first cache service, the first computing task, the first computing node, comprises:

acquiring the historical training speed of an algorithm of the first computing task;

and if the historical training speed of the algorithm of the first computing task is smaller than a preset speed value, determining expansion.

12. The method of claim 1, wherein each of the caching processes stores all of the data of the first data set.

13. A distributed training system, the system comprising a control node and a first cluster, wherein:

the control node is configured to:

creating a cache process of a first cache service corresponding to a first data set in a plurality of cache nodes of the first cluster, wherein the first data set is a data set in a remote database outside the first cluster;

creating a first computing task process corresponding to a first computing task in a plurality of first computing nodes of the first cluster, and setting the first cache service as input of the first computing task, wherein the first computing nodes belong to a computing node group;

Determining whether to expand the capacity according to the first cache service, the first computing task and the first computing node;

if the capacity expansion is determined, a cache process of the first cache service is created in the first computing node;

the first computing node in the first cluster is configured to read data from the cache process in the first computing node during training of the first computing task, so as to complete training of a first computing task process on the first computing node.

14. An electronic device, comprising:

a memory and a processor, the memory coupled with the processor;

the memory stores program instructions that, when executed by the processor, cause the electronic device to perform the distributed training method of any of claims 1 to 12.

15. A computer readable storage medium comprising a computer program, characterized in that the computer program, when run on an electronic device, causes the electronic device to perform the distributed training method of any of claims 1 to 12.