CN111796932A - A GPU resource scheduling method - Google Patents

Abstract

The invention relates to the technical field of communication application, and discloses a GPU resource scheduling method, which comprises the following steps: s1, collecting basic information of the GPU from the cluster, providing a GPU-usages interface, and entering the step S2; s2, creating GPU application, sending an application request to a Kubernetes scheduler, and entering the step S3; s3, the Kubernetes dispatcher traverses all GPU applications in the cluster after receiving the application request, and the step S4 is entered; s4, calculating a GPU meeting the scheduling requirement of the application through a GPU-usages interface, and entering the step S5; and S5, the GPU manager binds the appointed GPU resources into the application according to the machine where the GPU is located on the application. The sharing of a single GPU in a plurality of applications according to GPU video memory and GPU computing power percentages is realized, the utilization efficiency of the single GPU is greatly improved, and the cost of GPU application is reduced.

Description

A GPU resource scheduling method

technical field

The present invention relates to the technical field of communication applications, and in particular, to a method for scheduling GPU resources.

Background technique

With the explosive growth of device performance and the gradual popularization of virtualization technology, how to realize dynamic resource allocation, flexible scheduling, and improve resource utilization of multiple virtualized devices on existing physical devices to meet the needs of users in daily The demands at work are imminent.

The use of Kubernetes to manage enterprise server clusters greatly reduces the operation and maintenance costs of enterprises and improves the utilization of resources, but so far, the resource management of Kubernetes for each machine is mainly the management of CPU, memory, storage and other hardware . As more and more enterprises use GPUs for machine learning model training and online services, efficient management of GPU resources is becoming more and more important.

Defects of the prior art: The GPU resources are allocated in units of physical GPU cards, and multiple applications cannot share GPU resources, which will lead to the fact that even if a single application does not fully use the allocated computing resources, it cannot be exclusively used. The resources are allocated to other applications, so that the GPU resources cannot be fully utilized.

SUMMARY OF THE INVENTION

The main purpose of the present invention is to provide a GPU resource scheduling method to solve the problem that the GPU resources cannot be fully utilized by a single application at present.

In order to achieve the above object, the present invention provides the following technologies:

A GPU resource scheduling method, comprising the steps of:

S1. First, collect the basic information of the GPU from the cluster, and provide the gpu-usages interface, and enter step S2;

S2. Create a GPU application, send an application request to the Kubernetes scheduler, and go to step S3;

S3. After receiving the application request, the Kubernetes scheduler will traverse all GPU applications in the cluster, and enter step S4;

S4. Calculate the GPU that meets the scheduling requirements of the application through the gpu-usages interface, and enter step S5;

S5. The GPU manager binds the specified GPU resources to the application according to the machine where the GPU on the application is located.

Further, in step S2, in the process of creating the GPU application, the application provides the required video memory value and computing power value.

Further, in step S1, the basic information of the collected GPU includes the model of the GPU, the video memory and the GPU core.

Further, in step S4, if there is no GPU in the cluster that meets the scheduling requirement of the application, then proceed to step S6, S6, isolation of GPU resources.

Further, S6 includes steps S60 and S61. In S60, it is found that the video memory required by the application program exceeds the preset value, or is greater than the video memory value of all GPUs in the cluster, then returns the video memory allocation failure; S61, wraps the execution thread, periodically Check the core usage rate of the GPU by the program. If it exceeds the set core usage value, or is greater than the memory value of all GPUs in the cluster, the current execution thread is assigned to the waiting execution thread.

Further, in step S2, in the process of creating the GPU application, the model of the GPU and the number of GPUs required by the GPU application should also be provided.

Further, in step S4, the first GPU that meets the requirements is taken, and the name of the machine where the GPU is located and the number of the GPU in the machine are marked on the application.

Further, in step S4, a machine with a corresponding number of idle GPUs is searched through the gpu-usages interface, and the machine with the least number of idle GPUs is selected to add its name to the application.

Further, in step S5, the GPU manager uses an exhaustive method to allocate the GPU to the application to complete the scheduling and binding of GPU resources.

Further, the method completes the scheduling of GPU resources for one GPU application or multiple GPU applications.

Compared with the prior art, the present invention can bring the following technical effects:

1. Realize the sharing of a single GPU in multiple applications according to the percentage of GPU video memory and GPU computing power, which greatly improves the utilization efficiency of a single GPU and reduces the cost of GPU applications.

2. The topology structure between GPUs is considered when scheduling multiple GPUs, which maximizes the communication efficiency between multiple GPUs under the same application, and improves the performance of the application on GPUs.

3. Support centralized allocation of resources when scheduling GPU applications in a Kubernetes cluster, that is, support the use of machines with as many GPU applications as possible, to ensure that subsequent multi-card applications can still be successfully scheduled to the cluster.

Description of drawings

The accompanying drawings, which form a part hereof, are used to provide a further understanding of the invention, and to make other features, objects and advantages of the invention more apparent. The accompanying drawings and descriptions of the exemplary embodiments of the present invention are used to explain the present invention, and do not constitute an improper limitation of the present invention. In the attached image:

Fig. 1 is the overall flow chart of a kind of GPU resource scheduling method of the present invention;

Fig. 2 is the flow chart of the sharing of the default scheduling strategy in the prior art and the single entity GPU of the present invention;

Fig. 3 is the topological structure schematic diagram of DGX1 in the embodiment of the present invention;

Fig. 4 is a multi-GPU allocation diagram that has not considered the topology structure between GPUs and the present invention has considered the topology relationship of GPUs in the prior art;

5 is a flow chart of multiple GPU application scheduling of the default uniform scheduling strategy in the prior art and the centralized scheduling strategy of the present invention;

FIG. 6 is a schematic diagram of an example of a topology structure of a GPU in an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first", "second" and the like in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

In the present invention, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", The orientation or positional relationship indicated by "vertical", "horizontal", "horizontal", "longitudinal", etc. is based on the orientation or positional relationship shown in the drawings. These terms are primarily used to better describe the invention and its embodiments, and are not intended to limit the fact that the indicated device, element or component must have a particular orientation, or be constructed and operated in a particular orientation.

In addition, some of the above-mentioned terms may be used to express other meanings besides orientation or positional relationship. For example, the term "on" may also be used to express a certain attachment or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in the present invention can be understood according to specific situations.

Additionally, the term "plurality" shall mean two and more.

It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

Example 1

As shown in Figures 1 and 2, for an application that only needs one GPU, a method of resource allocation is supported according to the required GPU memory and the required number of cores, rather than allocating a complete GPU to the application. The default GPU resource manager does not support allocation according to the resources required by the application, but directly locks the entire GPU and allocates it to the required application.

A GPU resource scheduling method, comprising the steps of:

S1. First, basic information of the GPU is collected from the cluster, and a gpu-usages interface is provided, and the process proceeds to step S2; in step S1, the basic information of the GPU is collected, including the model of the GPU, the video memory, and the GPU core. It is convenient for the scheduler to obtain cluster GPU resource information.

S2. Create a GPU application, send an application request to the Kubernetes scheduler, and enter step S3; in step S2, in the process of creating a GPU application, the application provides the video memory value and computing power value required by the application. Since the number of cores of each GPU varies greatly and is not understood by application developers, it is directly converted to the ratio of the number of cores here. For example, a GPU application needs information similar to the following to the cluster: GPU resources with a model of T4 type, 4GB of video memory, and 25% of the number of cores.

S3. After receiving the application request, the Kubernetes scheduler will traverse all GPU applications in the cluster, and enter step S4;

S4. Calculate the GPU that meets the scheduling requirement of the application through the gpu-usages interface, and go to step S5; in step S4, if there is no GPU that meets the scheduling requirement of the application in the cluster, go to step S6 to isolate GPU resources. In step S4, the first GPU that meets the requirements is selected, and the name of the machine where the GPU is located and the number of the GPU in the machine are marked on the application.

S5. The GPU manager binds the specified GPU resources to the application according to the machine where the GPU on the application is located.

Further, S6 includes steps S60 and S61. In S60, it is found that the video memory required by the application program exceeds the preset value, or is greater than the video memory value of all GPUs in the cluster, then returns the video memory allocation failure; S61, wraps the execution thread, periodically Check the GPU core usage rate of the program. If it exceeds the set GPU core usage value, or is greater than the memory value of all GPUs in the cluster, the current execution thread is assigned to the waiting execution thread. After the shared scheduling of GPUs is completed, the GPU manager allocates GPU memory and GPU cores to applications according to the GPU application. However, if there is no corresponding resource isolation mechanism, it cannot guarantee that the application uses the GPU resources beyond the contract and other applications cannot be used normally. .

Further, the method can realize the scheduling of GPU resources for one GPU application or multiple GPU applications.

Example 2

As shown in Figures 1, 3, 4, 5, and 6; for applications that require multiple GPUs: allocate according to the GPU group with the highest communication efficiency. The connection structure of GPUs in the machine is different, and the communication speed between GPUs will also be different. As shown in Figure 3, the DGX-1 machine contains 8 GPUs, of which GPU0, GPU1, GPU2, GPU3, and GPU4 can be directly connected through NVLink, and the communication bandwidth can reach 40GB/s. The connection between GPU0 and GPU5, GPU6, and GPU7 needs to be completed through PCIe Switch and QPI, which greatly reduces the communication efficiency compared with NVLink. When allocating multiple GPUs to an application, the connection structure between the multiple GPUs allocated should be considered, also known as the topology of the GPUs. The topology structure between the GPUs can be obtained through the driving of the GPU, and the communication efficiency between the GPUs can be connected through the topology structure. An example of a GPU topology is shown in Figure 6.

Supports centralized occupancy schemes for GPU applications. The default Kubernetes resource scheduling method is a resource uniform scheduling scheme, that is, for a cluster, it tries to distribute the deployed applications evenly on each node, so as to ensure the availability of the application to the greatest extent, that is, when a machine has a problem Applications in other machines will not be affected. However, for multi-GPU applications, this scheduling scheme will cause it to fail to make full use of GPU resources. As shown in Figure 5, path a adopts the default uniform scheduling strategy, and GPU resources are used evenly. When a new multi-GPU appears The scheduling cannot be completed when the application demands; the b path adopts a centralized scheduling strategy, and tries to schedule the GPU application to a relatively busy machine, and can complete the scheduling when there are multiple GPU applications.

The following is a complete multi-GPU application deployment process:

S1. First, basic information of the GPU is collected from the cluster, and a gpu-usages interface is provided, and the process proceeds to step S2; in step S1, the basic information of the GPU is collected, including the model of the GPU, the video memory, and the GPU core. It is convenient for the scheduler to obtain cluster GPU resource information.

S2. Create a GPU application, send an application request to the Kubernetes scheduler, and enter step S3; in step S2, in the process of creating a GPU application, the application provides the video memory value and computing power value required by the application. Since the number of cores of each GPU varies greatly and is not understood by application developers, it is directly converted to the ratio of the number of cores here. For example, a GPU application needs information similar to the following to the cluster: GPU resources with a model of T4 type, 4GB of video memory, and 25% of the number of cores. In the process of creating a GPU application, you should also provide the required GPU model and the number of GPUs. If it is a multi-GPU application, you only need to provide the GPU model and the number of GPUs. For example, a GPU application needs information similar to the following to the cluster: the model is T4 type, 2GPU.

S3. After receiving the application request, the Kubernetes scheduler will traverse all GPU applications in the cluster, and enter step S4;

S4. Calculate the GPU that meets the scheduling requirement of the application through the gpu-usages interface, and go to step S5; in step S4, if there is no GPU that meets the scheduling requirement of the application in the cluster, go to step S6 to isolate GPU resources. In step S4, the first GPU that meets the requirements is selected, and the name of the machine where the GPU is located and the number of the GPU in the machine are marked on the application.

Find machines with a corresponding number of idle GPUs through the gpu-usages interface, and select the machine with the least number of idle GPUs to add its name to the application. If it is a multi-GPU application, you need to find the machine with the corresponding number of idle GPUs through the gpu-usages interface, and select the machine with the least number of idle GPUs to add its name to the application. For example, the application requires two T4 type GPUs. In this step, 3 and 4 idle T4 type GPUs are found in machine 1 and machine 2 respectively. Then machine 1 is selected as the scheduling machine of the application, and the information of the machine is added to the app.

S5. The GPU manager binds the specified GPU resources to the application according to the machine where the GPU on the application is located. In step S5, the GPU manager uses an exhaustive method to allocate GPUs to the application to complete the scheduling and binding of GPU resources.

The GPU manager uses an exhaustive method in the corresponding machine to find a set of GPUs with the highest connection efficiency and allocates them to the application to complete the scheduling and binding of GPU resources according to the machine to which the application is allocated. For example, in an application that requires two V100 GPUs, we assign them to a DGX-1 machine, where GPU0, GPU1, and GPU7 are idle. We follow (GPU0, GPU1), (GPU0, GPU7), ( The combination of GPU1, GPU7) is exhausted, and (GPU0, GPU1) is selected as the final bound GPU.

Further, S6 includes steps S60 and S61. In S60, it is found that the video memory required by the application program exceeds the preset value, or is greater than the video memory value of all GPUs in the cluster, then returns the video memory allocation failure; S61, wraps the execution thread, periodically Check the core usage rate of the GPU by the program. If it exceeds the set core usage value, or is greater than the memory value of all GPUs in the cluster, the current execution thread is assigned to the waiting execution thread. After the shared scheduling of GPUs is completed, the GPU manager allocates GPU memory and GPU cores to applications according to the GPU application. However, if there is no corresponding resource isolation mechanism, it cannot guarantee that the application uses the GPU resources beyond the contract and other applications cannot be used normally. .

Further, the method completes the scheduling of GPU resources for one GPU application or multiple GPU applications.

Example 3

As shown in Figures 1, 2, 3, 4, 5, and 6; for applications that require multiple GPUs: allocate according to the GPU group with the highest communication efficiency. The connection structure of GPUs in the machine is different, and the communication speed between GPUs will also be different. As shown in Figure 3, the DGX-1 machine contains 8 GPUs, of which GPU0, GPU1, GPU2, GPU3, and GPU4 can be directly connected through NVLink, and the communication bandwidth can reach 40GB/s. The connection between GPU0 and GPU5, GPU6, and GPU7 needs to be completed through PCIe Switch and QPI, which greatly reduces the communication efficiency compared with NVLink. When allocating multiple GPUs to an application, the connection structure between the multiple GPUs allocated should be considered, also known as the topology of the GPUs. The topology structure between the GPUs can be obtained through the driving of the GPU, and the communication efficiency between the GPUs can be connected through the topology structure. An example of a GPU topology is shown in Figure 6.

Supports centralized occupancy schemes for GPU applications. The default Kubernetes resource scheduling method is a resource uniform scheduling scheme, that is, for a cluster, it tries to distribute the deployed applications evenly on each node, so as to ensure the availability of the application to the greatest extent, that is, when a machine has a problem Applications in other machines will not be affected. However, for multi-GPU applications, this scheduling scheme will cause it to fail to make full use of GPU resources. As shown in Figure 5, path a adopts the default uniform scheduling strategy, and GPU resources are used evenly. When a new multi-GPU appears The scheduling cannot be completed when the application demands; the b path adopts a centralized scheduling strategy, and tries to schedule the GPU application to a relatively busy machine, and can complete the scheduling when there are multiple GPU applications.

The following is a complete multi-GPU application deployment process:

S1. First, basic information of the GPU is collected from the cluster, and a gpu-usages interface is provided, and the process proceeds to step S2; in step S1, the basic information of the GPU is collected, including the model of the GPU, the video memory, and the GPU core. It is convenient for the scheduler to obtain cluster GPU resource information.

S2. Create a GPU application, send an application request to the Kubernetes scheduler, and enter step S3; in step S2, in the process of creating a GPU application, the application provides the video memory value and computing power value required by the application. Since the number of cores of each GPU varies greatly and is not understood by application developers, it is directly converted to the ratio of the number of cores here. For example, a GPU application needs information similar to the following to the cluster: GPU resources with a model of T4 type, 4GB of video memory, and 25% of the number of cores. In the process of creating a GPU application, you should also provide the required GPU model and the number of GPUs. If it is a multi-GPU application, you only need to provide the GPU model and the number of GPUs. For example, a GPU application needs information similar to the following to the cluster: the model is T4 type, 2GPU.

S3. After receiving the application request, the Kubernetes scheduler will traverse all GPU applications in the cluster, and enter step S4;

S4. Calculate the GPU that meets the scheduling requirement of the application through the gpu-usages interface, and go to step S5; in step S4, if there is no GPU that meets the scheduling requirement of the application in the cluster, go to step S6 to isolate GPU resources. In step S4, the first GPU that meets the requirements is selected, and the name of the machine where the GPU is located and the number of the GPU in the machine are marked on the application.

Find machines with a corresponding number of idle GPUs through the gpu-usages interface, and select the machine with the least number of idle GPUs to add its name to the application. If it is a multi-GPU application, you need to find the machine with the corresponding number of idle GPUs through the gpu-usages interface, and select the machine with the least number of idle GPUs to add its name to the application. For example, the application requires two T4 type GPUs. In this step, 3 and 4 idle T4 type GPUs are found in machine 1 and machine 2 respectively. Then machine 1 is selected as the scheduling machine of the application, and the information of the machine is added to the app.

S5. The GPU manager binds the specified GPU resources to the application according to the machine where the GPU on the application is located. In step S5, the GPU manager uses an exhaustive method to allocate GPUs to the application to complete the scheduling and binding of GPU resources.

The GPU manager uses an exhaustive method in the corresponding machine to find a set of GPUs with the highest connection efficiency and allocates them to the application to complete the scheduling and binding of GPU resources according to the machine to which the application is allocated. For example, in an application that requires three V100 GPUs, we assign them to a DGX-1 machine, where GPU0, GPU1, GPU3, GPU5, GPU7 are idle, we follow (GPU0, GPU1, GPU3), ( GPU1, GPU3, GPU5), (GPU3, GPU5, GPU7), (GPU0, GPU1, GPU5), (GPU0, GPU1, GPU7), (GPU1, GPU3, GPU7), (...) The combination of exhaustive, select (GPU0, GPU1, GPU3) as the final bound GPU.

Further, S6 includes steps S60 and S61. In S60, it is found that the video memory required by the application program exceeds the preset value, or is greater than the video memory value of all GPUs in the cluster, then returns the video memory allocation failure; S61, wraps the execution thread, periodically Check the core usage rate of the GPU by the program. If it exceeds the set core usage value, or is greater than the memory value of all GPUs in the cluster, the current execution thread is assigned to the waiting execution thread. After the shared scheduling of GPUs is completed, the GPU manager allocates GPU memory and GPU cores to applications according to the GPU application. However, if there is no corresponding resource isolation mechanism, it cannot guarantee that the application uses the GPU resources beyond the contract and other applications cannot be used normally. .

Further, the method completes the scheduling of GPU resources for one GPU application or multiple GPU applications.

Compared with the prior art, the present invention can bring the following technical effects:

1. Realize the sharing of a single GPU in multiple applications according to the percentage of GPU video memory and GPU computing power, which greatly improves the utilization efficiency of a single GPU and reduces the cost of GPU applications.

2. The topology structure between GPUs is considered when scheduling multiple GPUs, which maximizes the communication efficiency between multiple GPUs under the same application, and improves the performance of the application on GPUs.

3. Support centralized allocation of resources when scheduling GPU applications in a Kubernetes cluster, that is, support the use of machines with as many GPU applications as possible, to ensure that subsequent multi-card applications can still be successfully scheduled to the cluster.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A GPU resource scheduling method is characterized by comprising the following steps:

s1, collecting basic information of the GPU from the cluster, providing a GPU-usages interface, and entering the step S2;

s2, creating GPU application, sending an application request to a Kubernetes scheduler, and entering the step S3;

s3, the Kubernetes dispatcher traverses all GPU applications in the cluster after receiving the application request, and the step S4 is entered;

s4, calculating a GPU meeting the scheduling requirement of the application through a GPU-usages interface, and entering the step S5;

and S5, the GPU manager binds the appointed GPU resources into the application according to the machine where the GPU is located on the application.

2. The method as claimed in claim 1, wherein in step S2, in the process of creating the GPU application, the application provides the required video memory value and computation force value.

3. The method as claimed in claim 1 or 2, wherein in step S1, the collected basic information of the GPU includes a model of the GPU, a video memory, and a GPU core.

4. The method as claimed in claim 3, wherein in step S4, if there is no GPU in the cluster meeting the scheduling requirement of the application, the method proceeds to steps S6, S6, and GPU resource isolation.

5. The method for scheduling GPU resources of claim 4, wherein S6 includes steps S60 and S61, S60, and if the video memory required by the application exceeds the preset value or is greater than all GPU video memory values in the cluster, then returning a video memory allocation failure; s61, packaging the execution thread, periodically checking the core utilization rate of the program to the GPU, if the core utilization rate exceeds the set core utilization value or is greater than the video memory values of all GPUs in the cluster, then transferring the current execution thread into the waiting execution thread.

6. A method as claimed in claim 1, 2, 4 or 5, wherein in step S2, the model number of the GPU and the number of GPUs required by the GPU should be provided in the process of creating the GPU application.

7. The method as claimed in claim 6, wherein in step S4, the first GPU meeting the requirement is taken, and the name of the machine where the GPU is located and the number of the GPU in the machine are marked on the application.

8. A method as claimed in claim 1, 2, 4, 5 or 7, wherein in step S4, the machines with the corresponding number of idle GPUs are found through the GPU-usages interface, and the machine with the least number of idle GPUs is selected from the machines and its name is added to the application.

9. A method as claimed in claim 1, 2, 4, 5 or 7, wherein in step S5, the GPU manager uses exhaustion to allocate the GPU to the application, so as to complete the scheduling and binding of GPU resources.

10. A method for scheduling GPU resources as claimed in any of claims 1 to 9, wherein the method performs scheduling of GPU resources for a GPU application or a plurality of GPU applications.

Images