CN116401062B - A server-less resource processing method, device and electronic equipment - Google Patents

Abstract

The disclosure provides a server non-perception resource processing method, a device and electronic equipment, relates to the technical field of computers, and aims to solve the technical problem that the utilization rate of a graphic processor is low and improve the utilization rate of the graphic processor under the condition that a server does not perceive. The method comprises the following steps: acquiring a first function call of a resource guarantee type task aiming at a target graphic processor, and acquiring the arrival rate of the first function call; distributing video memory for the first function call from the unified video memory address space according to the arrival rate of the first function call; acquiring a second function call of the opportunistic task aiming at any graphic processor, and acquiring the arrival rate of the second function call; acquiring a predetermined target rate; determining the leaving rate of the second function call according to the arrival rate and the target rate of the first function call; and distributing the video memory or the main memory for the second function call from the unified video memory address space according to the leaving rate of the second function call.

Description

A server-less resource processing method, device and electronic equipment

Technical field

The present disclosure relates to the field of computer technology, and in particular to a server-insensitive resource processing method, apparatus and electronic equipment.

Background technique

Container technology has been widely used in data centers as a resource management method. Container technology provides lightweight virtualization capabilities, significantly reducing the complexity and cost of data center management and deployment.

Deep learning (DL, Deep Learning) is an emerging load in data centers. With recent technological breakthroughs in deep neural network (DNN)-related research, deep learning models are increasingly integrated into applications and online services. Large enterprises have also built multi-user shared GPU (Graphics Processing Unit, graphics processor) clusters so that multiple teams can develop and train deep learning models at the same time.

To support deep learning workloads in container clouds, an entire GPU is usually statically allocated to the container. After a GPU is assigned to a container, the container gets exclusive access to the GPU. Even if the GPU is not fully utilized or even completely idle, other containers located on the same machine cannot use the GPU to ensure performance isolation. Performance isolation is crucial for production deployment. When a deep learning training task in a container has its own GPU, the deep learning training task will not be interfered by other tasks, so its throughput is also predictable.

However, the exclusive allocation strategy of GPU will lead to very low GPU utilization. Even if many GPUs are not fully utilized, some tasks may only be waiting in the waiting queue to be scheduled, resulting in longer task completion time. Related research shows that the average utilization of GPUs in a GPU cluster in one production environment is only 52%; another study shows that the median utilization of GPUs in a GPU cluster in another production environment is no higher than 10%.

This is a known issue with GPU clusters in production environments. This issue can be resolved by sharing the GPU and improving GPU utilization. The GPU sharing solutions of related technologies are divided into two categories: application layer solutions and system layer solutions. AntMan (a framework-layer GPU sharing solution from Alibaba Cloud Artificial Intelligence Credentials) is currently the most cutting-edge application layer solution. AntMan can provide high GPU utilization and performance isolation, but it requires complex modifications to the deep learning framework and restricts container users to only use specific versions of the deep learning framework. Therefore, this solution is not server-agnostic, and users must consider the compatibility of the corresponding deep learning framework when developing and configuring software. MPS (Multi-Process Server, multi-process server) is a system layer solution. However, MPS is not completely server-agnostic. Users need to set resource limits with application knowledge to ensure performance isolation, and it does not support GPU sharing when the video memory is oversubscribed. At the same time, it will merge multiple CUDA (Compute UnifiedDevice Architecture, a new hardware and software architecture for operating GPU computing) contexts into one, which leads to intolerable error sharing between tasks.

Contents of the invention

In view of the above problems, embodiments of the present disclosure provide a server-insensitive resource processing method, device, and electronic device to overcome the above problems or at least partially solve the above problems.

A first aspect of an embodiment of the present disclosure provides a server-insensitive resource processing method, including:

Obtain the first function call of the resource-guaranteed task for the target graphics processor, and obtain the arrival rate of the first function call. The target graphics processor is: one of multiple graphics processors included in the graphics processor cluster, The arrival rate of the first function call is: the arrival rate of the first function call at the rate monitor;

According to the arrival rate of the first function call, allocate video memory for the first function call from a unified video memory address space. The unified video memory address space includes: the video memory of the multiple graphics processors and the multiple graphics processors. The main memory of the graphics processor;

Obtain the second function call of the opportunistic task for any of the graphics processors, and obtain the arrival rate of the second function call;

Obtain a predetermined target rate, which is: the arrival rate of function calls to the target graphics processor when the resource guarantee task exclusively occupies the video memory of the target graphics processor;

According to the arrival rate of the first function call and the target rate, the departure rate of the second function call is determined, and the departure rate of the second function call is: the rate of the second function call leaving the rate controller ;

According to the departure rate of the second function call, video memory or main memory is allocated to the second function call from the unified video memory address space.

Optionally, determining the departure rate of the second function call based on the arrival rate of the first function call and the target rate includes:

Under the condition that the arrival rate of the first function call is guaranteed to be greater than or equal to the target rate, determine the maximum value of the departure rate of the second function call;

The maximum value of the departure rate of the second function call is determined as the departure rate of the second function call.

Optionally, determining the maximum value of the departure rate of the second function call while ensuring that the arrival rate of the first function call is greater than or equal to the target rate includes:

When the arrival rate of the first function call is greater than or equal to the target rate, linearly increase the departure rate of the second function call to obtain the maximum value of the departure rate of the second function call;

If the arrival rate of the first function call is less than the target rate, multiply the current departure rate of the second function call until the arrival rate of the first function call is not less than the target rate. , obtain the maximum value of the departure rate of the second function call.

Optionally, allocating video memory to the first function call from a unified video memory address space according to the arrival rate of the first function call includes:

Determine the first video memory size required for the first function call according to the arrival rate of the first function call;

When the size of the free video memory in the unified video memory address space is smaller than the size of the first video memory, the opportunistic tasks occupying the video memory are migrated to the main memory to obtain a size that is not smaller than the size of the first video memory. Free video memory;

Allocate video memory for the first function call.

Optionally, allocating video memory or main memory to the second function call from the unified video memory address space according to the departure rate of the second function call includes:

Determine the second video memory size required for the second function call according to the departure rate of the second function call;

When the size of free video memory in the unified video memory address space is not less than the size of the second video memory, allocate video memory for the second function call from the unified video memory address space;

When the size of free video memory in the unified video memory address space is smaller than the size of the second video memory, main memory is allocated for the second function call from the unified video memory address space.

Optionally, before the first function call of the resource-guaranteed task for the target graphics processor is obtained, it also includes:

Determine the video memory of the multiple graphics processors and the main memory of the multiple graphics processors;

The unified display memory address space is determined based on the display memories of the multiple graphics processors and the main memories of the multiple graphics processors.

Optionally, before the first function call of the resource-guaranteed task for the target graphics processor is obtained, it also includes:

When the resource-guaranteed task exclusively occupies the target graphics processor, the arrival rate of function calls to the target graphics processor is determined as the target rate.

Optionally, allocating video memory for the first function call from the unified video memory address space includes:

Allocate a virtual first physical address to the first function call from the unified video memory address space;

When the resource-guaranteed task makes an error in accessing the first virtual physical address, allocate the first real physical address of the video memory from the unified video memory address space to the first function call;

Allocating video memory or main memory for the second function call from the unified video memory address space includes:

Allocate a virtual second physical address to the second function call from the unified video memory address space;

When an error occurs when the opportunistic task accesses the second virtual physical address, a second real physical address of the video memory or a second real physical address of the main memory is allocated to the second function call from the unified video memory address space. .

A second aspect of the embodiment of the present disclosure provides a server-insensitive resource processing device, including:

The first acquisition module is used to acquire the first function call of the resource-guaranteed task to the target graphics processor, and obtain the arrival rate of the first function call. The target graphics processor is: a multi-purpose graphics processor included in the graphics processor cluster. One of the graphics processors, the arrival rate of the first function call is: the arrival rate of the first function call at the rate monitor;

A first allocation module, configured to allocate video memory to the first function call from a unified video memory address space according to the arrival rate of the first function call. The unified video memory address space includes: the multiple graphics processors The video memory and the main memory of the multiple graphics processors;

A second acquisition module, configured to acquire the second function call of the opportunistic task for any of the graphics processors, and acquire the arrival rate of the second function call;

A target acquisition module, configured to acquire a predetermined target rate, which is the arrival rate of function calls to the target graphics processor when the resource guarantee task exclusively occupies the video memory of the target graphics processor;

A rate determination module, configured to determine the departure rate of the second function call according to the arrival rate of the first function call and the target rate, where the departure rate of the second function call is: the second function call rate leaving the rate controller;

The second allocation module is configured to allocate video memory or main memory to the second function call from the unified video memory address space according to the departure rate of the second function call.

A third aspect of an embodiment of the present disclosure provides an electronic device, including: a processor; a memory for storing executable instructions by the processor; wherein the processor is configured to execute the executable instructions, To implement the server-insensitive resource processing method of the first aspect.

A fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the server of the first aspect. Unaware resource handling methods.

Embodiments of the present disclosure include the following advantages:

In the embodiment of the present disclosure, the video memory and main memory of multiple graphics processors on a physical machine can be unified into a unified video memory address space, and then the unified video memory address space can be allocated, thereby realizing resource guarantee tasks and opportunistic tasks on the GPU. Sharing improves GPU utilization without the server being aware of it. Among them, the address space allocated for resource-guaranteed tasks is video memory, which ensures the performance of resource-guaranteed tasks. Determining the departure rate of the second function call based on the arrival rate and target rate of the first function call can ensure that the performance of the opportunistic task is improved without affecting the resource guarantee task. In this way, the embodiments of the present disclosure improve GPU utilization without being aware of the server, without modifying the task framework, ensuring performance isolation, and without merging multiple CUDA contexts into one.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. , for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative labor.

Figure 1 is a schematic diagram of the architecture of TGS in an embodiment of the present disclosure;

Figure 2 is a step flow chart of a server-insensitive resource processing method in an embodiment of the present disclosure;

Figure 3 is a schematic flowchart of adaptive rate control in an embodiment of the present disclosure;

Figure 4 is a schematic diagram of the relationship between the arrival rate of the first function call and the departure rate of the second function call in an embodiment of the present disclosure;

Figure 5 is a schematic structural diagram of a server-less resource processing device in an embodiment of the present disclosure.

Detailed ways

In order to make the above objects, features and advantages of the present disclosure more obvious and understandable, the present disclosure will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

Resource-guaranteed tasks can obtain performance guarantees similar to those of exclusive GPU, while opportunistic tasks use GPU spare resources as much as possible without performance guarantees. By letting these two types of tasks share the GPU, GPU utilization can be improved.

For the resource guarantee task referred to in the embodiment of the present disclosure, for each function call sent to the graphics processor, the address space size of the requested graphics processor is close to the same size range. Therefore, the size of the address space required by a resource-guaranteed task can be estimated based on the rate at which the resource-guaranteed task issues function calls to the graphics processor.

Similarly, for each function call directed to the graphics processor sent by the opportunistic task referred to in the embodiment of the present disclosure, the address space size of the requested graphics processor is close to the same size range. Therefore, the size of the address space required by an opportunistic task can be estimated based on the rate at which the opportunistic task issues function calls to the graphics processor.

In a production environment, deep learning training tasks can usually be divided into two categories: resource guarantee tasks and opportunistic tasks. Each time a deep learning training task issues a function call to a function on a graphics processor, the address space size of the requested graphics processor is close to the same size range. Therefore, the resource guarantee tasks and opportunistic tasks referred to in the embodiments of the present disclosure may be deep learning training tasks, or other tasks that can estimate the required address space size based on the rate at which function calls are issued.

The embodiment of the present disclosure proposes a transparent non-aware GPU sharing system, named TGS. The TGS can be used to execute a server non-aware resource processing method proposed in the embodiment of the present disclosure. TGS is located between the GPU and the container, and the container and the applications in the container are unaware of TGS. Each GPU is exposed to the container like a normal GPU. Users can use any framework and tool to develop and train deep learning models. TGS can hijack GPU-related function calls issued from the container and control the usage of GPU resources by each container by controlling these function calls.

As shown in Figure 1, TGS can include a rate monitor, a rate controller, and a unified memory module. The rate monitor can obtain the rate at which each resource-guaranteed task and each opportunistic task issues function calls for any graphics processor. The rate controller can control the departure rate of opportunistic tasks. The unified memory module can transfer the number of functions on the physical machine to The video memory of multiple graphics processors and the main memory of multiple graphics processors are unified into a unified video memory address space.

The application code of the application in the container is encapsulated into a function and executed on the GPU. This code is called GPU kernel. GPU kernels are highly optimized based on the GPU architecture and execution model. A small deep neural network model training task may not use all the computing resources on a GPU. In this case, if a GPU is monopolized by a container, low utilization of the GPU will result. TGS allows one GPU to be shared by multiple containers to improve GPU utilization.

TGS can distinguish between resource guarantee tasks and opportunity tasks. TGS can ensure that resource-guaranteed tasks will not be affected by opportunistic tasks, and opportunistic tasks will only use the spare GPU computing resources of resource-guaranteed tasks.

GPU has video memory resources independent of main memory. Modern GPUs have video memory sizes ranging from a few GB to tens of GB. GPU memory is responsible for storing the state and data used by applications for GPU calculations. The GPU computing unit accesses video memory much faster than main memory. Similar to GPU computing resources, when a single container cannot use all the GPU memory, the GPU memory can be shared by multiple containers.

Referring to FIG. 2 , a flow chart of steps of a server-insensitive resource processing method in an embodiment of the present disclosure is shown. As shown in FIG. 2 , the server-insensitive resource processing method may specifically include steps S11 to S16 .

Step S11: Obtain the first function call of the resource-guaranteed task directed to the target graphics processor, and obtain the arrival rate of the first function call.

The target graphics processor is one of multiple graphics processors included in a graphics processor cluster. The arrival rate of the first function call is: the arrival rate of the first function call at the rate monitor.

The issue, arrival, and departure rates of a resource-guaranteed task's first function call to either graphics processor are the same. Among them, the issuing rate refers to the rate at which the resource-guaranteed task issues the first function call; the arrival rate refers to the rate at which the first function call reaches the rate monitor; the departure rate refers to the rate at which the first function call leaves the rate controller. . The first function call leaves the rate controller, indicating that the first function call will call the graphics processor. In order to ensure the performance of resource-guaranteed tasks, the rate of resource-guaranteed tasks is only monitored but not limited. Therefore, the issue rate, arrival rate and departure rate of a resource-guaranteed task for a first function call of any graphics processor All the same.

The target graphics processor may be any graphics processor among multiple graphics processors included in the GPU cluster.

Step S12: According to the arrival rate of the first function call, allocate video memory for the first function call from the unified video memory address space.

The unified video memory address space includes: video memory of the multiple graphics processors and main memory of the multiple graphics processors.

Before obtaining the first function call for the target graphics processor of the resource-guaranteed task, the unified video memory address space may be obtained first, which may specifically include: determining the video memory of the multiple graphics processors and the memory address of the multiple graphics processors. Main memory; determining the unified video memory address space according to the video memories of the multiple graphics processors and the main memories of the multiple graphics processors.

In related technology, GPU provides a feature called unified video memory, which unifies a GPU video memory and main memory in the same address space. The unified memory feature is often used to simplify GPU memory management. TGS innovatively uses CUDA unified video memory: TGS redirects all GPU video memory allocations in the same GPU cluster to unified video memory address space allocation to achieve GPU video memory resource sharing that ensures that applications are unaware of the system and perform performance isolation.

Specifically, TGS disguises the unified video memory address space as ordinary GPU video memory and exposes it to the container. When a container calls the GPU memory allocation API (Application Programming Interface, Application Programming Interface), whether the allocation request is for the GPU ordinary memory or the unified memory address space, TGS will hijack the function call and allocate it in the unified memory address space. Corresponding video memory. When resource-guaranteed tasks do not use all the video memory, opportunistic tasks can use the remaining GPU video memory.

Pseudo GPU memory refers to what appears to containers and applications as ordinary GPU memory. However, it may actually be GPU memory or main memory. The specific location is mainly adjusted according to the usage of GPU memory. It is worth noting that this disclosure does not change the virtual video memory system. Pseudo GPU memory is still virtual memory, and tasks in the container use virtual addresses to access pseudo GPU memory. A GPU virtual address is translated into a GPU physical address by the GPU's MMU (Memory Management Unit); a host virtual address is translated into a host physical address by the host's MMU.

The non-aware unified memory mechanism adopted by TGS is different from the unified video memory mechanism in related technologies in terms of performance isolation and non-aware overbooking of GPU memory. In order to improve performance isolation, TGS uses placement preferences to set up a unified memory address space so that resource-guaranteed tasks have higher memory priority. When the GPU memory is not fully used, all memory requests for all tasks are allocated to the GPU memory. When the GPU memory is full, TGS will try to put the data and status of resource-guaranteed tasks on the GPU, and evict the data and status of opportunistic tasks to the main memory when necessary. This process is completely unaware of the application within the container, because the container still uses the same virtual address to access their allocated video memory. A virtual address is just a physical address translated to a different location.

Because the arrival rate and departure rate of the first function call are the same, and the first function call departure rate controller represents that the first function call will call the graphics processor, and the function call for the graphics processor issued according to the resource guaranteed task The rate can be used to estimate the size of the address space required for resource guarantee tasks. Therefore, the size of the first video memory required for the first function call can be determined based on the arrival rate of the first function call.

In order to ensure the performance of resource-guaranteed tasks, video memory will be allocated to resource-guaranteed tasks first. Opportunistic tasks will only use video memory when there is free video memory. Therefore, when the size of the free video memory in the unified video memory address space is not less than the size of the first video memory, the video memory can be allocated directly from the free video memory for the first function call.

When the size of the free video memory in the unified video memory address space is smaller than the size of the first video memory, the opportunistic tasks occupying the video memory can be migrated to the main memory, thereby obtaining a free video memory that is not smaller than the size of the first video memory, and then from the free video memory to the main memory. In video memory, allocate video memory for the first function call.

Step S13: Obtain the second function call of the opportunistic task for any of the graphics processors, and obtain the arrival rate of the second function call.

The arrival rate of the second function call refers to the rate at which the second function call arrives at the rate monitor. The arrival rate of the second function call is the same as the corresponding issuance rate of the second function call. However, in order to ensure that opportunistic tasks do not affect the performance of resource-guaranteed tasks, it is necessary to ensure that the departure rate of the second function call does not affect the departure rate of the first function call.

Step S14: Obtain a predetermined target rate.

The target rate is: the arrival rate of function calls to the target graphics processor when the resource guarantee task exclusively occupies the video memory of the target graphics processor.

The target rate can be determined before obtaining the first function call for the target graphics processor for a resource-guaranteed task. The target rate is the arrival rate of function calls to the target GPU when a resource-guaranteed task exclusively occupies the target GPU. TGS can measure the target rate before opportunistic tasks share GPU resources with resource-guaranteed tasks.

Step S15: Determine the departure rate of the second function call based on the arrival rate of the first function call and the target rate.

The exit rate of the second function call is: the rate at which the second function call leaves the rate controller.

Based on the arrival rate of the first function call and the target rate, it may be determined whether the arrival rate of the first function call is affected. When the arrival rate of the first function call is greater than or equal to the target rate, it can be determined that the arrival rate of the first function call is not affected; when the arrival rate of the first function call is less than the target rate, it can be determined that the arrival rate of the first function call is not affected. The arrival rate of calls is affected. In order to ensure that the departure rate of the second function call does not affect the departure rate of the first function call, the departure rate of the second function call may be determined based on the arrival rate and the target rate of the first function call.

In order to improve the GPU utilization as much as possible without affecting the departure rate of the first function call, the maximum value of the departure rate of the second function call can be determined, and the maximum value of the departure rate of the second function call can be determined as Determines the departure rate for the second function call. The method of determining the maximum value of the departure rate of the second function call will be described in detail later.

Step S16: According to the departure rate of the second function call, allocate video memory or main memory to the second function call from the unified video memory address space.

The address space size required by the opportunistic task can be estimated based on the rate of function calls directed to the graphics processor issued by the opportunistic task. Therefore, the address space required by the second function call can be determined based on the departure rate of the second function call. Secondary memory size.

When the size of the free video memory in the unified video memory address space is not less than the size of the second video memory, the video memory can be allocated directly from the unified video memory address space for the second function call. When the size of free video memory in the unified video memory address space is smaller than the size of the second video memory, main memory can be allocated for the second function call from the unified video memory address space.

By adopting the technical solutions of the embodiments of the present disclosure, GPU utilization can be improved without the server being aware of it, without modifying the task framework, ensuring performance isolation, and without merging multiple CUDA contexts into one.

Here's how to determine the maximum departure rate for the second function call. Figure 3 is a schematic flowchart of adaptive rate control in an embodiment of the present disclosure. The rate control algorithm can control the departure rate of the second function call of the opportunistic task, so that the arrival rate of the first function call of the resource guarantee task is not affected while maximizing the departure rate of the second function call of the opportunistic task. . Formally speaking, let a _in be the arrival rate of the first function call, let a _out be the departure rate of the first function call, let β _in be the arrival rate of the second function call, let β _out be the arrival rate of the second function call Departure velocity, let R be the target velocity. Because it only monitors but does not limit the rate of resource guarantee tasks, a _in = a _out .

The goal of the rate control algorithm is to maximize β _out under the constraint of a _in >R. Among them, β _out is a controllable variable, and the change of a _in is affected by β _out . Let f be a function that captures the relationship between a _in and β _out , that is, a _in =f(β _out ). In this way, the rate control algorithm essentially solves the following optimization problem:

maxβ _out

stα _in =f(β _out )≥R

β _out ≥0

Although the exact form of f(β _out ) is not known, its approximate shape is known due to the nature of the problem itself. Figure 4 shows the relationship between the arrival rate of a first function call and the departure rate of a second function call. As shown in Figure 4(a), usually when β _out is small, a _in is flat and equal to R; when β _out is large, a _in decreases monotonically. The intuition behind this phenomenon is: when β _out is small, the GPU is not fully utilized, and the second function call to execute the opportunistic task will not affect the performance of the resource guarantee task, so the function is a straight line line; but after a turning point, the GPU is fully utilized. At this time, adding GPU resources for opportunistic tasks will affect the performance of resource-guaranteed tasks, so the function shows a monotonous downward trend. It is worth noting that the downward trend is not necessarily linear, and Figure 4(a) only depicts a general trend. The goal of the adaptive rate adjustment algorithm is to find the turning point where the function shows a downward trend.

Figure 4(a) only shows a common situation. There are two special cases. Figure 4(b) shows the special case where the GPU is completely occupied by resource-guaranteed tasks. At this time, even the second function call that releases only a small number of opportunistic tasks will affect performance. In this case, the curve has no straight parts. Figure 4(c) shows the special situation where opportunistic tasks occupy less GPU resources and will not affect the performance of resource-guaranteed tasks at all. In this case, the curve does not have a monotonically decreasing portion.

In order to find the optimal β _out , the embodiment of the present disclosure uses an AIMD (Additive Increase and Multiplicative Decrease) algorithm to control β _out . When opportunistic tasks and resource guarantee tasks begin to share, if α _in >R, TGS linearly increases β _out , otherwise it multiplicatively decreases β _out . The AIMD algorithm ensures that the final β _out will quickly converge to the turning point.

Furthermore, when a resource-guaranteed task changes its GPU utilization, TGS can detect that the arrival rate of the first function call changes by more than a threshold. In this case, the control module will reduce β _out multiplicatively. When the resource guarantee task rate stabilizes, the control module reuses AIMD to adjust β _out to converge to a new turning point.

Therefore, when the arrival rate of the first function call is greater than or equal to the target rate, the departure rate of the second function call is linearly increased to obtain the maximum value of the departure rate of the second function call; If the arrival rate of the first function call is less than the target rate, multiply the current departure rate of the second function call until the arrival rate of the first function call is not less than the target rate. , obtain the maximum value of the departure rate of the second function call.

TGS uses an adaptive rate control method. The core idea is to carefully control the departure rate of the second function call in the opportunistic queue according to the arrival rate of the first function call of the resource-guaranteed task, so that the opportunistic task can use exactly Use up remaining GPU resources. TGS decouples control from the low-level details of the GPU and does not require control of the GPU's internal hardware.

This method requires a feedback signal to tell the control loop whether to release the second function call in the opportunistic queue to increase GPU resource usage, or to reduce the departure rate of the second function call to avoid affecting resource-guaranteed tasks. Ideally, user performance (i.e., training throughput of a deep learning training task) can be used as a feedback signal, since this is the metric that users really care about. However, since obtaining training throughput requires the use of user-level knowledge, this goes against designing a server-agnostic system-level solution.

One possible design choice is to use GPU utilization, i.e. increase the exit rate of the second function call if the GPU utilization is less than 100%. However, the definition of GPU utilization is hardware-specific and often vague. Today's GPUs and deep learning accelerators themselves are heterogeneous devices, including a variety of computing units, such as Tensor Core and CUDA Core on NVIDIA GPUs. GPU utilization reported by the GPU driver is difficult to be accurate. Even if it were indeed accurate, it would be difficult to make real sense on a GPU with multiple compute units. Therefore, even when GPU utilization is below 100%, it does not mean that GPU resources for opportunistic tasks can be increased without affecting resource-guaranteed tasks. For example, a resource-guaranteed task and an opportunistic task may compete for the same resource that has been exhausted. Even if other resources are idle and the GPU utilization is not 100%, the resource-guaranteed task may be interfered with.

TGS uses the arrival rate of the first function call as the feedback rate. A deep learning training task constructs a computational graph based on a deep neural network model. The training task generates and sends function calls based on the computational graph. Computational graphs capture dependencies between function calls. The arrival rate of function calls directly reflects training throughput. If a training task slows down, the rate at which function calls arrive will slow down. Therefore, TGS uses a rate monitor to detect the arrival rate of the first function call of the resource-guaranteed task, and uses it as a feedback signal to control the departure rate of the second function call of the opportunistic task. It is worth noting that the competition between any resource-guaranteed tasks and opportunistic tasks can be measured with this metric, including GPU cache competition, CPU competition and network competition. Some of them are beyond the control of the GPU hardware design, and TGS can control them all. Because the arrival rate of function calls (including the first function call and the second function call) fluctuates slightly, TGS uses a moving average to smooth the estimate of the arrival rate of function calls. For the arrival rate of the first function call of resource-guaranteed tasks, TGS only performs a simple counting operation to estimate. TGS does not cache first function calls, but passes them directly to the GPU to minimize the performance impact on resource-guaranteed tasks.

TGS uses an adaptive rate control mechanism and a non-aware unified memory mechanism to solve the two main challenges of system-level non-aware shared GPUs. The first challenge is how to adaptively share GPU computing resources among different containers without application knowledge. The latest current solutions either restrict users from using modified deep learning frameworks, like AntMan; or, like MPS, require explicit use of application knowledge to set GPU computing resource limits. To address this challenge, TGS's rate monitor monitors the performance of each container and provides real-time signals to the control loop. Based on this signal, TGS's rate controller can adaptively control the function calls sent to the GPU by each container. The entire control loop will eventually automatically converge to a stable state, where opportunistic tasks use as much idle GPU resources as possible without affecting the performance isolation of resource-guaranteed tasks.

The second challenge is to provide GPU memory oversubscription without awareness. AntMan modified the deep learning framework to swap out part of the data in the GPU memory to the main memory when the GPU memory is oversold. The system layer solution MPS does not support GPU memory overbooking and requires the application itself to handle the swapping in and out between graphics memory and main memory. None of these methods are mindless. In order to solve this challenge, TGS uses a unified video memory module to unify the video memory and main memory of multiple GPUs under one address space. TGS hijacks and redirects the GPU memory of each container to a unified memory address space. When the GPU video memory is oversubscribed, TGS can automatically evict the data stored in the video memory by some opportunistic tasks to the main memory, and map the corresponding modified virtual address to the new physical address on the main memory. The entire application process is imperceptible and does not affect application operation. In order to ensure performance isolation, TGS uses placement preferences to prioritize data for resource-guaranteed tasks to GPU memory, and spare GPU memory can be allocated to opportunistic tasks.

The overall design of the TGS also reflects two other advantages. First of all, the overall architecture of TGS is lightweight. It does not use heavyweight binary translation to achieve GPU sharing, which ensures its low performance overhead and is consistent with the principles of container technology. Secondly, TGS provides the same error isolation capabilities as ordinary containers. Different containers using TGS use different CUDA contexts instead of merging multiple CUDA contexts into one like MPS. Therefore, errors applied within one container will not affect other containers.

The non-aware unified memory mechanism of TGS also solves the problem of excessive requests for GPU memory by existing deep learning frameworks without modifying the deep learning framework. Specifically, allocating video memory for the first function call from the unified video memory address space may include: allocating a virtual first physical address for the first function call from the unified video memory address space; When an error occurs when the resource guarantee task accesses the first virtual physical address, the first real physical address of the video memory is allocated to the first function call from the unified video memory address space. Allocating video memory or main memory for the second function call from the unified video memory address space may include: allocating a virtual second physical address for the second function call from the unified video memory address space; When an error occurs when the opportunistic task accesses the second virtual physical address, a second real physical address of the video memory or a second real physical address of the main memory is allocated to the second function call from the unified video memory address space. .

When the deep learning framework applies for all available GPU memory, TGS allocates the corresponding address range in the unified memory address space. However, the actual allocation of video memory occurs after the application triggers a page table error by accessing the address for the first time, and then the real physical address is actually allocated. The end result is that only truly active data is assigned a physical address in GPU memory. In this way, opportunistic tasks can obtain more GPU space to achieve efficient GPU sharing.

The TGS proposed in the embodiment of this disclosure is a system that supports non-aware sharing of GPUs for deep learning training tasks in a container cloud environment. TGS is placed at the system layer, so users are unaware of it. It realizes the convenience that application layer solutions can provide, and it allows users to develop and run tasks within containers using any version of any deep learning framework.

There are two main technical challenges in the process of developing a system-level GPU sharing solution that ensures performance isolation. The first technical challenge is how to adaptively adjust the GPU shared by different containers without application knowledge. In order to solve this problem, TGS will monitor the performance of resource guarantee tasks at runtime and adaptively adjust the resource quotas of opportunistic tasks. This control logic will eventually converge to an optimal resource allocation situation, that is, opportunistic tasks use as much idle GPU resources as possible without affecting resource guarantee tasks.

The second challenge is to achieve imperceptible GPU memory oversubscription. The GPU has its own independent video memory to save application state. This disclosure implements a unique non-aware unified memory management mechanism at the system layer, provides a unified memory perspective at the system layer, and does not require modification of applications and frameworks. When the GPU memory is oversubscribed, this mechanism can manage the swapping in and out of data in the GPU memory without being aware of it. TGS uses placement preferences to ensure the performance of resource-guaranteed tasks.

Experiments show that: (1) TGS can ensure the throughput of resource guarantee tasks. (2) TGS ensures that the throughput of opportunistic tasks is close to the latest application layer GPU sharing solution AntMan, and can increase throughput up to 15 times compared to the existing system layer solution MPS.

It should be noted that for the sake of simple description, the method embodiments are expressed as a series of action combinations. However, those skilled in the art should know that the disclosed embodiments are not limited by the described action sequence because Certain steps may be performed in other orders or simultaneously according to embodiments of the present disclosure. Secondly, those skilled in the art should also know that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present disclosure.

Figure 5 is a schematic structural diagram of a server-less resource processing device in an embodiment of the present disclosure. As shown in Figure 5, the device includes a first acquisition module, a first allocation module, a second acquisition module, a target acquisition module, a rate Determination module and second allocation module, where:

The first acquisition module is used to acquire the first function call of the resource-guaranteed task to the target graphics processor, and obtain the arrival rate of the first function call. The target graphics processor is: a multi-purpose graphics processor included in the graphics processor cluster. One of the graphics processors, the arrival rate of the first function call is: the arrival rate of the first function call at the rate monitor;

A first allocation module, configured to allocate video memory to the first function call from a unified video memory address space according to the arrival rate of the first function call. The unified video memory address space includes: the multiple graphics processors The video memory and the main memory of the multiple graphics processors;

A second acquisition module, configured to acquire the second function call of the opportunistic task for any of the graphics processors, and acquire the arrival rate of the second function call;

A target acquisition module, configured to acquire a predetermined target rate, which is the arrival rate of function calls to the target graphics processor when the resource guarantee task exclusively occupies the video memory of the target graphics processor;

A rate determination module, configured to determine the departure rate of the second function call according to the arrival rate of the first function call and the target rate, where the departure rate of the second function call is: the second function call rate leaving the rate controller;

The second allocation module is configured to allocate video memory or main memory to the second function call from the unified video memory address space according to the departure rate of the second function call.

Optionally, the rate determination module is specifically configured to perform:

Under the condition that the arrival rate of the first function call is guaranteed to be greater than or equal to the target rate, determine the maximum value of the departure rate of the second function call;

The maximum value of the departure rate of the second function call is determined as the departure rate of the second function call.

Optionally, when ensuring that the arrival rate of the first function call is greater than or equal to the target rate, determining the maximum value of the departure rate of the second function call includes:

When the arrival rate of the first function call is greater than or equal to the target rate, linearly increase the departure rate of the second function call to obtain the maximum value of the departure rate of the second function call;

If the arrival rate of the first function call is less than the target rate, multiply the current departure rate of the second function call until the arrival rate of the first function call is not less than the target rate. , obtain the maximum value of the departure rate of the second function call.

Optionally, the first allocation module is specifically used to execute:

Determine the first video memory size required for the first function call according to the arrival rate of the first function call;

When the size of the free video memory in the unified video memory address space is smaller than the size of the first video memory, the opportunistic tasks occupying the video memory are migrated to the main memory to obtain a size that is not smaller than the size of the first video memory. Free video memory;

Allocate video memory for the first function call.

Optionally, the second allocation module is specifically used to execute:

Determine the second video memory size required for the second function call according to the departure rate of the second function call;

When the size of free video memory in the unified video memory address space is not less than the size of the second video memory, allocate video memory for the second function call from the unified video memory address space;

When the size of free video memory in the unified video memory address space is smaller than the size of the second video memory, main memory is allocated for the second function call from the unified video memory address space.

Optionally, before the first function call of the resource-guaranteed task for the target graphics processor is obtained, it also includes:

A determining module, configured to determine the display memory of the multiple graphics processors and the main memory of the multiple graphics processors;

An address space module, configured to determine the unified video memory address space based on the video memories of the multiple graphics processors and the main memories of the multiple graphics processors.

Optionally, before the first function call of the resource-guaranteed task for the target graphics processor is obtained, it also includes:

A target determination module, configured to determine the arrival rate of function calls to the target graphics processor as the target rate when the resource guarantee task exclusively occupies the target graphics processor.

Optionally, the first allocation module is specifically configured to: allocate a virtual first physical address for the first function call from the unified video memory address space; and access the first virtual address when the resource guarantee task accesses the first function call. When the virtual physical address is incorrect, allocate the first real physical address of the video memory for the first function call from the unified video memory address space;

The second allocation module is specifically configured to: allocate a virtual second physical address for the second function call from the unified display memory address space; when the opportunistic task makes an error in accessing the second virtual physical address , from the unified display memory address space, allocate a second real physical address of the display memory or a second real physical address of the main memory for the second function call.

It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple. For relevant information, please refer to the method embodiment.

Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on its differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

It should be understood by those skilled in the art that embodiments of the present disclosure may be provided as methods, apparatuses, or computer program products. Accordingly, embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, embodiments of the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Embodiments of the disclosure are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, electronic devices and computer program products according to embodiments of the disclosure. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine such that the instructions are executed by the processor of the computer or other programmable data processing terminal device. Means are generated for implementing the functions specified in the process or processes of the flowchart diagrams and/or the block or blocks of the block diagrams.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing terminal equipment to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the The instruction means implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, so that a series of operating steps are performed on the computer or other programmable terminal equipment to produce computer-implemented processing, thereby causing the computer or other programmable terminal equipment to perform a computer-implemented process. The instructions executed on provide steps for implementing the functions specified in a process or processes of the flow diagrams and/or a block or blocks of the block diagrams.

Although preferred embodiments of the disclosed embodiments have been described, those skilled in the art will be able to make additional changes and modifications to these embodiments once the basic inventive concepts are apparent. Therefore, it is intended that the appended claims be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the disclosed embodiments.

Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or any such actual relationship or sequence between operations. Furthermore, the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or end device that includes a list of elements includes not only those elements, but also elements not expressly listed or other elements inherent to such process, method, article or terminal equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

The server-insensitive resource processing method, device and electronic equipment provided by the present disclosure have been introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation modes of the present disclosure. The description of the above embodiments is only for To help understand the methods and core ideas of the present disclosure; at the same time, for those of ordinary skill in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present disclosure. In summary, the content of this specification does not It should be understood as a limitation of this disclosure.

Claims (10)

1. A server non-aware resource handling method, comprising:

acquiring a first function call of a resource guarantee type task aiming at a target graphic processor, and acquiring the arrival rate of the first function call, wherein the target graphic processor is: one of a plurality of graphics processors included in a graphics processor cluster, the first function call arriving at a rate of: the first function calls a rate of arrival at a rate monitor;

and distributing video memory for the first function call from a unified video memory address space according to the arrival rate of the first function call, wherein the unified video memory address space comprises: the video memories of the plurality of graphics processors and the main memory of the plurality of graphics processors;

acquiring a second function call of the opportunistic task aiming at any graphic processor, and acquiring the arrival rate of the second function call;

obtaining a predetermined target rate, wherein the target rate is: when the resource guarantee type task solely occupies the display memory of the target graphic processor, aiming at the arrival rate of the function call of the target graphic processor;

determining the leaving rate of the second function call according to the arrival rate of the first function call and the target rate, wherein the leaving rate of the second function call is as follows: the second function invokes a rate of departure from the rate controller;

And distributing video memory or main memory for the second function call from the unified video memory address space according to the leaving rate of the second function call.

2. The method of claim 1, wherein the determining the departure rate of the second function call from the arrival rate of the first function call and the target rate comprises:

determining a maximum value of the departure rate of a second function call under the condition that the arrival rate of the first function call is ensured to be greater than or equal to the target rate;

and determining the maximum value of the leaving rate of the second function call as the leaving rate of the second function call.

3. The method of claim 2, wherein determining a maximum value of a departure rate of a second function call with the arrival rate of the first function call being guaranteed to be greater than or equal to the target rate comprises:

linearly increasing the departure rate of the second function call under the condition that the arrival rate of the first function call is greater than or equal to the target rate, so as to obtain the maximum value of the departure rate of the second function call;

and under the condition that the arrival rate of the first function call is smaller than the target rate, multiplicatively reducing the current departure rate of the second function call until the arrival rate of the first function call is not smaller than the target rate, and obtaining the maximum value of the departure rate of the second function call.

4. The method of claim 1, wherein allocating the memory for the first function call from the unified memory address space according to the arrival rate of the first function call comprises:

determining a first video memory size required by the first function call according to the arrival rate of the first function call;

under the condition that the size of the idle video memory in the unified video memory address space is smaller than the first video memory, transferring the opportunistic task occupying the video memory into a main memory to obtain the idle video memory with the size not smaller than the first video memory;

and distributing the video memory for the first function call.

5. The method of claim 1, wherein allocating memory or main memory for the second function call from the unified memory address space according to the departure rate of the second function call comprises:

determining a second video memory size required by the second function call according to the leaving rate of the second function call;

under the condition that the size of the idle video memory in the unified video memory address space is not smaller than the second video memory size, distributing the video memory for the second function call from the unified video memory address space;

And distributing main memory for the second function call from the unified video memory address space under the condition that the size of the idle video memory in the unified video memory address space is smaller than the second video memory size.

6. The method of claim 1, wherein prior to the first function call for the target graphics processor by the get resource-responsible task, further comprising:

determining the video memories of the plurality of graphics processors and the main memory of the plurality of graphics processors;

and determining the unified video memory address space according to the video memories of the plurality of graphic processors and the main memories of the plurality of graphic processors.

7. The method of claim 1, wherein prior to the first function call for the target graphics processor by the get resource-responsible task, further comprising:

and determining the arrival rate of the function call of the target graphic processor as the target rate when the resource guarantee type task solely occupies the target graphic processor.

8. The method of claim 1, wherein allocating memory for the first function call from a unified memory address space comprises:

Distributing a first virtual address for the first function call from the unified video memory address space;

when the resource guarantee type task accesses the first virtual address and is in error, a first real physical address of a video memory is allocated for the first function call from the unified video memory address space;

and the step of distributing the video memory or the main memory for the second function call from the unified video memory address space comprises the following steps:

distributing a second virtual address for the second function call from the unified video memory address space;

and when the opportunity type task accesses the second virtual address and is in error, distributing a second real physical address of the video memory or a second real physical address of the main memory for the second function call from the unified video memory address space.

9. A server non-aware resource handling device, comprising:

the first acquisition module is used for acquiring a first function call of a resource guarantee type task for a target graphic processor and acquiring the arrival rate of the first function call, and the target graphic processor is: one of a plurality of graphics processors included in a graphics processor cluster, the first function call arriving at a rate of: the first function calls a rate of arrival at a rate monitor;

The first allocation module is configured to allocate a video memory for the first function call from a unified video memory address space according to an arrival rate of the first function call, where the unified video memory address space includes: the video memories of the plurality of graphics processors and the main memory of the plurality of graphics processors;

the second acquisition module is used for acquiring a second function call of the opportunistic task for any graphic processor and acquiring the arrival rate of the second function call;

the target acquisition module is used for acquiring a predetermined target rate, wherein the target rate is as follows: when the resource guarantee type task solely occupies the display memory of the target graphic processor, aiming at the arrival rate of the function call of the target graphic processor;

the rate determining module is configured to determine, according to the arrival rate of the first function call and the target rate, the departure rate of the second function call, where the departure rate of the second function call is: the second function invokes a rate of departure from the rate controller;

and the second allocation module is used for allocating the video memory or the main memory for the second function call from the unified video memory address space according to the leaving rate of the second function call.

10. An electronic device, comprising:

a processor;

a memory for storing the processor-executable instructions;

wherein the processor is configured to execute the executable instructions to implement the server unaware resource handling method of any of claims 1 to 8.