CN107491340B - Implementation method of giant virtual machine across physical machines - Google Patents

Abstract

The present invention provides a method for realizing a giant virtual machine across physical machines, comprising the following steps: step 1, the creation step of the giant virtual machine; step 2, the interrupt processing step of the giant virtual machine; step 3, the processing step of equipment interruption; The fourth step is a remote access step based on the high-speed shared memory access technology. The invention realizes that multiple physical machines provide computing, IO and other resources for one virtual machine. Remote memory access is performed between machines through remote high-speed memory access technology, realizing high-speed distributed shared memory.

Description

Implementation method of giant virtual machine across physical machines

technical field

The present invention relates to a method for realizing a giant virtual machine, in particular, to a method for realizing a giant virtual machine (Giant VM) across physical machines.

Background technique

Virtualization refers to virtualizing a physical computer into multiple logical computers through virtualization technology. Each logical computer can run a different operating system, independent of each other, without affecting each other, providing a logical view of data, computing power, storage, and other resources. With the development of machine learning and other fields, people have put forward higher requirements for the performance of virtual machines, and the resources of a single physical computer, including CPU, memory and other devices (such as GPU, FPGA), can no longer meet the requirements. It is a feasible solution to virtualize multiple physical machines into a single virtual machine (referred to as "multi-virtual-one") so that the virtual machine has massive resources.

Although there are some multi-virtual distributed systems, they are mainly implemented by adding an additional virtualization abstraction layer at the bottom layer of the architecture, bypassing the monitoring and management of the host system and virtual machine management software. Such an architecture not only cannot utilize the latest virtual machine management software, but also brings huge complexity when adjusting the distributed system architecture, making it difficult to realize real-time resource allocation. At the same time, the distributed shared memory based on traditional network has bottlenecks in transmission speed, which limits the scalability of the system.

SUMMARY OF THE INVENTION

In view of the defects in the prior art, the purpose of the present invention is to provide a method for implementing a giant virtual machine across physical machines, which enables multiple physical machines to provide computing, IO (input and output) and other resources for a virtual machine. Remote memory access is performed between machines through remote high-speed memory access technology (such as RDMA), which realizes high-speed distributed shared memory. In addition, the present invention also creatively provides access support for new computing hardware devices such as GPU and FPGA, so that the virtual machine can meet the needs of new high-performance computing, machine learning and other fields.

According to an aspect of the present invention, a method for implementing a giant virtual machine across physical machines is provided, which is characterized by comprising the following steps:

Step 1, the steps of creating a giant virtual machine;

Step 2, the interrupt processing steps of the giant virtual machine;

Step 3, the processing steps of equipment interruption;

The fourth step is a remote access step based on the high-speed shared memory access technology.

Preferably, the step one includes the following steps:

Step 11: The user sends a giant virtual machine creation instruction to the global virtual machine management software, including the type and number of CPU, memory and other devices required by the giant virtual machine;

Step 12: The global virtual machine management software parses the user request, discovers all physical computers, and notifies the main virtual machine management software of the parsed request according to the protocol;

Step 13: The main virtual machine management software creates a virtual machine instance locally, and creates the vCPU instance specified by the request;

Step 14: The main virtual machine management software creates a global address space of distributed shared memory according to the request; in the global address space, only part of the memory is located locally, and other memory is located remotely;

Step 15: The main virtual machine management software creates a local device instance according to the request;

Step 16: The main virtual machine management software returns the vCPU, memory, PCIE bus information to the global virtual machine management software, and enters the waiting state;

Step seventeen: the global virtual machine management software sends the information returned by the main virtual machine management software together with the user request to the virtual machine management software of the remote computer;

Step 18: The virtual machine management software of the remote computer creates a virtual machine instance locally, synchronizes the PCIE bus of the main virtual machine, creates a vCPU instance specified by the request, and sets the vCPU state;

Step 19: the virtual machine management software of the remote computer designates the memory as a part of the global distributed shared memory address space according to the request;

Step 20: The virtual machine management software of the remote computer creates a local IO device instance according to the request;

Step 21: The virtual machine management software of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software to enter a waiting state;

Step 22: The virtual machine management software of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software;

Step 23: The global virtual machine management software notifies the main virtual machine management software to start running the virtual machine.

Preferably, the second step includes the following steps:

Step 31: A certain core in a certain physical computer sends an interrupt to a certain core;

Step 32: The interrupt is sent to the interrupt route to determine the type of the interrupt; if the interrupt target and the sender are located in the same physical CPU, it is an interrupt between the same CPU core; if the interrupt target and the sender are located in the same physical CPU For different CPUs of the computer, it is a cross-CPU-core interrupt; if the target and sender of the interrupt are located on different physical computers, it is a cross-machine-core interrupt;

Step 33: Interrupt routing is forwarded according to the type of interrupt; for interrupts between the same CPU core and across CPU cores, the same CPU and NUMA interrupt mechanisms are used for delivery; It is responsible for delivering the interrupt to the target virtual machine management software; the target virtual machine management software injects the interrupt into the target core after receiving it.

Preferably, the step 3 includes the following steps:

Step 41: A device located in a physical computer sends an interrupt to a core;

Step 42: The interrupt is sent to the interrupt route to determine the type of the interrupt; if the interrupt target and the sender are located on the same physical computer, it is an interrupt on the same machine; if the interrupt target and the sender are located on different physical computers, It is interrupted across physical machine devices;

Step 43: The interrupt routing is forwarded according to the interrupt type; for the same-machine device interrupt, the device interrupt mechanism is used for delivery; for the cross-physical machine device interrupt, the interrupt is sent to the interrupt route, which is responsible for delivering the interrupt to the target The hypervisor; the target hypervisor injects the interrupt into the target core after receiving it.

Preferably, the step 4 includes the following steps:

Step 41: The application running on the virtual machine operating system accesses or modifies the page of a certain address;

Step 42: The hardware searches through the page table of the virtual machine operating system and the extended page table of the virtual machine. If a page missing occurs, exit to the virtual machine management software for processing;

Step 43: The virtual machine management software processes the page fault, and if it is found that the memory page is located on the remote computer, it sends a remote memory operation request to the shared memory route;

Step 44: The shared memory routing performs a remote memory operation on the target computer according to the routing table;

Step 45: The target computer returns the remote memory operation result to the shared memory route;

Step 46: The shared memory routing forwards the result returned by the remote memory operation to the virtual machine management software;

Step 47: The virtual machine management software returns the memory to the virtual machine operating system.

Compared with the prior art, the present invention has the following beneficial effects: the present invention can realize the virtualization of multiple physical computers, thereby providing one or more high-performance giant virtual machines with massive computing resources to meet the needs of high-performance computing. demand. The invention realizes that multiple physical machines provide computing, IO and other resources for one virtual machine. Remote memory access is performed between machines through remote high-speed memory access technology (such as RDMA), which realizes high-speed distributed shared memory. In addition, the present invention also creatively provides access support for new computing hardware devices such as GPU and FPGA, so that the virtual machine can meet the needs of new high-performance computing, machine learning and other fields. Aiming at large-scale distributed computing systems, the invention abstracts computing resources (CPU, GPU, FPGA) on the basis of distributed shared memory, realizes multiple virtual ones across machines, and presents a unified giant virtual machine with massive resources to the outside world. .

Description of drawings

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

FIG. 1 is a schematic diagram of the structure of the present invention.

FIG. 2 is a schematic diagram of a virtual PCIE bus.

Figure 3 is a schematic diagram of an inter-core interrupt.

Figure 4 is a schematic diagram of equipment interruption.

Figure 5 is a schematic diagram of a remote access process based on high-speed shared memory access technology.

FIG. 6 is a schematic diagram of application awareness.

Detailed ways

The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

The method for implementing the giant virtual machine (Giant VM) across physical machines of the present invention comprises the following steps:

Step 1, the steps of creating a giant virtual machine;

Step 2, the interrupt processing steps of the giant virtual machine;

Step 3, the processing steps of equipment interruption;

Step 4, a remote access step based on a high-speed shared memory access technology (such as RDMA).

Step 1 includes the following steps:

Step 11: The user sends a giant virtual machine creation instruction to the global virtual machine management software Coordinator, including the CPU (Central Processing Unit, central processing unit), memory, and other device types and numbers required by the giant virtual machine;

Step 12: The global virtual machine management software Coordinator parses the user request, discovers all physical computers, and notifies the master virtual machine management software (master) of the parsed request according to the protocol;

Step 13: The master virtual machine management software (master) creates a virtual machine instance locally, and creates a vCPU (virtual central processing unit) instance specified by the request. vCPUs are divided into local vCPUs and remote vCPUs. The local vCPUs will be executed on the local physical CPU, while the remote vCPUs will be synchronized to other physical computers for execution in subsequent steps; all vCPUs are linked to the virtual PCIE of the virtual machine instance. on the bus;

Step 14: The master virtual machine management software (master) creates the global address space of the distributed shared memory according to the request. Only part of the memory in the global address space is located locally, and other memory is located remotely;

Step 15: The master virtual machine management software (master) creates a local device instance, such as a GPU, according to the request, and mounts it on the PCIE bus of the virtual machine;

Step 16: The master virtual machine management software (master) returns the vCPU, memory, and PCIE bus information to the Coordinator, and enters the waiting state;

Step 17: The global virtual machine management software Coordinator sends the information returned by the master virtual machine management software (master) together with the user request to the virtual machine management software (worker) of the remote computer;

Step 18: The virtual machine management software (worker) of the remote computer creates a virtual machine instance locally, synchronizes the PCIE bus of the main virtual machine, creates the vCPU instance specified by the request, and sets the vCPU state;

Step 19: The virtual machine management software (worker) of the remote computer designates the memory as a part of the globally distributed shared memory address space according to the request;

Step 20: The virtual machine management software (worker) of the remote computer creates a local IO device instance, such as a GPU, according to the request, and mounts it on the PCIE bus of the virtual machine;

Step 21: The virtual machine management software (worker) of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software Coordinator, and enters a waiting state;

Step 22: The virtual machine management software (worker) of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software Coordinator;

Step 23: The global virtual machine management software Coordinator notifies the master virtual machine management software (master) to start running the virtual machine.

From the perspective of the giant virtual machine, after an interrupt is sent from the interrupt source, it is delivered to the interrupt target through the interrupt routing of the giant virtual machine. As shown in Figure 3, step 2 includes the following steps:

Step 31: A certain core (core) located in a certain physical computer sends an interrupt to a certain core (core).

Step 32: The interrupt is sent to the interrupt route, and the type of the interrupt is determined. If the interrupt target and the sender are located on the same physical CPU, it is an interrupt between CPU cores; if the interrupt target and the sender are located on different CPUs of the same physical computer, it is a cross-CPU core interrupt; if the interrupt target and If the sender is on a different physical computer, it is a cross-machine-core interrupt.

Step 33: The interruption route is forwarded according to the interruption type. Inter-CPU and inter-CPU-core interrupts are delivered by intra-CPU and NUMA interrupt mechanisms; inter-core inter-core interrupts are sent to the interrupt routing, which is responsible for delivering the interrupt to the target virtual machine management software. The target hypervisor receives the interrupt and injects it into the target core (core).

As shown in Figure 4, step 3 includes the following steps:

Step 41: A device located in a certain physical computer sends an interrupt to a certain core (core).

Step 42: The interrupt is sent to the interrupt route, and the type of the interrupt is determined. If the target and sender of the interruption are located on the same physical computer, it is an interruption of the same machine device; if the target and sender of the interruption are located on different physical computers, it is an interruption across physical machines.

Step 43: The interruption route is forwarded according to the interruption type. For the same-machine device interruption, the device interruption mechanism is used for delivery; for the cross-physical machine device interruption, the interruption is sent to the interruption route, which is responsible for delivering the interruption to the target virtual machine management software. The target hypervisor receives the interrupt and injects it into the target core (core).

From the perspective of a giant virtual machine, it has a uniform, contiguous physical memory address space. But in fact, the physical memory to which this physical memory address space is mapped is located on multiple physical computers. When the operating system or application in the giant virtual machine accesses the memory, if the memory corresponding to the memory address is located on the remote machine, the request to access the memory will be sent to the memory route, and the memory route will send the request to the target, and return after the result is obtained. to the requester. As shown in Figure 5, step 4 includes the following steps:

Step 41: The application running on the virtual machine operating system accesses or modifies the page of a certain address;

Step 42: The hardware searches through the page table of the virtual machine operating system and the virtual machine extended page table (such as EPT), and if a page missing occurs, exit to the virtual machine management software for processing;

Step 43: The virtual machine management software processes the page fault, and if it is found that the memory page is located on the remote computer, it sends a remote memory operation request to the shared memory route;

Step 44: The shared memory routing performs a remote memory operation on the target computer according to the routing table;

Step 45: The target computer returns the remote memory operation result to the shared memory route;

Step 46: The shared memory routing forwards the result returned by the remote memory operation to the virtual machine management software;

Step 47: The virtual machine management software returns the memory to the virtual machine operating system.

At the application level, because the underlying resources are virtualized, they can only perceive that they are running on a giant virtual machine with massive resources. Under the giant virtual machine, the virtual machine management software running on different physical computers cooperates to provide computing resources for the upper layer, as shown in Figure 6.

The created giant virtual machine has the following characteristics:

First, all physical computers are connected to each other through a high-speed network, and the global virtual machine management software Coordinator is responsible for discovery, control and management. The virtual machine management software with one and only one physical machine selected by the Coordinator is used as the master, and the virtual machine management software on other physical machines is used as the worker.

Second, the master and the worker discover each other through the Coordinator, after which they can communicate through the Coordinator or communicate directly.

Third, all physical computer nodes of the giant virtual machine maintain the same or different numbers of vCPUs. If the vCPU is local, it belongs to the local vCPU and executes locally. If remote, execution is guaranteed by the remote computer. Inter-core interrupts for vCPUs can be delivered across cores, CPUs, and machines through interrupt routing.

Fourth, all physical computers of the giant virtual machine use high-speed shared memory access technology to realize distributed shared memory.

Fifth, all physical computers of the giant virtual machine use shared storage technology to realize distributed shared storage.

The giant virtual machine has a unique virtual PCIE (including PCI) bus, and the devices of each physical machine will be mounted on the bus, as shown in Figure 2. In addition to CPUs, new computing hardware devices such as GPUs and FPGAs can be mounted on the virtual PCIE bus to provide services for giant virtual machines.

The specific deployment example is a cluster composed of three ordinary servers, each of which is equipped with a network card that supports InfiniBand. The servers are connected to the central InfiniBand switch via fiber. The invention is not limited by the type and number of servers, and can be extended to more than three heterogeneous servers to form a cluster.

Each server is equipped with Ubuntu Server 16.04.1LTS 64bit, and is equipped with two CPUs with a total of 56 cores and 64GB of memory. The specific development of the present invention is based on the source code version of QEMU 2.8.1.1 and Linux kernel 4.8.10 as an illustration, and is also applicable to other virtual machine managers and kernels of other versions of Linux.

As shown in Figure 1, different servers use shared storage to maintain consistent disk images, and use RDMA-based distributed shared memory to maintain consistent memory. The Coordinator is responsible for discovering, coordinating and managing each server. After receiving the user request, it sends the request to the main virtual machine management software and the remote virtual machine management software to create a giant virtual machine across servers.

The giant virtual machine has 168 vCPUs, each vCPU corresponds to a physical core, of which 56 vCPUs run on the physical core of the machine, and the remaining 112 vCPUs run on the other two remote servers. The giant virtual machine has 192GB of distributed shared memory, of which 64GB is local memory, and the remaining 128GB is remote memory, which can be accessed at high speed through RDMA. At the same time, the virtual machine owns and is able to use computing, IO and other devices such as GPU, FPGA, etc. located on different computers.

The architecture is completely transparent to users, and users can create, change and destroy giant virtual machines by directly sending instructions to the Coordinator. Applications running on giant virtual machines can transparently use the vast amount of virtualized resources.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and those skilled in the art can make various variations or modifications within the scope of the claims, which do not affect the essential content of the present invention.

Claims (4)

1. a giant virtual machine implementation method across physical machines, is characterized in that, comprises the following steps: Step 1, the steps of creating a giant virtual machine; Step 2, the interrupt processing steps of the giant virtual machine; Step 3, the processing steps of equipment interruption; Step 4, remote access step based on high-speed shared memory access technology; The first step includes the following steps: Step 11: The user sends a giant virtual machine creation instruction to the global virtual machine management software, including the type and number of CPU, memory and other devices required by the giant virtual machine; Step 12: The global virtual machine management software parses the user request, discovers all physical computers, and notifies the main virtual machine management software of the parsed request according to the protocol; Step 13: The main virtual machine management software creates a virtual machine instance locally, and creates the vCPU instance specified by the request; Step 14: The main virtual machine management software creates a global address space of distributed shared memory according to the request; in the global address space, only part of the memory is located locally, and other memory is located remotely; Step 15: The main virtual machine management software creates a local device instance according to the request; Step 16: The main virtual machine management software returns the vCPU, memory, PCIE bus information to the global virtual machine management software, and enters the waiting state; Step seventeen: the global virtual machine management software sends the information returned by the main virtual machine management software together with the user request to the virtual machine management software of the remote computer; Step 18: The virtual machine management software of the remote computer creates a virtual machine instance locally, synchronizes the PCIE bus of the main virtual machine, creates a vCPU instance specified by the request, and sets the vCPU state; Step 19: the virtual machine management software of the remote computer designates the memory as a part of the global distributed shared memory address space according to the request; Step 20: The virtual machine management software of the remote computer creates a local IO device instance according to the request; Step 21: The virtual machine management software of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software to enter a waiting state; Step 22: The virtual machine management software of the remote computer returns the vCPU, memory, and PCIE bus information to the global virtual machine management software; Step 23: The global virtual machine management software notifies the main virtual machine management software to start running the virtual machine. 2. The method for implementing a giant virtual machine across physical machines according to claim 1, wherein the step 2 comprises the following steps: Step 31: A certain core in a certain physical computer sends an interrupt to a certain core; Step 32: The interrupt is sent to the interrupt route to determine the type of the interrupt; if the interrupt target and the sender are located in the same physical CPU, it is an interrupt between the same CPU core; if the interrupt target and the sender are located in the same physical CPU For different CPUs of the computer, it is a cross-CPU-core interrupt; if the target and sender of the interrupt are located on different physical computers, it is a cross-machine-core interrupt; Step 33: Interrupt routing is forwarded according to the interrupt type; for interrupts between the same CPU core and across CPU cores, the same CPU and NUMA interrupt mechanisms are used for delivery; for inter-machine core interrupts, it is sent to the interrupt routing, It is responsible for delivering the interrupt to the target virtual machine management software; the target virtual machine management software injects the interrupt into the target core after receiving it. 3. The method for implementing a giant virtual machine across physical machines according to claim 1, wherein the step 3 comprises the following steps: Step 41: A device located in a physical computer sends an interrupt to a core; Step 42: The interrupt is sent to the interrupt route to determine the type of the interrupt; if the interrupt target and the sender are located on the same physical computer, it is an interrupt on the same machine device; if the interrupt target and the sender are located on different physical computers, It is interrupted across physical machine devices; Step 43: The interrupt routing is forwarded according to the interrupt type; for the same-machine device interrupt, the device interrupt mechanism is used for delivery; for the cross-physical machine device interrupt, the interrupt is sent to the interrupt route, which is responsible for delivering the interrupt to the target The hypervisor; the target hypervisor injects the interrupt into the target core after receiving it. 4. The method for implementing a giant virtual machine across physical machines according to claim 1, wherein the step 4 comprises the following steps: Step 41: The application running on the virtual machine operating system accesses or modifies the page of a certain address; Step 42: The hardware searches through the page table of the virtual machine operating system and the extended page table of the virtual machine. If a page missing occurs, exit to the virtual machine management software for processing; Step 43: The virtual machine management software handles the page fault, and if the current page fault is found on the remote computer, it sends a remote memory operation request to the shared memory route; Step 44: The shared memory routing performs remote memory operations on the target computer according to the routing table; Step 45: The target computer returns the remote memory operation result to the shared memory route; Step 46: The shared memory routing forwards the result returned by the remote memory operation to the virtual machine management software; Step 47: The virtual machine management software returns the memory to the virtual machine operating system.

Images