CN106970824B - A bandwidth-aware virtual machine migration compression method and system - Google Patents

Abstract

The invention discloses a virtual machine migration and compression method and system based on bandwidth perception, belonging to the field of computer virtualization. The method of the present invention detects the network bandwidth at a preset frequency, uses the bandwidth and each pair of compression rate and compression speed in the compression policy table to calculate the migration speed, and selects the compression method corresponding to the maximum migration speed for compression migration. Before performing memory data compression First, multiple memory pages are combined into one data packet, and then the entire data packet is compressed and migrated until the compression migration is completed; at the same time, the present invention also provides a bandwidth-aware virtual machine migration and compression system. The technical solution of the present invention is based on bandwidth The dynamic adjustment of the compression method enables the migration system to obtain a greater migration speed, thereby obtaining a shorter migration time, reducing the amount of data transmission, and saving network resources.

Description

A bandwidth-aware virtual machine migration compression method and system

technical field

The invention belongs to the field of computer virtualization, and more specifically relates to a method and system for virtual machine migration and compression based on bandwidth perception.

Background technique

In recent years, with the development of cloud computing and virtualization technologies, virtual machines are being more and more widely deployed in data centers and cluster environments. Since the virtual machine can abstractly simulate computer resources and simulate virtual hardware resources on the basis of existing computer hardware resources, it has many advantages such as simulating different platforms, improving computer resource utilization, easy management, and application isolation.

Virtual machine migration refers to the migration of virtual machines between different physical hosts while ensuring the normal operation of services in the virtual machine. In order to guarantee the availability of virtual machine services during the migration process, the migration process has only a very short downtime. Since the downtime switching time is very short, users do not feel service interruption, so the migration process is transparent to users. Virtual machine migration is applicable to many scenarios such as data center load balancing, energy saving, and system maintenance, so it is a very important feature of virtualization technology.

Virtual machine migration is usually carried out in the local area network. In this environment, the virtual machine uses shared storage to access the external memory. Therefore, it is only necessary to migrate the memory data of the virtual machine and the status of virtual cpu and other devices, and the memory of the virtual machine Accounts for the vast majority of the data that needs to be migrated. The pre-copy method is the most important migration algorithm widely adopted by various virtualization platforms. The pre-copy migration process is to first copy the complete virtual machine memory image to the target host. During this process, since the virtual machine is still running, some of its memory pages will be modified, and these modified memory dirty pages need to be transferred to the target host again in the next iteration. After that, the dirty pages generated in each round of iteration need to be retransmitted in the next round, so as to ensure the consistency of the memory state. After multiple rounds of iterations, the number of remaining dirty pages is relatively small, and when the preset threshold is reached, the shutdown copy can be performed to end the iterative copy process.

Although the existing pre-copy migration method can achieve short downtime, there are the following problems: because the memory data needs multiple rounds of iterative transmission, the network transmission data volume is large, and the migration time is also long; in addition, if the virtual machine If the load running in the virtual machine is memory-intensive, the virtual machine memory may become dirty too quickly. At this time, the pre-copy migration method cannot converge normally and enter the shutdown copy stage, and the migration process cannot be completed normally. These problems have greatly affected the performance of virtual machine migration, resulting in that the expected effect cannot be achieved when using the virtual machine migration technology in the data center.

Contents of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides a bandwidth-aware virtual machine migration compression method and system, the purpose of which is to first detect the compression ratio and compression ratio of various compression methods for various typical loads. Speed, establish a compression cable strategy table, and then perceive the bandwidth at a predetermined frequency, calculate the migration speed corresponding to each compression method under the current bandwidth, and use the compression method corresponding to the maximum migration speed to perform compression migration, thereby solving the problem of conventional compression migration technology .

In order to achieve the above object, according to one aspect of the present invention, a virtual machine migration and compression method based on bandwidth awareness is provided. The method detects the network bandwidth at a preset frequency, and utilizes each pair of compression ratio and Compression speed calculates the migration speed, and selects the compression method corresponding to the maximum migration speed to compress and migrate the current memory data of the virtual machine.

Further, the method of the present invention specifically includes the following steps:

(1) Monitor the network bandwidth to obtain the real-time network bandwidth St available for virtual machine migration;

(2) Using the compression rate ρ _i and the compression speed Sc _i corresponding to various compression methods in the compression strategy table to calculate the migration speed Smgt _i ,

Smgt _i =min(Sc _i ,St×ρ _i ),

Get multiple migration speeds, and compare them to get the maximum migration speed;

(3) Find out the compression rate and compression speed to obtain the maximum migration speed, and use the corresponding compression method in the compression policy table to compress and migrate the current memory data of the virtual machine.

Further, the compression strategy table is pre-obtained by the following method:

In the data center operating environment, select multiple typical loads to run in the virtual machine in sequence, and use various compression methods to perform compression detection of memory data. Each compression method obtains a pair of compression ratio and compression speed. All compression methods and other The corresponding compression rate and compression speed form a compression strategy table.

Further, the method for obtaining the compression strategy table specifically includes the following sub-steps:

(31) Select a typical load in a data center and run it in a virtual machine;

(32) Select a compression method for compression migration, and set all memory pages as dirty pages after each round of compression, and replace another compression method for compression migration, m compression methods iterate m rounds in total; Record the total compression time required for each round of compression and the size of the compressed data;

(33) replace another kind of typical load, and return to step (31), until the typical load compression in n kinds of data centers is completed;

(34) Calculate the compressibility ρ _ij of the i-th compression method for the j-th load,

ρ _ij = data size before compression/data size after compression,

Calculate the compression speed Sc _ij of the i-th compression method for the j-th load,

Sc _ij = data size before compression/total compression time;

Among them, 1≤i≤m; 1≤j≤n;

(35) Calculate the average compression ratio ρ _i of the i-th compression method for n types of loads,

ρ _i =(ρ _i1 +ρ _i2 +...+ρ _in )/n,

Calculate the average compression speed Sc _i of the i-th compression method for n types of loads,

Sc _i =(Sc _i1 +Sc _i2 +...+Sc _in )/n

A total of m pairs of average compression ratio ρ _i and average compression speed Sc _i corresponding to m compression methods are obtained, and the compression method and the corresponding average compression ratio and average compression speed together constitute a compression strategy table.

Further, the inventive method also includes a merging step:

Combining step: Before performing memory data compression, multiple memory pages are combined into one data package, and then the data package is compressed as a whole.

According to another aspect of the present invention, a virtual machine migration compression system based on bandwidth awareness is provided, the system is used to detect network bandwidth at a preset frequency, and utilize each pair of compression ratio and compression speed in the bandwidth and compression policy table Calculate the migration speed, and select the compression method corresponding to the maximum migration speed to compress and migrate the current memory data of the virtual machine.

Further, the system of the present invention specifically includes the following parts:

The bandwidth detection module is used to monitor the network bandwidth and obtain the real-time network bandwidth St available for virtual machine migration;

The migration speed calculation module is used to calculate the migration speed Smgt _i using the compression rate ρ _i and the compression speed Sc _i corresponding to various compression methods in the compression strategy table,

Smgt _i =min(Sc _i ,St×ρ _i ),

Get multiple migration speeds, and compare them to get the maximum migration speed;

The compression migration module is used to find out the compression rate and compression speed for obtaining the maximum migration speed, and use the corresponding compression method in the compression policy table to perform compression migration on the current memory data of the virtual machine.

Further, the compression strategy table is obtained by using the following modules in advance:

The compression strategy table module is used to select multiple typical loads to run in the virtual machine in turn in the data center operating environment, and use various compression methods to perform compression detection of memory data. Each compression method obtains a pair of compression ratio and compression Speed, all compression methods and their corresponding compression ratios and compression speeds form the compression strategy table.

Further, the compression policy table module specifically includes the following parts:

The load running unit is used to select a typical load in the data center and run it in the virtual machine;

The iterative compression unit is used to select a compression method for compression migration, and set all memory pages as dirty pages after each round of compression, and replace another compression method for compression migration, m compression methods are iterated m rounds; record the total compression time required for each round of compression and the size of the compressed data;

Replace the load unit for another type of typical load, return to the load running unit until the compression of the typical load in n types of data centers is completed;

A calculation unit, used to calculate the compression rate ρ _ij of the i-th compression method for the j-th load,

ρ _ij = data size before compression/data size after compression,

Calculate the compression speed Sc _ij of the i-th compression method for the j-th load,

Sc _ij = data size before compression/total compression time;

Among them, 1≤i≤m; 1≤j≤n;

The strategy table construction unit is used to calculate the average compression rate ρ _i of the i-th compression method for n types of loads,

ρ _i =(ρ _i1 +ρ _i2 +...+ρ _in )/n,

Calculate the average compression speed Sc _i of the i-th compression method for n types of loads,

Sc _i =(Sc _i1 +Sc _i2 +...+Sc _in )/n

A total of m pairs of average compression ratio ρ _i and average compression speed Sc _i corresponding to m compression methods are obtained, and the compression method and the corresponding average compression ratio and average compression speed together constitute a compression strategy table.

Further, the system of the present invention also includes a merge module:

The merging module is used for merging multiple memory pages into a data package before compressing the memory data, and then compressing the data package as a whole.

Generally speaking, compared with the prior art, the above technical solution conceived by the present invention has the following technical characteristics and beneficial effects:

(1) The present invention dynamically adjusts the compression method according to the real-time bandwidth of the migration. Compared with using a single fixed compression method, a shorter migration time can be obtained. When the network bandwidth is higher, the present invention will select the fast and lower compression method. Compression method; on the contrary, when the bandwidth is low, a slow compression method with a high compression rate will be selected. Such a dynamic adjustment process can enable the migration system to obtain a greater throughput rate, thereby obtaining a shorter migration time;

(2) Since the data content in the memory of the virtual machine differs greatly when different loads are running in the virtual machine, the compression rate and speed obtained by the same compression method are different when compressing the memory of different loads. Invention obtains the average compression ratio and compression speed through several typical loads, thus forms the method of compression strategy table, avoids adopting different compression strategy tables for different loads, which makes the specific implementation of the present invention more feasible and convenient;

(3) The present invention packs a plurality of virtual machine memory pages and compresses the method, excavated the compression window of the compression algorithm is much larger than the characteristics of a single memory page, and the compression algorithm is to look for redundant data in the scope of the compression window so as to compress, Therefore, the larger compression granularity proposed by the present invention can further improve the compression rate of memory data, thereby reducing the amount of data transmission and saving network resources.

Description of drawings

Fig. 1 is a schematic flow sheet of the inventive method;

Fig. 2 is the system structural diagram of the method embodiment of the present invention;

Fig. 3 is a schematic diagram of the change curve of the compression rate and the compression speed of the LZ4 compression algorithm;

Figure 4a is a schematic diagram of the migration time comparison between the method of the present invention and the existing pre-copy migration;

Fig. 4b is a schematic diagram of a comparison of the migration data volume between the method of the present invention and the existing pre-copy migration.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

As shown in Figure 1, the method of the present invention detects the network bandwidth at a preset frequency, calculates the migration speed by using each pair of compression rate and compression speed in the bandwidth and compression strategy table, and selects the compression method corresponding to the maximum migration speed for the current virtual machine. Memory data is compressed and migrated; at the migration-out end, multiple memory pages are combined into one data packet before compression, and then the data packet is compressed as a whole; at the migration-in end, the received network data packets are decompressed to obtain a virtual machine memory pages.

Specifically divided into the following sub-steps:

(1) Monitor the network bandwidth to obtain the real-time network bandwidth St available for virtual machine migration;

(2) Using the compression rate ρ _i and the compression speed Sc _i corresponding to various compression methods in the compression strategy table to calculate the migration speed Smgt _i ,

Smgt _i =min(Sc _i ,St×ρ _i ),

Get multiple migration speeds, and compare them to get the maximum migration speed;

(3) Find out the compression ratio and the compression speed that obtain the maximum migration speed, and use its corresponding compression method in the compression strategy table to compress and migrate the current memory data of the virtual machine;

At the migration end, set a buffer, and copy the virtual machine memory pages to be transferred to the buffer first. When the buffer is full, compress the entire buffer once, and transfer the compressed data packets to the migration end. ;

On the migration-in side, set a buffer with the same size as the migration-out side, put the memory pages obtained after decompressing the received data packets into the buffer, and then put each memory page into the corresponding virtual machine address in space;

The compression method selected in this way is the one that maximizes the migration speed at the current stage; the compression method is dynamically adjusted according to the network bandwidth so that the compression method with the highest throughput at the current stage is used at all stages of the entire migration process, thereby significantly shortening the migration time.

The compression strategy table mentioned in the above content is pre-obtained by the following method: select multiple typical loads to run in the virtual machine in sequence in the data center operating environment, and use various compression methods to perform compression detection of memory data. Each compression method Get a pair of compression ratio and compression speed, all compression methods and their corresponding compression ratio and compression speed form a compression strategy table

Specifically divided into the following steps:

(31) Select a typical load in a data center and run it in a virtual machine;

(32) Select a compression method for compression migration. Before compression, set a buffer, and first copy the virtual machine memory pages to be transferred to the buffer. When the buffer is full, compress the entire buffer once. And after each round of compression, set all memory pages as dirty pages, and replace another compression method for compression migration. m compression methods iterate m rounds in total; record the total compression time required for each round of compression and the post-compression data size;

In this way, the modification of the pre-copy migration can make all the memory pages in the virtual machine address space undergo multiple rounds of transmission, and in each round of transmission, a compression method is selected to measure the compression rate and compression speed, so that only one migration is required Operation can measure the compression rate and compression speed of all compression methods for a certain load, which simplifies the process of obtaining the compression strategy table;

Specifically in the embodiment, the LZ4 algorithm is adopted, and 16 different LZ4 acceleration values are selected, which are 1, 3, 5, 7...31 respectively, thus forming 16 compression methods, because two adjacent LZ4 acceleration values ( Such as 1 and 2) the compression rate and the compression speed difference produced are small, so this embodiment chooses to use 2 as the number of steps to select the acceleration value;

(33) replace another kind of typical load, and return to step (31), until the typical load compression in n kinds of data centers is completed;

(34) Calculate the compressibility ρ _ij of the i-th compression method for the j-th load,

ρ _ij = data size before compression/data size after compression,

Calculate the compression speed Sc _ij of the i-th compression method for the j-th load,

Sc _ij = data size before compression/total compression time;

Among them, 1≤i≤m; 1≤j≤n;

(35) Calculate the average compression ratio ρ _i of the i-th compression method for n types of loads,

ρ _i =(ρ _i1 +ρ _i2 +...+ρ _in )/n,

Calculate the average compression speed Sc _i of the i-th compression method for n types of loads,

Sc _i =(Sc _i1 +Sc _i2 +...+Sc _in )/n

A total of m pairs of average compression ratio ρ _i and average compression speed Sc _i corresponding to m compression methods are obtained, and the compression method and the corresponding average compression ratio and average compression speed together constitute a compression strategy table.

As shown in Figure 2, in this embodiment, the embodiment of the present invention is implemented on the KVM/QEMU open source virtualization platform, and at the migration end, the bandwidth monitoring module obtains the current migration network bandwidth once per second , submitted to the bandwidth-aware compression module; after the compression module obtains the network bandwidth, it uses the compression strategy table to dynamically adjust the compression method; at the same time, at the migration end, a buffer is set, and the virtual machine memory page to be transferred is first copied to the buffer In the process, when the buffer is full, pack and compress the data in the buffer; transmit the compressed data packets to the inbound end; after the inbound end receives the packed and compressed data into the buffer, the decompression module decompresses it , and submit the obtained memory page to the corresponding virtual machine address space.

The method for dynamically adjusting the compression according to the real-time migration bandwidth proposed by the present invention can greatly improve the throughput rate of the migration system, thereby significantly shortening the migration time. Different compression algorithms have different compression ratios and compression speeds. Generally speaking, compression algorithms with high compression ratios have lower compression speeds, and vice versa; in addition, many compression algorithms provide parameters for users to choose, so that the difference between compression ratio and compression speed make adjustments. Taking the LZ4 compression algorithm used in the actual implementation of the present invention as an example, the algorithm provides an acceleration value parameter. FIG. 3 shows the change of the compression effect of LZ4 on the virtual machine memory when different acceleration values are used. As the acceleration value becomes larger, the compression speed is continuously increased, at the expense of a continuous decrease in the compression ratio.

It can be seen that under a certain bandwidth, compression methods with different compression ratios and compression speeds are used in the migration. In this embodiment, the LZ4 algorithm with different acceleration values is used, and the system migration speeds are different. The method proposed by the invention can find out the compression method that can bring the maximum migration speed to the migration system under a certain bandwidth, and dynamically select the optimal compression method as the bandwidth changes, so as to obtain the shortest migration time.

The present invention obtains the average compression ratio and compression speed through several typical loads, thereby composes the compression strategy table, avoids the use of different compression strategy tables for different loads, and makes the specific implementation of the present invention more convenient and feasible. When different loads are running in the virtual machine, the data content in the memory is different. Therefore, when using LZ4 with a certain acceleration value to compress the memory of different loads, the obtained compression ratio and compression speed are different, which brings The migration speed of different systems is also different. However, our experiments show that different loads achieve the maximum system migration speed at the same or close to the acceleration value. Therefore, the present invention proposes to select some typical loads, measure their compression ratio and compression speed under different compression methods, average these data, obtain the average compression speed and compression ratio, form a compression strategy table, and calculate a certain Optimal compression method under a bandwidth. Experiments show that the compression strategy table thus obtained is suitable for all loads within a certain error range. This makes the present invention only need to store a set of compression policy tables during implementation, which simplifies system implementation and improves the feasibility of the present invention.

As shown in Figure 4a, compared with the migration time of the existing pre-copy migration using the method of the present invention, the method of the present invention takes less time to complete the migration;

As shown in FIG. 4 b , compared with the migration data volume of the existing pre-copy migration using the method of the present invention, the migration and transmission data volume of the method of the present invention is smaller.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. within range.

Claims (4)

1. A bandwidth-aware virtual machine migration compression method, characterized in that the method of the present invention is: detect network bandwidth at a preset frequency, and calculate migration speed by using each pair of compression ratio and compression speed in the bandwidth and compression strategy table , select the compression method corresponding to the maximum migration speed to compress and migrate the current memory data of the virtual machine; Specifically include the following steps: (1) Monitor the network bandwidth to obtain the real-time network bandwidth St available for virtual machine migration; (2) Using the compression rate ρ _i and the compression speed Sc _i corresponding to various compression methods in the compression strategy table to calculate the migration speed Smgt _i , Smgt _i =min(Sc _i ,St×ρ _i ), Get multiple migration speeds, and compare them to get the maximum migration speed; (3) Find out the compression ratio and the compression speed that obtain the maximum migration speed, and use its corresponding compression method in the compression strategy table to compress and migrate the current memory data of the virtual machine; The compression policy table is obtained in advance by the following method: In the data center operating environment, select multiple typical loads to run in the virtual machine in sequence, and use various compression methods to perform compression detection of memory data. Each compression method obtains a pair of compression ratio and compression speed. All compression methods and other The corresponding compression rate and compression speed form a compression strategy table; The method for obtaining the compression policy table specifically includes the following sub-steps: (31) Select a typical load in a data center and run it in a virtual machine; (32) Select a compression method for compression migration, and set all memory pages as dirty pages after each round of compression, and replace another compression method for compression migration, m compression methods iterate m rounds in total; Record the total compression time required for each round of compression and the size of the compressed data; (33) replace another kind of typical load, and return to step (31), until the typical load compression in n kinds of data centers is completed; (34) Calculate the compressibility ρ _ij of the i-th compression method for the j-th load, ρ _ij = data size before compression/data size after compression, Calculate the compression speed Sc _ij of the i-th compression method for the j-th load, Sc _ij = data size before compression/total compression time; Among them, 1≤i≤m; 1≤j≤n; (35) Calculate the average compression ratio ρ _i of the i-th compression method for n types of loads, ρ _i =(ρ _i1 +ρ _i2 +...+ρ _in )/n, Calculate the average compression speed Sc _i of the i-th compression method for n types of loads, Sc _i =(Sc _i1 +Sc _i2 +...+Sc _in )/n A total of m pairs of average compression ratio ρ _i and average compression speed Sc _i corresponding to m compression methods are obtained, and the compression method and the corresponding average compression ratio and average compression speed together constitute a compression strategy table. 2. A kind of bandwidth-aware based virtual machine migration and compression method according to claim 1, characterized in that, the method of the present invention also includes a merging step: Combining step: Before performing memory data compression, multiple memory pages are combined into one data package, and then the data package is compressed as a whole. 3. A virtual machine migration compression system based on bandwidth awareness, characterized in that: the system is used to detect network bandwidth at a preset frequency, and utilizes each pair of compression ratio and compression speed in the bandwidth and compression strategy table to calculate migration speed, Select the compression method corresponding to the maximum migration speed to compress and migrate the current memory data of the virtual machine; the details include the following parts: The bandwidth detection module is used to monitor the network bandwidth and obtain the real-time network bandwidth St available for virtual machine migration; The migration speed calculation module is used to calculate the migration speed Smgt _i using the compression rate ρ _i and the compression speed Sc _i corresponding to various compression methods in the compression strategy table, Smgt _i =min(Sc _i ,St×ρ _i ), Get multiple migration speeds, and compare them to get the maximum migration speed; The compression migration module is used to find out the compression rate and compression speed for obtaining the maximum migration speed, and use the corresponding compression method in the compression strategy table to perform compression migration on the current memory data of the virtual machine; The compression strategy table is obtained by using the following modules in advance: The compression policy table module is used to select multiple typical loads to run in the virtual machine in sequence in the data center operating environment, and use various compression methods to perform compression detection of memory data. Each compression method obtains a pair of compression ratio and compression Speed, all compression methods and their corresponding compression ratios and compression speeds form a compression strategy table; The compression strategy table module specifically includes the following parts: The load running unit is used to select a typical load in the data center and run it in the virtual machine; The iterative compression unit is used to select a compression method for compression migration, and set all memory pages as dirty pages after each round of compression, and replace another compression method for compression migration, m compression methods are iterated m rounds; record the total compression time required for each round of compression and the size of the compressed data; Replace the load unit for another type of typical load, return to the load running unit until the compression of the typical load in n types of data centers is completed; A calculation unit, used to calculate the compression rate ρ _ij of the i-th compression method for the j-th load, ρ _ij = data size before compression/data size after compression, Calculate the compression speed Sc _ij of the i-th compression method for the j-th load, Sc _ij = data size before compression/total compression time; Among them, 1≤i≤m; 1≤j≤n; The strategy table construction unit is used to calculate the average compression rate ρ _i of the i-th compression method for n types of loads, ρ _i =(ρ _i1 +ρ _i2 +...+ρ _in )/n, Calculate the average compression speed Sc _i of the i-th compression method for n types of loads, Sc _i =(Sc _i1 +Sc _i2 +...+Sc _in )/n A total of m pairs of average compression ratio ρ _i and average compression speed Sc _i corresponding to m compression methods are obtained, and the compression method and the corresponding average compression ratio and average compression speed together constitute a compression strategy table. 4. A kind of bandwidth-aware based virtual machine migration and compression system according to claim 3, characterized in that, the system of the present invention also includes a merging module: The merging module is used for merging multiple memory pages into a data package before compressing the memory data, and then compressing the data package as a whole.