CN106648469B - A cache data processing method, device and storage controller - Google Patents

Abstract

A data processing method, a corresponding device and a storage controller are disclosed. Wherein, the data processing method includes: obtaining data compression ratio D, deduplication ratio C, average stripe filling rate I, target stripe size S in the cache, and valid data size W in the target stripe; item parameter, using the formula

, obtain the amount of eliminated data N, and perform one of the following operations according to the value of N: in the case of T _min ≤ N ≤ T _max , write data of size N into the target The striped data is eliminated to the storage medium, wherein when T _min is a preset first threshold value, T _max is a preset second threshold value; in the case of N>T _max , data of size N is written into the Target stripe; in the case of N<T _min , the data of the target stripe is eliminated to the storage medium. Using this scheme, a more reasonable design is made for the time point when the cached data is downloaded to the disk.

Description

A cache data processing method, device and storage controller

technical field

The present invention relates to inventing storage technology, in particular to the field of logs of storage technology.

Background technique

Cache (cache) is an important technology used to solve the speed mismatch between high-speed and low-speed devices. It is widely used in various fields such as storage systems, which can reduce application response time and improve efficiency.

In a storage system, in order to improve the performance of the storage system, a cache layer (also called write-back cache, write-back cache) is usually added between the processing device and the hard disk. The cache layer can improve the system read and write performance, so sometimes the cache layer is also called the performance layer.

The medium used for the cache is, for example, a solid state disk (SSD), whose read and write performance is better than that of a persistent storage medium (such as a hard disk). Write requests from host applications can be processed more quickly than hard drives. The storage capacity of the cache is limited, so when the free space is insufficient, a part of the cache medium data needs to be eliminated to the hard disk according to the elimination algorithm, so as to free up space for subsequent write requests. This part of the data sent from the cache to the hard disk is called dirty data.

In order to make the cache have enough free space as soon as possible, it is necessary to eliminate the dirty data to the hard disk as soon as possible when the free space is insufficient. Otherwise, it will be difficult for the write request to be processed in time.

In the storage field, dirty data in the cache is usually delivered to the hard disk in stripes. Striping is a logical concept, similar to a container for data. If the delivered data is less than one strip, it means that there are only some valid data in the strip, and there are free positions in the strip. See the strip in Figure 1, which is only 50% utilized. This stripe that is not "filled" with valid data, like a stripe that is "filled" with valid data, will occupy the storage space of a complete stripe on the hard disk, and will generate a set of metadata. Therefore, if the proportion of valid data and free space in the stripe is larger. You can save more hard disk storage space and reduce the amount of metadata. If the stripe is completely "filled" with valid data and then distributed, the hard disk storage space can be maximized and the least metadata can be generated.

It can be seen that there are two contradictory requirements for the elimination of dirty data: considering the need to obtain more free cache space as soon as possible, dirty data needs to be eliminated to the hard disk as soon as possible, and it is best to perform the elimination operation immediately once the free space is insufficient; and Considering the need to improve hard disk utilization and reduce metadata, it is necessary to fill up the stripes as much as possible and then eliminate the dirty data to the hard disk. How to find a balance between the two is a problem that the industry needs to solve.

This problem is not well solved in the prior art. The number of reads and writes per second (IOPS) jitter is too large, and the hard disk storage space is insufficiently utilized.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a cache data processing method, device, and storage controller, which can alleviate severe fluctuations in IOPS and make the performance of the storage system more stable.

获取淘汰数据量N，在T_min≤N≤T_max的情况下，将大小为N的数据写入所述目标分条，把所述目标分条的数据淘汰给存储介质，其中当T_min是预设的第一阈值，T_max是预设的第二阈值。在另外一种可选方式中，在N＞T_max的情况下，将大小为N的数据写入所述目标分条。在另外一种可选方式中，在N＜T_min的情况下，把所述目标分条的数据淘汰给所述存储介质。这三种可选方式都是可选的，执行其中任意一个即可形成一个完整的流程。基于该方法，在对缓存没有明显影响的情况下，分条容纳尽量多的数据后再淘汰到存储介质中。在尽早获得空闲的缓存空间以及提高磁盘利用率这两个互相矛盾的需求之间获得平衡。使得缓存数据更平滑的淘汰给存储介质，减少了IOPS严重波动的情况，使存储系统更加稳定。In a first aspect of the present invention, the present invention provides a cache data processing method, the method comprising: obtaining data compression ratio D, deduplication ratio C, average stripe filling rate I, target stripe size S in the cache, and all Describe the effective data size W in the target strip; according to the formula

Acquire the amount of eliminated data N, in the case of T _min ≤ N ≤ T _max , write the data of size N into the target stripe, and eliminate the data of the target stripe to the storage medium, where when T _min is A preset first threshold, T _max is a preset second threshold. In another optional manner, in the case of N>T _max , data of size N is written into the target stripe. In another optional manner, in the case of N<T _min , the data of the target stripe is eliminated to the storage medium. These three optional methods are all optional, and a complete process can be formed by executing any one of them. Based on this method, under the condition that there is no obvious impact on the cache, the stripe accommodates as much data as possible before being eliminated into the storage medium. Strike a balance between the conflicting needs of getting free cache space early and improving disk utilization. The cached data is eliminated to the storage medium more smoothly, which reduces the severe fluctuation of IOPS and makes the storage system more stable.

In a first possible implementation manner of the first aspect, after writing data of size N into the target stripe, the method further includes: after the target stripe is filled with subsequent data, writing the target stripe to the target stripe. Stripe out to the storage medium. This possible implementation proposes a solution for eliminating data in the case of N>T _m .

In a second possible implementation manner of the first aspect, according to the method for eliminating cached data according to claim 1, the D, C, and I are statistical values, and their initial values are all 1.

In a third possible implementation manner of the first aspect, S is described by the number of the smallest unit (or an integer multiple of the smallest unit) for delivering data from the cache to the storage medium, for example, by the number of cache pages.

计算从淘汰数据量N；数据处理模块，用于在T_min≤N≤T_max的情况下，将大小为N的数据写入所述目标分条，把所述目标分条的数据淘汰给存储介质，其中当T_min是预设的第一阈值，T_max是预设的第二阈值。在另外一种可选方式中，数据处理模块用于在N＞T_max的情况下，将大小为N的数据写入所述目标分条。在另外一种可选方式中，在N＜T_min的情况下，数据处理模块用于把所述目标分条的数据淘汰给所述存储介质。这三种可选方式都是可选的，执行其中任意一个即可形成一个完整的流程。基于该缓存数据处理装置，在对缓存没有明显影响的情况下，分条容纳尽量多的数据后再淘汰到存储介质中。在尽早获得空闲的缓存空间以及提高磁盘利用率这两个互相矛盾的需求之间获得平衡。使得缓存数据更平滑的淘汰给存储介质，减少了IOPS严重波动的情况，使存储系统更加稳定。A second aspect of the present invention provides a cache data processing device, the device includes: an acquisition module, which can be used to acquire the compression ratio D, deduplication ratio C and average stripe filling rate I of data, and can also be used to acquire data in the cache The target strip size S, the effective data size W in the target strip; the calculation module, used for formulating

Calculate the amount of eliminated data N; the data processing module is used to write data of size N into the target _strip under the condition of T _{min≤N≤Tmax} , and eliminate the data of the target strip to storage medium, wherein when T _min is a preset first threshold value, and T _max is a preset second threshold value. In another optional manner, the data processing module is configured to write data of size N into the target strip under the condition of N>T _max . In another optional manner, in the case of N<T _min , the data processing module is configured to eliminate the data of the target stripe to the storage medium. These three optional methods are all optional, and a complete process can be formed by executing any one of them. Based on the cache data processing device, under the condition that there is no obvious impact on the cache, the strips accommodate as much data as possible before being eliminated into the storage medium. Strike a balance between the conflicting needs of getting free cache space early and improving disk utilization. The cached data is eliminated to the storage medium more smoothly, which reduces the severe fluctuation of IOPS and makes the storage system more stable.

In the first possible implementation manner of the second aspect of the present invention, the processing module is further configured to: after data of size N is written into the target stripe, if the target stripe is filled with subsequent data, write The target stripes out to the storage medium. This possible implementation proposes a solution for eliminating data in the case of N>T _m .

In the second possible implementation manner of the second aspect of the present invention, the D, C and I are statistical values, and their initial values are all 1.

In a third possible implementation manner of the second aspect of the present invention, S is described by the number of the smallest unit (or an integer multiple of the smallest unit) for delivering data from the cache to the storage medium, for example, by the number of cache pages.

获取淘汰数据量N；在T_min≤N≤T_max的情况下，将大小为N的数据写入所述目标分条，把所述目标分条的数据淘汰给存储介质，其中当T_min是预设的第一阈值，T_max是预设的第二阈值。在另外一种可选方式中，在N＞T_max的情况下，将大小为N的数据写入所述目标分条。在另外一种可选方式中，在N＜T_min的情况下，把所述目标分条的数据淘汰给所述存储介质。这三种可选方式都是可选的，执行其中任意一个即可形成一个完整的流程。基于该存储控制器，在对缓存没有明显影响的情况下，分条容纳尽量多的数据后再淘汰到存储介质中。在尽早获得空闲的缓存空间以及提高磁盘利用率这两个互相矛盾的需求之间获得平衡。A third aspect of the present invention provides a storage controller, the storage controller includes a processor and a cache, the cache is used to temporarily store data, and optionally, a storage medium (eg, a hard disk or a solid-state disk) for storing obsolete data. The processor executes the following steps by running a program: obtaining the compression ratio D, the deduplication ratio C, and the average stripe filling rate I of the data, and is also used to obtain the target stripe size S in the cache, and the target stripe size S in the cache. Effective data size W; according to the formula

Obtain the amount of eliminated data N; in the case of T _min ≤ N ≤ T _max , write the data of size N into the target stripe, and eliminate the data of the target stripe to the storage medium, where T _min is A preset first threshold, T _max is a preset second threshold. In another optional manner, in the case of N>T _max , data of size N is written into the target stripe. In another optional manner, in the case of N<T _min , the data of the target stripe is eliminated to the storage medium. These three optional methods are all optional, and a complete process can be formed by executing any one of them. Based on the storage controller, under the condition that there is no obvious impact on the cache, the stripe accommodates as much data as possible before being eliminated into the storage medium. Strike a balance between the conflicting needs of getting free cache space early and improving disk utilization.

In the first possible implementation manner of the third aspect of the present invention, the processor is further configured to execute: after data of size N is written into the target stripe, if the target stripe is filled with subsequent data, Retire the target stripe to the storage medium. This possible implementation proposes a solution for eliminating data in the case of N>T _m .

In the second possible implementation manner of the third aspect of the present invention, the D, C and I are statistical values, and their initial values are all 1.

In a third possible implementation manner of the third aspect of the present invention, S is described by the number of the smallest unit (or an integer multiple of the smallest unit) of delivering data from the cache to the storage medium, for example, by the number of cache pages.

In a fourth aspect of the present invention, a storage system is further provided, including the above-mentioned storage controller and a storage medium (eg, a hard disk, a solid-state hard disk), and the storage medium is used for storing obsolete data.

In a fifth aspect of the present invention, a storage system is further provided, including the above-mentioned data processing device and a storage medium (eg, a hard disk, a solid-state hard disk), and the storage medium is used to store discarded data.

Description of drawings

Figure 1 is a schematic diagram of the use of strips;

2 is a structural diagram of an embodiment of a storage system of the present invention;

3 is a flowchart of an embodiment of a method for processing cached data according to the present invention;

FIG. 4 is a topology diagram of an embodiment of a cache data processing apparatus.

Detailed ways

The solution of the present application can be applied to a storage device. The storage device is, for example, a storage controller. The storage controller and storage medium (such as hard disk, solid-state disk, etc.) are relatively independent, and together they constitute a storage system. Unless otherwise specified, the following embodiments take this situation as an example to describe the solutions of the embodiments of the present invention, and the storage medium takes a hard disk as an example.

It should be noted that the storage device may also be a storage device having both computing management capabilities and storage media, such as a server. The part of the server that has computing management capabilities is equivalent to the storage controller. The storage controller communicates with a storage medium inside the storage device, or communicates with a storage medium outside the storage device. Since the principle is similar, it will not be described in detail.

Referring to FIG. 2, it is a structural diagram of an embodiment of the storage system of the present invention. The host 11 communicates with the storage system 12 , and the storage system includes a storage controller 12 and a storage medium 13 . The storage controller 12 includes a processor 121 , a cache 122 and a memory 123 . The memory 123 may provide storage space for the processor 121 to run a program, and by running the program, the processor may execute the embodiment of the cache data processing method provided by the present invention. The memory 123 and the cache 122 can be integrated or separated. The storage controller 12 and the property medium 13 are independent in the figure. In other embodiments, the storage controller 12 may also integrate the storage medium 13, such as a general-purpose server.

The cache can be read and written by the host 11 . Since the cache space is limited, the data in the cache needs to be migrated to the storage medium 13 when the cache space becomes small. Specifically, the effective data of the stripes located in the cache 121 is migrated to the storage medium 13 in the unit of stripes. This migration is also called downloading the disk. The earlier the disk is downloaded, the earlier the cache can vacate the space. space.

In the block storage technology, the cache data is downloaded to the disk with the granularity of stripes. A stripe is a logical unit, and each stripe is composed of multiple stripes. Each/each group of hard disks stores one stripe of the stripe when data is unloaded from the cache. In the process of migrating the cache to the hard disk with the stripe as the granularity, it does not mean that the entire stripe is valid data. It is possible that a part of the stripe has no valid data. For example, only part of the stripe is valid data, and the remaining stripes are valid data. No valid data.

On the one hand, downloading the disk as soon as possible allows the cache to obtain free storage space as soon as possible; on the other hand, after the stripe is removed from the disk, the valid data and invalid data in the stripe will occupy the disk space. If the stripe has not yet accommodated as much valid data as possible Just downloading the disk will cause the utilization rate of the hard disk to be low. Therefore, from the perspective of improving the disk utilization rate, it is best not to download the disk as soon as possible, but to fill up the strips and then download the disk. By applying the algorithm according to the embodiment of the present invention, a better balance can be made between these two contradictory requirements.

Referring to FIG. 3 , it is a flowchart of an embodiment of a cache data processing method of the present invention, which can be executed by a storage controller.

21. Obtain the data deduplication ratio C, the data compression ratio D, and the average stripe filling ratio I from the memory.

The deduplication ratio C, the compression ratio D, and the average allocation fill rate I are all statistical values for historical data. When initialized, it is all 1; but with the operation of the system, these three values will change dynamically due to the continuous deduplication, compression, and filling of new stripes. After these data are counted, they can be stored in memory or other media.

For each stripe in the cache, before being removed from the disk, the valid data in the stripe can be deduplicated. For each block in the stripe, if the same block already exists in the hard disk, it means that this block is duplicated with the existing block on the hard disk, and this block does not need to be removed from the disk. This can reduce the amount of data to be downloaded and save the storage space of the disk. The deduplication ratio C is the ratio of the amount of data before and after deduplication in the cache over a period of time. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN. The minimum value of the deduplication ratio C is 1 (during the deduplication process, no data blocks are deleted due to duplication, or no deduplication is performed). The stripe data is deduplicated before the stripe is downloaded to save disk space.

For each stripe in the cache, before being removed from the disk, in addition to deduplication, the valid data in the stripe can also be compressed. Compression can be performed after deduplication operations. The compression ratio D is the ratio of the amount of data before and after the stripe compression in the cache in the past period of time. Theoretically, the minimum value of the compression ratio D is 1 (the compression operation is performed, but no data is successfully compressed; or the compression operation is not performed). The stripe data is compressed before the stripe is downloaded to the disk, which can save disk space. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN.

The average stripe filling rate I is the average value of the ratio of the valid data of each stripe in the cache to the total stripe length in the past period of time. Wherein, when calculating the average stripe filling rate I, the effective data to be counted may be the effective data after the compression operation and the deduplication operation are performed. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN.

It should be noted that the embodiments of the present invention describe two concepts of memory and cache. The memory in the embodiment of the present invention is, for example, a RAM, which can temporarily store operation data in the CPU and provide space for the computer to run programs. Memory can be part of memory, or it can be physically separate from memory. In this embodiment of the present invention, a cache (cache) may be used to temporarily store stripes; the cache may also be RAM, or may be other media such as solid-state drives (SSDs).

22. Obtain the target stripe size S in the memory and the effective data size W in the target stripe.

The target strip size S can be described by the length of the target strip, and the unit is, for example, bytes. It can also be described in terms of unit length, such as the number of cache pages occupied by the target stripe.

Similarly, the effective data size W in the target stripe can be described by the length of the target stripe, and the unit is Byte; it can also be described by the quantity of the unit length. The description method of the effective data size W in the target stripe is consistent with the description method of the target stripe size S.

According to the formula N=(S*D*C-W)/I, obtain the amount of eliminated data N, and perform one of the following operations according to the value range of N:

获取淘汰数据量N。23, according to the formula

Get the amount of obsolete data N.

The compression ratio C and the deduplication ratio D are optional. If there is no compression, there is no compression ratio C; if there is no deduplication, there is no deduplication ratio D.

When using this embodiment to process the data in the cache for the first time, since no stripe has been processed in the history, D=1, C=1, and I=1 by default.

Obviously, the unit of N is the same as that of S and W. When the unit of S is bytes, the unit of N is also bytes. The actual value of N may be an integer multiple of the minimum unit of the lower disk, and the difference from (S*D*C-W)/I is less than one minimum unit. For example, the minimum unit of the lower disk is 10 bytes, and the calculation result of (S*D*C-W)/I is 54 bytes, then an optional method is that the value of N is 50 bytes; another optional method is The value of N is 60 bytes.

While S is described by the number of cached pages, N is also the number of cached pages. At this time, if the calculation result of (S*D*C-W)/I is a decimal, the actual value of N can be the rounded value of (S*D*C-W)/I, an optional way is to down For example, if the calculation result of (S*D*C-W)/I is 10.3 cache pages, the value of N can be 10 cache pages. In another implementation manner, it may also be rounded up, and 10.3 cache pages are rounded up to 11 cache pages.

24. Perform a corresponding operation according to the value of the eliminated data amount N.

(1) In the case of T _min ≤ N ≤ T _max , write the data of size N into the target stripe, and then eliminate the data of the target stripe to the persistent storage medium (SSD, hard disk, etc.) . After data of size N is written into the target strip, current data of the target strip = data of size N + original data of the target strip.

Wherein T _min is a preset first threshold value, and T _maX is a preset second threshold value. _Tmin and _Tmax . The unit is the same as N. In different scenarios, T _min and T _max may have different requirements and may have different settings, so they are not limited. For example, the value of T _min may be 10% of the strip length*C*D, and the value of T _max may be 80% of the strip length*C*D.

(2) In the case of N>T _max , write data of size N into the target stripe. The difference from (1) is that the target stripe operation can be temporarily not performed. Later, new data is written to this stripe, so that the stripe is full, and then this stripe is eliminated to the persistent storage medium storage medium.

(3) In the case of N<T _min , the data of the target stripe is eliminated to the persistent storage medium. The difference from (1) is that the data of the size N is not written into the target stripe, and the target stripe is directly unloaded.

It should be noted that both (1) and (2) above involve writing the cached data to the target stripe. Logically, this write operation can be understood as attributing data that does not belong to the target stripe to the target stripe. After the write operation is performed, the written data is still physically located in the cache. But at the logical level, writing data of size N to the target stripe writes the data in the cache layer to the storage pool layer. The cache layer is used to provide read and write for the host and mainly interacts with the host; while the storage pool is used to temporarily store the data to be written to the hard disk and mainly interact with the hard disk, so these are two different functional layers.

For the convenience of understanding, the foregoing embodiment is described below with a specific example.

For example, obtain from memory: the current deduplication rate is 2, the current compression rate is 4, and the current statistical average fill rate of stripes is I=0.9. The size of the cache page is 4KB, and the stripe size is 8MB, that is, S=(8MB/4KB)=2K. The current stripe has written 15K cache pages, assuming Tmin=128, Tmax=4K.

According to the formula (S*D*C-W)/I=(2K*2*4-15K)/0.9≈1137.8. Round down to N=1137.

Consistent with the case (2) Tmin<N<Tmax. Therefore, according to the embodiment of the present invention, the data of 1137 cache pages is written into the current stripe, and then all the data in the current stripe is sent to the hard disk. After the sending is completed, the storage space occupied by the target stripe in the cache is released. In this example, the number of cached pages is used as S because it is a common practice to use cached pages as the granularity when sending data in the cache to the hard disk. In other embodiments, there may be other schemes, for example, the stripe size divided by the multiple of the cache page is used as S. In another embodiment, there may be no concept of cache pages, and the minimum granularity of the disk or an integer multiple of the minimum granularity may be used as the stripe size. Or directly use the stripe size as S, and after calculating the value of N, convert it into the number of cache pages.

Referring to FIG. 4 , the present invention further provides an embodiment of an apparatus for processing cached data. The cache data processing device 3 is a hardware or program virtual device, and can execute the above method. The cache data processing device 3 includes: a parameter acquisition module, a calculation module 32 connected to the parameter acquisition module, and a data processing module 33 connected to the calculation module 32 .

The parameter obtaining module is used to obtain the compression ratio D, the deduplication ratio C, and the average strip filling rate I of the data, and is also used to obtain the target strip size S in the cache and the effective data size W in the target strip .

The calculation module 32 is used to calculate according to the formula Get the amount of obsolete data N.

The data processing module 33 is configured to perform one of the following operations according to the value of N: in the case of T _min ≤ N ≤ T _max , write data with a size of N into the target stripe, and The data of the target stripe is eliminated to the storage medium, wherein when T _min is the preset first threshold value, T _max is the preset second threshold value; in the case of N>T _max , the data of size N is written into the target stripe; in the case of N<T _min , the data of the target stripe is eliminated to the storage medium.

The function of each module is introduced in detail below.

The parameter obtaining module is used to obtain the data deduplication ratio C, the data compression ratio D, and the average stripe filling ratio I from the memory.

The deduplication ratio C, the compression ratio D, and the average allocation fill rate I are all statistical values for historical data. When initialized, it is all 1; but with the operation of the system, these three values will change dynamically due to the continuous deduplication, compression, and filling of new stripes. After these data are counted, they can be stored in memory or other media.

For each stripe in the cache, before being removed from the disk, the valid data in the stripe can be deduplicated. For each block in the stripe, if the same block already exists in the hard disk, it means that this block is duplicated with the existing block on the hard disk, and this block does not need to be removed from the disk. This can reduce the amount of data to be downloaded and save the storage space of the disk. The deduplication ratio C is the ratio of the amount of data before and after deduplication in the cache over a period of time. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN. The minimum value of the deduplication ratio C is 1 (during the deduplication process, no data blocks are deleted due to duplication, or no deduplication is performed). The stripe data is deduplicated before the stripe is downloaded to save disk space.

For each stripe in the cache, before being removed from the disk, in addition to deduplication, the valid data in the stripe can also be compressed. Compression can be performed after deduplication operations. The compression ratio D is the ratio of the amount of data before and after the stripe compression in the cache in the past period of time. Theoretically, the minimum value of the compression ratio D is 1 (the compression operation is performed, but no data is successfully compressed; or the compression operation is not performed). The stripe data is compressed before the stripe is downloaded to the disk, which can save disk space. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN.

The average stripe filling rate I is the average value of the ratio of the valid data of each stripe in the cache to the total stripe length in the past period of time. Wherein, when calculating the average stripe filling rate I, the effective data to be counted may be the effective data after the compression operation and the deduplication operation are performed. The statistical stripe range can be all the stripes in the cache; it can also be a part of the stripes, for example, the data after downloading belongs to the stripes of a specific LUN.

It should be noted that the embodiments of the present invention describe two concepts of memory and cache. The memory in the embodiment of the present invention is, for example, a RAM, which can temporarily store operation data in the CPU and provide space for the computer to run programs. Memory can be part of memory, or it can be physically separate from memory. In this embodiment of the present invention, a cache (cache) may be used to temporarily store stripes; the cache may also be RAM, or may be other media such as solid-state drives (SSDs).

The parameter obtaining module is used to obtain the target strip size S in the memory and the effective data size W in the target strip.

The target strip size S can be described by the length of the target strip, and the unit is, for example, bytes. It can also be described in terms of unit length, such as the number of cache pages occupied by the target stripe.

Similarly, the effective data size W in the target stripe can be described by the length of the target stripe, and the unit is Byte; it can also be described by the quantity of the unit length. The description method of the effective data size W in the target stripe is consistent with the description method of the target stripe size S.

According to the formula N=(S*D*C-W)/I, obtain the amount of eliminated data N, and perform one of the following operations according to the value range of N:

获取淘汰数据量N。calculation module 32 for following the formula

Get the amount of obsolete data N.

The compression ratio C and the deduplication ratio D are optional. If there is no compression, there is no compression ratio C; if there is no deduplication, there is no deduplication ratio D.

When using this embodiment to process the data in the cache for the first time, since no stripe has been processed in the history, D=1, C=1, and I=1 by default.

Obviously, the unit of N is the same as that of S and W. When the unit of S is bytes, the unit of N is also bytes. The actual value of N can be an integer multiple of the minimum unit of the lower disk, and is the same as

The difference of (S*D*C-W)/I is less than one minimum unit. For example, the minimum unit of the lower disk is 10 bytes, and the calculation result of (S*D*C-W)/I is 54 bytes, then an optional method is that the value of N is 50 bytes; another optional method is The value of N is 60 bytes.

While S is described by the number of cached pages, N is also the number of cached pages. At this time, if the calculation result of (S*D*C-W)/I is a decimal, the actual value of N can be the rounded value of (S*D*C-W)/I, an optional way is to down For example, if the calculation result of (S*D*C-W)/I is 10.3 cache pages, the value of N can be 10 cache pages. In another implementation manner, it may also be rounded up, and 10.3 cache pages are rounded up to 11 cache pages.

34. Perform a corresponding operation according to the value of the eliminated data amount N.

(1) In the case of T _{min≤N≤T maX} , write the data of size N into the target stripe, and then eliminate the data of the target stripe to the persistent storage medium (SSD, hard disk, etc.) . After data of size N is written into the target strip, current data of the target strip = data of size N + original data of the target strip.

Wherein T _min is a preset first threshold value, and T _max is a preset second threshold value. _Tmin and _Tmax . The unit is the same as N. In different scenarios, T _min and T _max may have different requirements and may have different settings, so they are not limited. For example, the value of T _min may be 10% of the strip length*C*D, and the value of T _max may be 80% of the strip length*C*D.

(2) In the case of N>T _max , write data of size N into the target stripe. The difference from (1) is that the target stripe operation can be temporarily not performed. Later, new data is written to this stripe, so that the stripe is full, and then this stripe is eliminated to the persistent storage medium storage medium.

(3) In the case of N<T _min , the data of the target stripe is eliminated to the persistent storage medium. The difference from (1) is that the data of the size N is not written into the target stripe, and the target stripe is directly unloaded.

It should be noted that both (1) and (2) above involve writing the cached data to the stripe. This write operation logically attributes data that does not belong to the target stripe to the target stripe. After the write operation is performed, the written data is still physically located in the cache. But at the logical level, this is writing the data of the cache layer to the storage pool. The cache layer is used to provide read and write for the host and mainly interacts with the host; while the storage pool is used to temporarily store the data to be written to the hard disk and mainly interact with the hard disk, so these are two different functional layers. Steps 31, 32 and 33 are performed by the buffer pool layer.

For the convenience of understanding, the foregoing embodiment is described below with a specific example.

For example, obtain from memory: the current deduplication rate is 2, the current compression rate is 4, and the current statistical average fill rate of stripes is I=0.9. The size of the cache page is 4KB, and the stripe size is 8MB, that is, S=(8MB/4KB)=2K. The current stripe has written 15K cache pages, assuming Tmin=128, Tmax=4K.

According to the formula (S*D*C-W)/I=(2K*2*4-15K)/0.9≈1137.8. Round down to N=1137.

Consistent with the case (2) Tmin<N<Tmax. Therefore, according to the embodiment of the present invention, the data of 1137 cache pages is written into the current stripe, and then all the data in the current stripe is sent to the hard disk. After the sending is completed, the storage space occupied by the target stripe in the cache is released. In this example, the number of cached pages is used as S because it is a common practice to use cached pages as the granularity when sending data in the cache to the hard disk. In other embodiments, there may be other schemes, for example, the stripe size divided by the multiple of the cache page is used as S. In another embodiment, there may be no concept of cache pages, and the minimum granularity of the disk or an integer multiple of the minimum granularity may be used as the stripe size. Or directly use the stripe size as S, and after calculating the value of N, convert it into the number of cache pages.

The units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The above descriptions are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto, and changes or substitutions based on the present invention should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (11)

1. A method for processing cache data, the method comprising:

acquiring a compression ratio D, a deduplication ratio C and an average stripe filling rate I of data, and acquiring a target stripe size S in a cache and an effective data size W in the target stripe;

according to the formula

Obtaining eliminated data quantity N, and executing one of the following operations according to the value of N:

at T_min≤N≤T_maxWriting data of size N into the target stripe, and eliminating the data of the target stripe to the storage medium, wherein when T is_minIs a preset first threshold value, T_maxIs a preset second threshold;

at N > T_maxWriting data of size N into the target stripe;

at N < T_minIn case of (2), evicting the target stripe data to the storage medium.

2. The method of cache data processing according to claim 1, wherein after writing data of size N to the target stripe, further comprising:

and eliminating the target stripe to the storage medium after the target stripe is fully written with subsequent data.

3. The cache data processing method according to claim 1, wherein:

the initial values of D, C and I are both 1.

4. The cache data processing method of claim 1, further comprising:

where S is described by the number of cache pages.

5. A cache data processing apparatus, comprising:

the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a compression ratio D, a deduplication ratio C and an average stripe filling rate I of data, and is also used for acquiring a target stripe size S in a cache and an effective data size W in the target stripe;

a calculation module for following the formula

Obtaining eliminated data quantity N;

a data processing module, configured to perform one of the following operations according to the value of N:

at T_min≤N≤T_maxWriting data of size N into the target stripe, and eliminating the data of the target stripe to the storage medium, wherein when T is_minIs a preset first threshold value, T_maxIs a preset second threshold;

at N > T_maxWriting data of size N into the target stripe;

at N < T_minIn case of (2), evicting the target stripe data to the storage medium.

6. The cache data processing apparatus of claim 5, wherein the processing module is further to:

after data of size N is written into the target stripe, if the target stripe is full of subsequent data, the target stripe is evicted to the storage medium.

7. The cache data processing apparatus of claim 5, wherein:

the initial values of D, C and I are both 1.

8. The cache data processing apparatus of claim 5, wherein:

the S is described by the number of cache pages.

9. A memory controller, comprising a processor and a buffer for temporarily storing data, wherein the processor executes by running a program to:

the method comprises the steps of obtaining a compression ratio D, a deduplication ratio C and an average stripe filling rate I of data, and also obtaining a target stripe size S in a cache and an effective data size W in the target stripe;

according to the formulaObtaining eliminated data quantity N;

and executing one of the following operations according to the value of N: at T_min≤N≤T_maxWriting data of size N into the target stripe, and eliminating the data of the target stripe to the storage medium, wherein when T is_minIs a preset first threshold value, T_maxIs a preset second threshold; at N > T_maxWriting data of size N into the target stripe; at N < T_minIn case of (2), evicting the target stripe data to the storage medium.

10. The storage controller of claim 9, the processor further to:

after data of size N is written into the target stripe, if the target stripe is full of subsequent data, the target stripe is evicted to the storage medium.

11. A storage medium characterized in that,

the storage medium stores a computer program which, when executed by a processor, is capable of implementing the method of any one of claims 1 to 4.

Images