CN116048425B - A layered cache method, system and related components - Google Patents

Abstract

This application discloses a layered caching method, system and related components, which relate to the field of distributed storage. The layered caching method is applied to each computing node in the distributed storage system, including: using the client process to monitor the distribution of client directions The file IO operation request issued by the storage system, when the file IO operation request is detected, the file IO operation request is redirected to the server process; the server process is used to determine whether the target storage location corresponding to the file IO operation request is the aggregation cache layer; If not, read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregated cache layer; if so, read the data from the aggregated cache layer, and return the data to the client process, so that the client The end process returns data to the client. The application can improve the IO performance of massive small file data sets, and improve the performance bottleneck caused by metadata in high-concurrency metadata-intensive file system services.

Description

A layered cache method, system and related components

technical field

The present application relates to the field of distributed storage, in particular to a layered cache method, system and related components.

Background technique

；要素二，HPC高性能计算过程，包括数据预处理，数据标记，数据压缩等；要素三，分布式数据一致性同步更新。With the rapid growth of HPC (High Performance Computing, high-performance computer group) computing power, large-scale, high-concurrency applications have brought greater pressure on the distributed storage system IO (Input Output, input and output). Three key elements involved in high-performance HPC scenarios: Element 1, the data and label metadata attributes of each high-performance computing node need to perform large-scale, high-concurrency storage IO, of which 80% of the IO includes loading and modifying data sets , for HPC training and random data retrieval, a typical intensive IO model is

; Element two, HPC high-performance computing process, including data preprocessing, data labeling, data compression, etc.; element three, distributed data consistency and synchronous update.

By analyzing the three key elements of the HPC scenario, it is found that the large number of small files generated by high-performance computing and high concurrent IO and random access will easily cause the IO read and write performance of the distributed storage system to be saturated. For example, common HPC datasets generally contain 3000 different files. For small files of more than 2 million types, if the storage IO read-write software stack cannot meet the HPC requirements for large-scale operations, it will block high-performance computing services. Therefore, the IO performance of distributed storage systems is crucial to high-performance computing services . Existing technical solutions have proposed some optimization solutions for high-performance computing to improve storage IO performance, such as prefetching and caching. However, there are still large-scale and high-concurrency storage IO using existing solutions in HPC high-performance computing scenarios. Many technical challenges, such as reading-intensive high-performance IO for small files, will generate huge metadata service overhead for distributed storage systems, which in turn will affect data storage efficiency.

Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present.

Contents of the invention

The purpose of this application is to provide a layered cache method, system and related components, which can improve the IO performance of massive small file data sets, and improve the performance bottleneck caused by metadata in high-concurrency metadata-intensive file system services.

In order to solve the above technical problems, the present application provides a hierarchical caching method, which is applied to each computing node of the distributed storage system, and the hierarchical caching method includes:

Using the client process to monitor the file IO operation request sent by the client to the distributed storage system, when the file IO operation request is detected, redirecting the file IO operation request to the server process;

Using the server process to determine whether the target storage location corresponding to the file IO operation request is an aggregation cache layer;

If not, read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer;

If so, read the data from the aggregation cache layer, and return the data to the client process, so that the client process returns the data to the client.

Optionally, the process of using the server process to determine whether the target storage location corresponding to the file IO operation request is an aggregation cache layer includes:

Utilize the server process to insert the received file IO operation request into a shared queue;

In the shared queue, determine whether the data corresponding to the file IO operation request is cached data;

If yes, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer.

Optionally, in the shared queue, the process of determining whether the data corresponding to the file IO operation request is cached data includes:

In the shared queue, a data thread is used to determine whether the data corresponding to the file IO operation request is cached data; the data thread is a thread generated when the server process instance is constructed.

Optionally, the layered caching method further includes:

When the file IO operation request sent by the client to the distributed storage system is detected, the server process is started, and the state of the computing node and other computing nodes adjacent to the computing node are used to dynamically construct the Instance of the server process described above.

Optionally, the process of reading the data from the aggregation cache layer includes:

The file IO operation request is redirected to the aggregation cache layer through a data thread, so as to read the data from the aggregation cache layer.

Optionally, when using the server process to insert the received file IO operation request into the shared queue, the layered caching method further includes:

Configure the shared queue with a mutex.

Optionally, the shared queue is a FIFO queue.

Optionally, the data corresponding to the file IO operation request includes a file descriptor, read offset and length.

Optionally, the layered caching method further includes:

The storage location of the data in the aggregation cache layer is determined based on the file path and the computing node to which it belongs.

Optionally, the layered caching method further includes:

Broadcast the file IO operation request to computing nodes adjacent to the current computing node.

Optionally, the layered caching method further includes:

Judging whether the data set corresponding to the file IO operation request is greater than the total capacity of the local storage medium;

If so, perform cache eviction and replacement operations.

Optionally, the layered caching method further includes:

A dynamic link library is built based on environment variables; the dynamic link library is used to intercept the file IO operation request.

Optionally, the process of redirecting the file IO operation request to the server process includes:

The file IO operation request is redirected to the server process through a hash algorithm.

Optionally, the aggregation cache layer is a cache layer composed of high-speed storage media in each of the computing nodes in the distributed storage system.

Optionally, the high-speed storage medium is Nvme SSD.

Optionally, the layered caching method further includes:

When the clearing condition is met, clear the data stored in the high-speed storage medium on the computing node.

In order to solve the above technical problems, the present application also provides a layered cache system, which is applied to each computing node of the distributed storage system, and the layered cache system includes:

A monitoring module, configured to use a client process to monitor the file IO operation request sent by the client to the distributed storage system, and redirect the file IO operation request to the server process when the file IO operation request is detected;

A processing module, configured to use the server process to judge whether the target storage location corresponding to the file IO operation request is an aggregation cache layer, if not, trigger the first reading module, and if so, trigger the second reading module;

The first reading module is configured to read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer;

A second reading module, configured to read the data from the aggregation cache layer, and return the data to the client process, so that the client process returns the data to the client .

In order to solve the above technical problems, the present application also provides an electronic device, including:

memory for storing computer programs;

A processor, configured to implement the steps of the hierarchical cache method described in any one of the above when executing the computer program.

In order to solve the above technical problems, the present application also provides a distributed storage system, which includes a storage base module and a plurality of nodes, each of which includes a layered client process, a layered server process, and a storage medium, each The storage medium of the node constitutes an aggregation cache layer, wherein:

The layered client process is used to monitor the file IO operation request sent by the client, and when the file IO operation request is detected, redirect the file IO operation request to the layered server process;

The layered server process is used to judge whether the target storage location corresponding to the file IO operation request is the aggregation cache layer, and if not, read the data corresponding to the file IO operation request from the storage bottom layer module , and send the data to the aggregation cache layer, if so, read the data from the aggregation cache layer, and return the data to the layered client process so that the layered client The end process returns the data to the client;

The aggregation cache layer is used to store the data sent by the layered server process.

In order to solve the above-mentioned technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it realizes any of the above-mentioned Steps of the tiered caching approach.

This application provides a layered caching method. After the client process monitors the file IO operation request, it redirects the file IO operation request to the server process. When there is a miss, the data is retrieved from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the highly concurrent metadata-intensive file system business. The present application also provides a layered caching system, electronic equipment, a distributed storage system and a computer-readable storage medium, which have the same beneficial effect as the above-mentioned layered caching method.

Description of drawings

In order to illustrate the embodiments of the present application more clearly, the following will briefly introduce the accompanying drawings used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Fig. 1 is a flow chart of the steps of a layered caching method provided by the present application;

FIG. 2 is a schematic structural diagram of a layered cache system provided by the present application;

FIG. 3 is a schematic diagram of a hierarchical cache framework of a distributed storage system provided by the present application;

FIG. 4 is a schematic structural diagram of a hierarchical caching system provided by the present application.

Detailed ways

The core of this application is to provide a layered cache method, system and related components, which can improve the IO performance of massive small file data sets, and improve the performance bottleneck caused by metadata in high-concurrency metadata-intensive file system services.

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

In the first aspect, please refer to FIG. 1. FIG. 1 is a flow chart of the steps of a layered caching method provided by the present application. The layered caching method includes:

S101: Use the client process to monitor the file IO operation request sent by the client to the distributed storage system, and when the file IO operation request is detected, redirect the file IO operation request to the server process;

构建缓存，通过聚合分布式集群节点本地和相邻节点本地存储来加速读取存储数据性能，以提高海量小文件数据集的IO性能。This embodiment proposes an aggregated cache layer, which is a transparent read-only cache layer for typical intensive IO models

Build a cache to accelerate the performance of reading and storing data by aggregating the local storage of distributed cluster nodes and adjacent nodes, so as to improve the IO performance of massive small file data sets.

特点。参照图2所示，本申请所提供的分层缓存系统的体系结构，由两个主要组件组成：分层缓存客户端进程和分层缓存服务端进程，这两个组件部署在分布式存储系统的客户端层，在HPC上的一组计算节点上分配作业时，启动分层缓存服务端进程，利用分布式存储节点和相邻节点本地存储动态构建服务端进程实例。分布式存储系统的每个节点均部署分层缓存客户端进程和服务端进程，该进程会把HPC高性能计算作业请求的数据缓存到本节点的Nvme SSD高速存储介质设备上。The client accesses the distributed storage system through the POSIX file system interface provided by the distributed storage system, so as to accelerate the storage IO access performance of HPC high-performance scenarios. This scenario has read-only data with a high re-read rate and has a typical intensive IO model.

features. Referring to Figure 2, the architecture of the layered cache system provided by this application consists of two main components: a layered cache client process and a layered cache server process, these two components are deployed in the distributed storage system In the client layer, when jobs are assigned to a group of computing nodes on HPC, a layered cache server process is started, and a server process instance is dynamically constructed using distributed storage nodes and local storage of adjacent nodes. Each node of the distributed storage system deploys a hierarchical cache client process and server process, which will cache the data requested by the HPC high-performance computing job to the Nvme SSD high-speed storage media device of the node.

Specifically, the layered caching process of the present application is described, as shown in FIG. 3 , first, the client process is preloaded, and the client process is used to monitor and intercept file system operations, such as open, read, and close. File system calls are made through the Process interception does not need to modify the underlying file system of existing high-performance computing applications or distributed storage systems. It can be understood that the layered cache client process is composed of a file system IO interface forwarding module, which captures the file system calls to the distributed storage system and redirects them to the corresponding layered cache server process. Therefore, the layered cache The cache client process first reads hit data through the aggregation cache layer, which is helpful for high-performance business performance requirements.

S102: Use the server process to determine whether the target storage location corresponding to the file IO operation request is the aggregation cache layer, if not, execute S103, and if so, execute S104;

S103: Read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer;

S104: Read data from the aggregation cache layer, and return the data to the client process, so that the client process returns the data to the client.

After the server process receives the file IO operation request intercepted by the client process, it will only retrieve the file from the bottom layer of the distributed storage system when the cache misses. It is understandable that the following two reading scenarios are explained separately. , the two reading scenarios include first-time reading and non-first-time reading.

For first read:

The computing node client in the HPC high-performance scenario initiates a read request to the dataset directory on the distributed storage system, and the hierarchical cache client process intercepts any incoming file IO operation requests and starts tracking in the dataset directory. The RPC (remote procedure call) handler of the layered cache client process redirects the requested file IO operation request to the corresponding layered cache server process, and the internal management RPC handler of the layered cache client process and server process , which is responsible for sending and receiving messages over the network.

As an optional embodiment, the process of using the server process to determine whether the target storage location corresponding to the file IO operation request is an aggregation cache layer includes: using the server process to insert the received file IO operation request into the shared queue; In the shared queue, it is determined whether the data corresponding to the file IO operation request is cached data; if so, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer.

As an optional embodiment, in the shared queue, the process of determining whether the data corresponding to the file IO operation request is cached data includes: in the shared queue, determining whether the data corresponding to the file IO operation request is cached data through a data thread Cache data; the data thread is the thread generated when the server process instance is constructed.

Specifically, when the hierarchical cache server process receives a file IO operation request, the RPC handler will insert the forwarded file IO into the shared FIFO (First Input First Output, first-in-first-out queue) queue. In the shared FIFO queue, pass The move-data data thread checks whether the file has been cached. Since it is the first time to read the file, it needs to pull the data into the aggregation cache to determine the cached file descriptor, read offset, and length.

For non-first reads:

As an optional embodiment, the process of reading data from the aggregation cache layer includes:

Redirect the file IO operation request to the aggregation cache layer through the data thread, so as to read data from the aggregation cache layer.

Specifically, the data thread redirects the IO to the aggregation cache to read the file, returns the file descriptor to the corresponding hierarchical cache Client process, and then the process returns the file descriptor, read offset and length and other data For HPC applications, that is, the HPC high-performance client request side. The file IO operation request is visible. In this embodiment, after the client process detects the file IO operation request, it redirects the file IO operation request to the server process. The server process first retrieves the file in the aggregation cache layer, and in the aggregation cache layer When there is a miss, the data is retrieved from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the highly concurrent metadata-intensive file system business.

It can be seen that in this embodiment, after the client process detects the file IO operation request, the file IO operation request is redirected to the server process, and the server process first retrieves the file in the aggregation cache layer, and when the aggregation cache layer misses, Then retrieve data from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the high-concurrency metadata-intensive file system business.

On the basis of above-mentioned embodiment:

As an optional embodiment, when using the server process to insert the received file IO operation request into the shared queue, the layered caching method also includes:

Configure the shared queue with a mutex.

Specifically, considering that multiple hierarchical cache client processes will request a single file at the same time, a mutex is used on the shared queue to ensure consistency and avoid duplicative copying of files into the aggregated cache.

As an optional embodiment, the hierarchical caching method further includes:

Determine the storage location of the data in the aggregation cache layer based on the file path and the computing node to which it belongs.

As an optional embodiment, the hierarchical caching method further includes:

Broadcast the file IO operation request to the computing nodes adjacent to this computing node.

Specifically, the layered cache process broadcasts the file search request to adjacent nodes, which helps to balance the load pressure among the nodes.

As an optional embodiment, the hierarchical caching method further includes:

Determine whether the data set corresponding to the file IO operation request is greater than the total capacity of the local storage medium;

If so, perform cache eviction and replacement operations.

Specifically, the aggregation cache elimination mechanism is related to the repeated read data set job. If the data set is larger than the total capacity of the node's local cache, cache eviction and replacement will be performed.

As an optional embodiment, the hierarchical caching method further includes:

Build a dynamic link library based on environment variables; the dynamic link library is used to intercept file IO operation requests.

Specifically, this embodiment formulates the redirection only read aggregation cache, that is, intercepts the read IO mechanism, based on the initial prototype of the distributed storage system high-performance scene operation, the aggregation cache layer of this embodiment is helpful for analyzing high-performance business scenarios, especially The IO read calls in the three key elements to understand how the data loader in the framework accesses the file. In this embodiment, the aggregation cache layer is designed to intercept IO-related function calls, and is constructed by selectively loading the same function mechanism in different dynamic link libraries. This mechanism avoids forcing applications to modify their code bases to support the aggregation cache layer the necessity of.

Specifically, the intercepted read IO mechanism of this embodiment is redirected to the dynamic link library Only_Read_Performance.so. The technical optimization point of the dynamic link library is that once the function of the dynamic library changes, it is transparent to the executable program and can The executable program does not need to be recompiled. For statically linked programs in other technologies, a small change in the function library requires recompilation and release of the entire program. Among them, the static link is to compile all the referenced functions or variables into the executable file. Dynamic linking does not compile functions into executable files, but dynamically loads the function library when the program is running, that is, running the link. Therefore, the redirection to the dynamic link library Only_Read_Performance.so has compatibility and portability, and is of great value to the distributed storage system.

The specific embodiment steps of intercepting the read IO mechanism are as follows:

调用来访问底层分布式存储文件系统；(1) For HPC high-performance computing job client requests, and file system requests that meet standard POSIX semantics, in line with typical intensive IO models

Call to access the underlying distributed storage file system;

(2) Use the environment variable LD_PRELOAD of the Linux server of the distributed storage system, which is characterized by dynamic library loading and has the highest priority, which is the embodiment method of this application to intercept the read request processing logic;

(3) Intercept the Input of the read IO mechanism:

文件系统调用；a.

file system calls;

b. LD_PRELOAD environment variable;

c. The dynamic link library of the local aggregation cache layer is recorded as Only_Read_Performance.so;

(4) Intercept the Output of the read IO mechanism:

Execute Only_Read_Performance.so, and perform read cache logic processing at the cache aggregation layer as described above.

As an optional embodiment, the process of redirecting the file IO operation request to the server process includes:

Redirect the file IO operation request to the server process through the hash algorithm.

In this embodiment, the hash algorithm is used to perform redirection, which can avoid metadata search bottlenecks, and aims to improve random read performance. The hierarchical cache client process uses Hash to redirect IO and looks up the cache on the hierarchical cache server process, so as not to store the cached file metadata in the distributed metadata storage or in-memory database. In an aggregated cache, the file cache location is determined using the file path and owning node.

As an optional embodiment, the aggregation cache layer is a cache layer composed of high-speed storage media in each computing node in the distributed storage system.

As an optional embodiment, the high-speed storage medium is Nvme (Non-Volatile Memory express, non-volatile memory host controller interface specification) SSD (Solid State Disk, solid state disk).

As an optional embodiment, the hierarchical caching method further includes:

When the clearing condition is met, clear the data stored in the high-speed storage medium on the computing node.

In this embodiment, the life cycle of the data set in the cache is coupled with the life cycle of the job on the HPC high-performance client. After the job is completed, the cached data set will be cleared from the local storage of the node.

构建缓存，基于RPC远程过程调用机制，采用聚合分布式集群节点本地和相邻节点本地存储来加速读取存储数据性能，以提高海量小文件数据集的IO性能；采用分布式hash算法IO重定向来确定数据请求的缓存位置，基于LD_PRELOAD环境变量设计最高优先级动态链接库Only_Read_Performance.so用以截获Read IO请求，避免对分布式存储系统元数据查找造成瓶颈，目标是提高随机读取性能；同时提高分布式存储系统缓存命中率，有效避免元数据查找和文件锁竞争对存储系统IO带来的性能瓶颈，与此同时，由于本申请的聚合缓存层与分布式存储系统相互独立，因此也具有可移植性和通用性的特点。To sum up, the aggregation cache layer proposed in this application is independent of the metadata module and data module storage mechanism of the distributed storage system. It is a transparent read-only cache layer and is aimed at typical intensive IO models.

Construct a cache, based on the RPC remote procedure call mechanism, use the aggregated distributed cluster node local and adjacent node local storage to speed up the performance of reading and storing data, so as to improve the IO performance of massive small file data sets; use distributed hash algorithm IO redirection To determine the cache location of the data request, the highest priority dynamic link library Only_Read_Performance.so is designed based on the LD_PRELOAD environment variable to intercept the Read IO request, avoiding the bottleneck caused by the metadata search of the distributed storage system, and the goal is to improve the random read performance; at the same time Improve the cache hit rate of the distributed storage system, effectively avoid the performance bottleneck caused by metadata search and file lock competition on the storage system IO. At the same time, because the aggregation cache layer of this application is independent of the distributed storage system, it also has Features of portability and versatility.

In terms of performance, this application effectively improves the performance of the high-concurrency IO model in HPC high-performance scenarios, and improves the performance bottleneck caused by metadata in high-concurrency metadata-intensive file system services; in terms of stability, this application designs the cache aggregation layer independently In the underlying system of distributed storage, if there is a failure of the cache aggregation layer storage medium, it will not affect the normal call of the business and has stability; in terms of security, the hierarchical cache architecture is loosely coupled with the distributed storage system, and there is no security risk; in terms of cost On the one hand, this application improves the universal storage IO performance problem of HPC high-performance scenarios, which can improve the competitiveness and maintenance cost of distributed file storage; on the other hand, this application has portability and versatility, and does not need to modify HPC operations Applications, improve the linear scalability of distributed clusters, and are compatible with common file quotas, snapshots and other features.

For the second aspect, please refer to FIG. 4. FIG. 4 is a schematic structural diagram of a layered cache system provided by the present application, which is applied to each computing node of the distributed storage system. The layered cache system includes:

The monitoring module 41 is used to monitor the file IO operation request sent by the client to the distributed storage system by using the client process, and when the file IO operation request is detected, the file IO operation request is redirected to the server process;

The processing module 42 is used to utilize the server process to judge whether the target storage location corresponding to the file IO operation request is an aggregation cache layer, if not, trigger the first reading module 43, and if so, trigger the second reading module 44;

The first reading module 43 is used to read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer;

The second reading module 44 is configured to read data from the aggregation cache layer and return the data to the client process, so that the client process returns the data to the client.

构建缓存，通过聚合分布式集群节点本地和相邻节点本地存储来加速读取存储数据性能，以提高海量小文件数据集的IO性能。This embodiment proposes an aggregated cache layer, which is a transparent read-only cache layer for typical intensive IO models

Build a cache to accelerate the performance of reading and storing data by aggregating the local storage of distributed cluster nodes and adjacent nodes, so as to improve the IO performance of massive small file data sets.

特点。参照图2所示，本申请所提供的分层缓存系统的体系结构，由两个主要组件组成：分层缓存客户端进程和分层缓存服务端进程，这两个组件部署在分布式存储系统的客户端层，在HPC上的一组计算节点上分配作业时，启动分层缓存服务端进程，利用分布式存储节点和相邻节点本地存储动态构建服务端进程实例。分布式存储系统的每个节点均部署分层缓存客户端进程和服务端进程，该进程会把HPC高性能计算作业请求的数据缓存到本节点的Nvme SSD高速存储介质设备上。The client accesses the distributed storage system through the POSIX file system interface provided by the distributed storage system, so as to accelerate the storage IO access performance of HPC high-performance scenarios. This scenario has read-only data with a high re-read rate and has a typical intensive IO model.

features. Referring to Figure 2, the architecture of the layered cache system provided by this application consists of two main components: a layered cache client process and a layered cache server process, these two components are deployed in the distributed storage system In the client layer, when jobs are assigned to a group of computing nodes on HPC, a layered cache server process is started, and a server process instance is dynamically constructed using distributed storage nodes and local storage of adjacent nodes. Each node of the distributed storage system deploys a hierarchical cache client process and server process, which will cache the data requested by the HPC high-performance computing job to the Nvme SSD high-speed storage media device of the node.

Specifically, the layered caching process of the present application is described, as shown in FIG. 3 , first, the client process is preloaded, and the client process is used to monitor and intercept file system operations, such as open, read, and close. File system calls are made through the Process interception does not need to modify the underlying file system of existing high-performance computing applications or distributed storage systems. It can be understood that the layered cache client process is composed of a file system IO interface forwarding module, which captures the file system calls to the distributed storage system and redirects them to the corresponding layered cache server process. Therefore, the layered cache The cache client process first reads hit data through the aggregation cache layer, which is helpful for high-performance business performance requirements.

After the server process receives the file IO operation request intercepted by the client process, it will only retrieve the file from the bottom layer of the distributed storage system when the cache misses. It is understandable that the following two reading scenarios are explained separately. , the two reading scenarios include first-time reading and non-first-time reading.

For first read:

The computing node client in the HPC high-performance scenario initiates a read request to the dataset directory on the distributed storage system, and the hierarchical cache client process intercepts any incoming file IO operation requests and starts tracking in the dataset directory. The RPC (remote procedure call) handler of the layered cache client process redirects the requested file IO operation request to the corresponding layered cache server process, and the internal management RPC handler of the layered cache client process and server process , which is responsible for sending and receiving messages over the network.

As an optional embodiment, the process of using the server process to determine whether the target storage location corresponding to the file IO operation request is an aggregation cache layer includes: using the server process to insert the received file IO operation request into the shared queue; In the shared queue, it is determined whether the data corresponding to the file IO operation request is cached data; if so, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer. As an optional embodiment, in the shared queue, the process of determining whether the data corresponding to the file IO operation request is cached data includes: in the shared queue, determining whether the data corresponding to the file IO operation request is cached data through a data thread Cache data; the data thread is the thread generated when the server process instance is constructed.

Specifically, when the hierarchical cache server process receives a file IO operation request, the RPC handler will insert the forwarded file IO into the shared FIFO queue, and in the shared FIFO queue, check whether the file has been cached through the move-data data thread. Since it is the first time to read a file, the data needs to be pulled into the aggregation cache to determine the cached file descriptor, read offset, and length.

For non-first reads:

As an optional embodiment, the process of reading data from the aggregation cache layer includes:

Redirect the file IO operation request to the aggregation cache layer through the data thread, so as to read data from the aggregation cache layer.

Specifically, the data thread redirects the IO to the aggregation cache to read the file, returns the file descriptor to the corresponding hierarchical cache Client process, and then the process returns the file descriptor, read offset and length and other data For HPC applications, that is, the HPC high-performance client request side. The file IO operation request is visible. In this embodiment, after the client process detects the file IO operation request, it redirects the file IO operation request to the server process. The server process first retrieves the file in the aggregation cache layer, and in the aggregation cache layer When there is a miss, the data is retrieved from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the highly concurrent metadata-intensive file system business.

It can be seen that in this embodiment, after the client process detects the file IO operation request, the file IO operation request is redirected to the server process, and the server process first retrieves the file in the aggregation cache layer, and when the aggregation cache layer misses, Then retrieve data from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the high-concurrency metadata-intensive file system business.

As an optional embodiment, the process of using the server process to determine whether the target storage location corresponding to the file IO operation request is the aggregation cache layer includes:

Use the server process to insert the received file IO operation request into the shared queue;

In the shared queue, determine whether the data corresponding to the file IO operation request is cached data;

If yes, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer.

As an optional embodiment, in the shared queue, the process of determining whether the data corresponding to the file IO operation request is cached data includes:

In the shared queue, the data thread is used to determine whether the data corresponding to the file IO operation request is cached data; the data thread is the thread generated when the server process instance is constructed.

As an optional embodiment, the hierarchical caching system further includes:

The preprocessing module is used to monitor the file IO operation request sent by the client to the distributed storage system, start the server process, and dynamically build the server by using the status of the computing node and the status of other computing nodes adjacent to the computing node process instance.

As an optional embodiment, the process of reading data from the aggregation cache layer includes:

Redirect the file IO operation request to the aggregation cache layer through the data thread, so as to read data from the aggregation cache layer.

As an optional embodiment, while using the server process to insert the received file IO operation request into the shared queue, the hierarchical caching system also includes:

The configuration module is used to configure a mutual exclusion lock for a shared queue.

As an optional embodiment, the shared queue is a FIFO queue.

As an optional embodiment, the data corresponding to the file IO operation request includes a file descriptor, a read offset, and a length.

As an optional embodiment, the hierarchical caching system further includes:

The determination module is configured to determine the storage location of the data in the aggregation cache layer based on the file path and the computing node to which it belongs.

As an optional embodiment, the hierarchical caching system further includes:

The broadcast module is configured to broadcast the file IO operation request to computing nodes adjacent to the computing node.

As an optional embodiment, the hierarchical caching system further includes:

A judging module, configured to judge whether the data set corresponding to the file IO operation request is larger than the total capacity of the local storage medium, and if so, perform a cache eviction operation and a replacement operation.

As an optional embodiment, the hierarchical caching system further includes:

The building module is used to build a dynamic link library based on environment variables; the dynamic link library is used to intercept file IO operation requests.

As an optional embodiment, the process of redirecting the file IO operation request to the server process includes:

Redirect the file IO operation request to the server process through the hash algorithm.

As an optional embodiment, the aggregation cache layer is a cache layer composed of high-speed storage media in each computing node in the distributed storage system.

As an optional embodiment, the high-speed storage medium is Nvme SSD.

As an optional embodiment, the hierarchical caching system further includes:

The clearing module is used to clear the data stored in the medium and high-speed storage medium on the computing node when the clearing condition is met.

In a third aspect, the present application also provides an electronic device, including:

memory for storing computer programs;

A processor, configured to implement the steps of any one of the above hierarchical cache methods when executing a computer program.

Specifically, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer-readable instructions, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage medium. The processor provides computing and control capabilities for the electronic device. When executing the computer program stored in the memory, the following steps can be implemented: use the client process to monitor the file IO operation request sent by the client to the distributed storage system, and when the file IO operation is detected Request, redirect the file IO operation request to the server process; use the server process to determine whether the target storage location corresponding to the file IO operation request is the aggregated cache layer; if not, read the corresponding file IO operation request from the bottom layer of the distributed storage system , and cache the data to the aggregated cache layer; if so, read the data from the aggregated cache layer and return the data to the client process, so that the client process can return the data to the client.

It can be seen that in this embodiment, after the client process detects the file IO operation request, the file IO operation request is redirected to the server process, and the server process first retrieves the file in the aggregation cache layer, and when the aggregation cache layer misses, Then retrieve data from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the high-concurrency metadata-intensive file system business.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: utilize the server process to insert the received file IO operation request into the shared queue; in the shared queue, determine Whether the data corresponding to the file IO operation request is cached data; if so, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: in the shared queue, determine whether the data corresponding to the file IO operation request is cached data through the data thread; It is a thread generated when building a server process instance.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: when monitoring the file IO operation request sent by the client to the distributed storage system, start the server process, and use this calculation The status of the node and the status of other computing nodes adjacent to the computing node dynamically build a server process instance.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: redirecting the file IO operation request to the aggregation cache layer through the data thread, so as to read data from the aggregation cache layer.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: configuring the shared queue with a mutual exclusion lock.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: determining the storage location of the data in the aggregation cache layer based on the file path and the computing node to which it belongs.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: broadcasting the file IO operation request to computing nodes adjacent to the computing node.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps can be implemented: determine whether the data set corresponding to the file IO operation request is greater than the total capacity of the local storage medium; if so, perform cache eviction operation and replace operation.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: building a dynamic link library based on environment variables; the dynamic link library is used to intercept file IO operation requests.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: redirecting the file IO operation request to the server process through a hash algorithm.

As an optional embodiment, when the processor executes the computer subroutine stored in the memory, the following steps may be implemented: when the clearing condition is met, clear the data stored in the high-speed storage medium on the computing node.

On the basis of the foregoing embodiments, as a preferred implementation manner, the electronic device further includes:

The input interface is connected with the processor, and is used to obtain the computer program, parameters and instructions imported from the outside, and store them in the memory under the control of the processor. The input interface can be connected with an input device to receive parameters or instructions manually input by the user. The input device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the terminal shell.

The display unit is connected with the processor and used for displaying the data sent by the processor. The display unit may be a liquid crystal display or an electronic ink display.

The network port is connected with the processor and is used for communication connection with various external terminal devices. The communication technology used in the communication connection can be wired communication technology or wireless communication technology, such as mobile high-definition link technology (MHL), universal serial bus (USB), high-definition multimedia interface (HDMI), wireless fidelity technology (WiFi), Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.

In the fourth aspect, the present application also provides a distributed storage system, including a storage bottom module and a plurality of nodes, each node includes a layered client process, a layered server process and a storage medium, and the storage medium of each node Constitutes the aggregation cache layer, where:

The layered client process is used to monitor the file IO operation request sent by the client, and when the file IO operation request is detected, the file IO operation request is redirected to the layered server process;

The layered server process is used to judge whether the target storage location corresponding to the file IO operation request is the aggregation cache layer, if not, read the data corresponding to the file IO operation request from the storage underlying module, and send the data to the aggregation cache layer, If so, read data from the aggregation cache layer and return the data to the layered client process so that the layered client process returns the data to the client;

The aggregation cache layer is used to store the data sent by the layered server process.

In the fifth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above-mentioned layered caching methods are implemented.

Specifically, the computer-readable storage medium may include: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc., which can store programs. The medium of the code. A computer program is stored on the storage medium, and the following steps are implemented when the computer program is executed by the processor: use the client process to monitor the file IO operation request sent by the client to the distributed storage system, and when the file IO operation request is detected, the file IO The operation request is redirected to the server process; the server process is used to determine whether the target storage location corresponding to the file IO operation request is the aggregation cache layer; if not, the data corresponding to the file IO operation request is read from the bottom layer of the distributed storage system, and the The data is cached to the aggregation cache layer; if so, the data is read from the aggregation cache layer, and the data is returned to the client process, so that the client process returns the data to the client.

It can be seen that in this embodiment, after the client process detects the file IO operation request, the file IO operation request is redirected to the server process, and the server process first retrieves the file in the aggregation cache layer, and when the aggregation cache layer misses, Then retrieve data from the bottom layer of the distributed storage system to improve the IO performance of massive small file datasets and improve the performance bottleneck caused by metadata in the high-concurrency metadata-intensive file system business.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: using the server process to insert the received file IO operation request into the shared queue; In the shared queue, it is determined whether the data corresponding to the file IO operation request is cached data; if so, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps can be specifically implemented: in the shared queue, determine whether the data corresponding to the file IO operation request is The data has been cached; the data thread is the thread generated when the server process instance is constructed.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps can be specifically implemented: when the monitoring client sends a file IO operation request to the distributed storage system, start the service The end process uses the status of the computing node and the status of other computing nodes adjacent to the computing node to dynamically build a server process instance.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps can be specifically implemented: redirect the file IO operation request to the aggregation cache layer through the data thread, so that the The cache layer reads data.

As an optional embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: configuring the shared queue with a mutual exclusion lock.

As an optional embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: determining the storage location of the data in the aggregation cache layer based on the file path and the computing node to which it belongs.

As an optional embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: broadcasting the file IO operation request to computing nodes adjacent to the computing node.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: judging whether the data set corresponding to the file IO operation request is greater than the total capacity of the local storage medium; If so, perform cache eviction and replacement operations.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps can be specifically implemented: constructing a dynamic link library based on environment variables; the dynamic link library is used to intercept file IO operation requests .

As an optional embodiment, when the computer subprogram stored in the computer-readable storage medium is executed by the processor, the following steps may be specifically implemented: redirecting the file IO operation request to the server process through a hash algorithm.

As an optional embodiment, when the computer subroutine stored in the computer-readable storage medium is executed by the processor, the following steps can be specifically implemented: When the clearing condition is met, clear the data stored in the medium and high-speed storage medium on the computing node .

It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations There is no such actual relationship or order between the operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (20)

1. A layered cache method, characterized in that it is applied to each computing node of a distributed storage system, the distributed storage system includes a client layer, and the client layer deploys a client process and a server process, The hierarchical caching method includes: Using the client process to monitor the file IO operation request sent by the client to the distributed storage system, and redirecting the file IO operation request to the server process when the file IO operation request is detected; Using the server process to determine whether the target storage location corresponding to the file IO operation request is an aggregation cache layer; If not, read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer; If so, read the data from the aggregation cache layer, and return the data to the client process, so that the client process returns the data to the client. 2. The layered cache method according to claim 1, wherein the process of using the server process to judge whether the target storage location corresponding to the file IO operation request is an aggregation cache layer comprises: Utilize the server process to insert the received file IO operation request into a shared queue; In the shared queue, determine whether the data corresponding to the file IO operation request is cached data; If yes, it is determined that the target storage location corresponding to the file IO operation request is the aggregation cache layer. 3. The layered cache method according to claim 2, wherein, in the shared queue, the process of determining whether the data corresponding to the file IO operation request is cached data comprises: In the shared queue, a data thread is used to determine whether the data corresponding to the file IO operation request is cached data; the data thread is a thread generated when the server process instance is constructed. 4. The layered caching method according to claim 3, wherein the layered caching method further comprises: When the file IO operation request sent by the client to the distributed storage system is detected, the server process is started, and the state of the computing node and other computing nodes adjacent to the computing node are used to dynamically construct the Instance of the server process described above. 5. The layered cache method according to claim 3, wherein the process of reading the data from the aggregation cache layer comprises: The file IO operation request is redirected to the aggregation cache layer through a data thread, so as to read the data from the aggregation cache layer. 6. The layered caching method according to claim 2, wherein, while utilizing the server process to insert the received file IO operation request into the shared queue, the layered caching method also includes: Configure the shared queue with a mutex. 7. The hierarchical caching method according to claim 2, wherein the shared queue is a FIFO queue. 8. The hierarchical caching method according to claim 1, wherein the data corresponding to the file IO operation request includes a file descriptor, read offset and length. 9. The layered caching method according to claim 1, wherein the layered caching method further comprises: The storage location of the data in the aggregation cache layer is determined based on the file path and the computing node to which it belongs. 10. The layered cache method according to claim 1, wherein the layered cache method further comprises: Broadcast the file IO operation request to computing nodes adjacent to the current computing node. 11. The layered caching method according to claim 1, wherein the layered caching method further comprises: Judging whether the data set corresponding to the file IO operation request is greater than the total capacity of the local storage medium; If so, perform cache eviction and replacement operations. 12. The layered caching method according to claim 1, wherein the layered caching method further comprises: A dynamic link library is built based on environment variables; the dynamic link library is used to intercept the file IO operation request. 13. The layered caching method according to claim 1, wherein the process of redirecting the file IO operation request to the server process comprises: The file IO operation request is redirected to the server process through a hash algorithm. 14. The hierarchical caching method according to any one of claims 1-13, wherein the aggregated caching layer is a cache composed of high-speed storage media in each of the computing nodes in the distributed storage system layer. 15. The hierarchical caching method according to claim 14, wherein the high-speed storage medium is NvmeSSD. 16. The layered caching method according to claim 14, wherein the layered caching method further comprises: When the clearing condition is met, clear the data stored in the high-speed storage medium on the computing node. 17. A layered cache system, characterized in that it is applied to each computing node of a distributed storage system, the distributed storage system includes a client layer, and the client layer deploys a client process and a server process, The hierarchical caching system includes: The monitoring module is configured to use the client process to monitor the file IO operation request sent by the client to the distributed storage system, and when the file IO operation request is detected, redirect the file IO operation request to the server process; A processing module, configured to use the server process to judge whether the target storage location corresponding to the file IO operation request is an aggregation cache layer, if not, trigger the first reading module, and if so, trigger the second reading module; The first reading module is configured to read the data corresponding to the file IO operation request from the bottom layer of the distributed storage system, and cache the data to the aggregation cache layer; A second reading module, configured to read the data from the aggregation cache layer, and return the data to the client process, so that the client process returns the data to the client . 18. An electronic device, characterized in that it comprises: memory for storing computer programs; A processor, configured to implement the steps of the hierarchical caching method according to any one of claims 1-16 when executing the computer program. 19. A distributed storage system, characterized in that it includes a storage bottom module and a plurality of nodes, each of which includes a layered client process, a layered server process, and a storage medium, and the storage of each of the nodes The media constitutes the aggregate caching layer, where: The layered client process is used to monitor the file IO operation request sent by the client, and when the file IO operation request is detected, redirect the file IO operation request to the layered server process; The layered server process is used to judge whether the target storage location corresponding to the file IO operation request is the aggregation cache layer, and if not, read the data corresponding to the file IO operation request from the storage bottom layer module , and send the data to the aggregation cache layer, if so, read the data from the aggregation cache layer, and return the data to the layered client process so that the layered client The end process returns the data to the client; The aggregation cache layer is used to store the data sent by the layered server process. 20. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the analysis according to any one of claims 1-16 is realized. Steps of the layer caching method.

Images