CN106874383B - Decoupling distribution method of metadata of distributed file system - Google Patents

Abstract

The invention discloses a method for decoupling and distributing metadata of a distributed file system, comprising: separating metadata of a distributed file system to obtain metadata of a directory, metadata of a directory item, and metadata of a file; The directory metadata is stored centrally in the directory metadata index node, and does not contain pointers to directory entries. A directory operation is performed based on the directory inode. After the metadata of each directory item is divided, the related file metadata is stored in the same node, and a reverse index pointing to the directory metadata is established. The present invention has the following advantages: it reduces the information interaction between nodes when the distributed file system accesses the metadata, and reduces the delay of metadata access; at the same time, by separating the content of the directory, it decouples the connection between the file and the directory. Strong association can achieve high throughput, thus improving the metadata processing efficiency of the distributed file system.

Description

A Distributed File System Metadata Decoupling Distribution Method

technical field

The invention relates to the field of computers, in particular to a method for decoupling and distributing metadata of a distributed file system.

Background technique

Distributed file system is a new type of storage system that supports massive data storage, and is widely used in data centers, supercomputing centers and public cloud platforms. Distributed file systems have many advantages over traditional centralized storage. For example, the storage data can be horizontally expanded, and the storage capacity can be dynamically expanded by adding storage nodes, and the simultaneous improvement of access throughput can be guaranteed. Secondly, compared with the traditional centralized storage, the distributed file system has a flexible fault tolerance strategy, and can use the copy mechanism and erasure code for distributed fault tolerance. The distributed file system can also use cheaper storage and computing devices to build a large-scale storage cluster to ensure access to large amounts of data. However, limited by the access standard of the file system (POSIX), the metadata access of the distributed file system often becomes the bottleneck of its performance. Metadata access often cannot meet the requirements of high throughput and low latency, but in actual systems, more than half of data access needs to go through metadata nodes. In order to solve the scalability of distributed file system metadata, the existing technologies mainly include the following three types:

One is a distributed metadata node extension method based on a dynamic directory tree. This method is characterized by dividing the name space of the distributed file system into different subtrees according to subdirectories, and each subtree is independently stored in a certain node. , and more dynamically adjust the storage nodes according to the access load. The advantage of this method is that it can dynamically adjust the location of access according to the synchronization of the load, but this method cannot solve the path backtracking problem of file access. When accessing a file, it is necessary to access all directories of the entire path, and these directories It is often not stored in the same node, which often causes a large access delay.

The other is a metadata extension method based on a hash algorithm, which is characterized by assigning metadata to different nodes by hashing the files in a directory. The advantage of this method is that it can reduce the load of file access when there are a large number of files in a directory. But it can't solve the scalability problem of the directory.

The third method is to use the key-value database to store file metadata. This method takes advantage of the fast access and low latency of the key-value database, but this method still has paths such as the path search that exists in the first method. problem, still can not solve the problem of low access time delay.

In order to solve the problem of path delay, these methods often cache metadata on the client side, but this brings a lot of inconsistency overhead, which makes it impossible to solve the problem fundamentally.

Contents of the invention

The present invention aims to solve at least one of the above-mentioned technical problems.

Therefore, an object of the present invention is to propose a method for decoupling and distributing metadata of a distributed file system, so as to solve the problems of metadata scalability, low throughput and low delay of the distributed file system.

In order to achieve the above object, the embodiment of the present invention discloses a distributed file system metadata decoupling distribution method, including the following steps: S1: Separating the metadata of the distributed file system to obtain the metadata of the directory index node Data, metadata of directory items, and metadata of files; S2: set the metadata of the directory in the directory index node; S3: divide each directory item according to the distribution of the file, and store it in the node where the file is stored related directory entries, and build an inverted index pointing to the directory metadata.

Further, the directory operation includes creating the directory, deleting the directory, reading the directory, obtaining all metadata of the directory, changing the user group where the directory belongs, and changing the user to which the directory belongs.

Further, it also includes: providing a globally unique identification of the file; calculating a hash value of the global identification of the file to be accessed; and locating the node where the metadata is stored according to the hash value.

Further, the identifier is the full path of the file.

Further, it also includes: when creating a file or directory, creating a node containing all directory entries containing the parent directory path of the file or directory at the node where the file or directory is created; if all or part of the directory entries are already in The node is created, then the remaining directory entries are created.

Further, the method further includes: when deleting a file, deleting the metadata of the node where the file is located and the directory entry metadata corresponding to the node where the file is located pointing to the item of the file.

Further, it also includes: when the operation of reading or deleting a directory is performed, accessing all metadata nodes to obtain all directory entries under the read or deleted directory.

Further, it also includes: providing a client cache, wherein the directory metadata cached by the client is used to determine whether the client has the right to create a file when creating a file; when the client accesses the metadata of a file, Access the metadata of the directory to obtain the access right; when the client has the access right, access the metadata of the file.

Further, it also includes: when the cache of the directory metadata client becomes invalid, changing the authority of the directory of the directory metadata and deleting the directory.

According to the decoupling distribution method of distributed file system metadata in the embodiment of the present invention, all metadata operations on files visit nodes at most twice, and in the case of directory metadata cache, only one node needs to be visited, and the delay The RTT for a round trip is only for one access. Because of the use of key-value storage, the delay for metadata acquisition is very low, which is negligible in the RTT delay of Ethernet, so this method can effectively reduce the distributed file system. The information interaction between each node when accessing metadata reduces the delay of metadata access. At the same time, by separating the content of the directory, the strong correlation between the file and the directory is decoupled, and a high throughput can be achieved. Therefore, the processing efficiency of the distributed file system for metadata is improved.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

1 is a flowchart of a method for decoupling and distributing distributed file system metadata according to an embodiment of the present invention;

Fig. 2 is an overall system architecture diagram of an embodiment of the present invention;

Fig. 3 is a schematic diagram of directory division and decoupling according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

These and other aspects of embodiments of the invention will become apparent with reference to the following description and drawings. In these descriptions and drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited by This restriction. On the contrary, the embodiments of the present invention include all changes, modifications and equivalents coming within the spirit and scope of the appended claims.

The present invention is described below in conjunction with accompanying drawing.

FIG. 1 is a flowchart of a method for decoupling and distributing metadata of a distributed file system according to an embodiment of the present invention. As shown in FIG. 1, the method for decoupling and distributing metadata of a distributed file system according to an embodiment of the present invention includes the following steps:

S1: Separate the metadata of the distributed file system to obtain the metadata of the directory, the metadata of the directory item and the metadata of the file.

S2: Set the metadata of the directory on the index node of the directory.

S3: Divide each directory entry according to the distribution of the files, store the directory entries related to it in the node where the files are stored, and establish a reverse index pointing to the directory metadata.

In one embodiment of the present invention, directory metadata of the file system is centrally stored on one node. In this way, the metadata information of the directory index node does not contain the address pointing to the metadata of the directory item. Only basic directory data is retained, including but not limited to the creation time of the directory, the permission ID of the directory, the group ID of the directory, and the user ID of the directory. On this basis, most of the metadata operations related to the metadata of the directory index node but not related to the directory item metadata will only be performed on the node storing the directory index metadata. Directory operations include creating a directory, deleting a directory, reading a directory, obtaining all metadata of a directory, changing the user group where the directory belongs, and changing the user to which the directory belongs.

In one embodiment of the present invention, a hash-based distributed file metadata storage mechanism is also included. This storage mechanism supports extending the storage and access of file metadata to multiple nodes, so as to achieve the purpose of balancing system load. This algorithm can uniquely determine the identifier of a file globally: when the client performs metadata operations on the file, the client locates the node where the file is stored by calculating the hash value of the globally unique identifier of the file to be accessed. Operate on metadata at this node. This method guarantees that all file metadata operations will modify the information of at most one metadata node.

In one embodiment of the present invention, the identifier is the full path of the file.

In one embodiment of the present invention, a reverse storage method of directory entries is also included. This method divides the metadata of directory items and distributes them to multiple nodes, which ensures that no additional node access overhead is added during the process of file creation and deletion. The allocation method is that when creating a file or directory, it is not necessary to modify the metadata information on nodes other than the node that created the file, but to create a node containing the path of the parent directory of the file or directory at the node that created the file or directory All directory entries. If all or part of these directory entries have already been created at this node, the remaining directory entries are created. This kind of directory entry ensures that when creating and deleting files, multiple metadata nodes will not be modified through this reverse storage method, which reduces the distributed synchronization overhead caused by accessing multiple nodes.

In one embodiment of the present invention, when a file is deleted, the metadata of the node where the file is located and the directory entry metadata corresponding to the node where the file is located are deleted, pointing to the item of the file.

In an embodiment of the present invention, it further includes: when performing the operation of reading a directory or deleting a directory, accessing all metadata nodes to obtain all directory entries under the directory being read or deleted.

In one embodiment of the present invention, it also includes: providing a client cache, wherein the directory metadata cached by the client is used to determine whether the client has the right to create a file when creating a file; the client accesses the metadata of the file , access the metadata of the directory to obtain access rights; when the client has access rights, access the metadata of the file.

In an embodiment of the present invention, it further includes: when the cache of the directory metadata client becomes invalid, modifying the permission of the directory of the directory metadata and deleting the directory.

In order for those skilled in the art to further understand the present invention, the following examples will be described in detail.

As shown in FIG. 2 , a metadata node for storing directories is provided, and the metadata node is used for processing all requests about directory metadata in a distributed file system. During implementation, it uses the Ethernet-based RPC protocol to receive requests from clients by providing a standard POSIX access interface. It is divided into four modules, one is to provide an access interface for performing metadata request operations, and the other is a key-value store for persisting metadata to disk. In addition, there is a module that stores directory content to cache the directory content of each node. When a metadata node is started, it scans every value in the key-value store to build the contents of the directory.

In the processing of directory metadata, the directory metadata includes the creation time of the directory, the authority identifier of the directory, the group identifier of the directory, the user identifier of the directory, and the globally unique identifier of the directory. Wherein, the path length of the directory is a variable-length character string, and the metadata of the other directories are respectively an 8-byte fixed-length identifier. This kind of metadata node supports related operations on directories, including creating directories, deleting directories, reading directories, obtaining all metadata of directories, changing the user group where the directory is located, and changing the user to which the directory belongs.

The directory creation process includes receiving a directory creation request from the client, and the request includes a path for the client to create the directory, a permission mode for the client to create the directory, and a user group ID and user ID for the client to create the directory. The permission mode of the directory includes the read and write permissions of the owner of the directory, the read and write permissions of the user group users, and the read and write permissions of other users. After receiving these requests on the node side, it will first check and confirm the authority of the parent directory of the directory created by the customer. When it is determined that the creation conditions are met, the metadata of the directory will be written into the storage of the directory metadata node. The creation conditions met include the existence of the specified parent directory and the access authority of the parent directory.

For operations such as deleting a directory, reading all the metadata of the directory, changing the user group where the directory is located, and changing the attributes of the directory, when receiving a request from the client, it will first check whether the directory exists and whether it has access rights. When these two items are determined, the corresponding operation is performed on the directory. What is more special is that when deleting a directory, it must be confirmed that its subdirectory and the files in the subdirectory have been deleted.

In the whole architecture, there is logically only one directory metadata node in the whole system. All directory metadata are stored in a metadata node. In order to ensure its reliability, the method of synchronous backup can be used instead of the method of distributed expansion.

FIG. 2 also describes a node for storing file metadata. This metadata node is used to process all metadata requests of the distributed file system and store metadata of the distributed file system. It includes a standard POSIX access interface that supports file operations, a metadata management layer for classified storage of metadata, and a key-value store that persists metadata to disk.

The metadata of each file includes the global unique identifier of its parent directory mentioned in the first aspect, the file name, the access time of the file, the permission of the file, the user group identification of the file, the modification time of the file, and the access time of the file content , the size of the file, the size of the file block, the global identifier of the metadata node that created the file, and the metadata node where the file was created. The globally unique identifier of its parent directory and the name of the file name are combined to form a variable-length string. Other metadata fields are fixed-length 8-byte fields. This metadata supports operations related to file metadata, including file creation, file deletion, obtaining file metadata, changing the user group of the file, changing the user to which the file belongs, changing the permissions of the file, reading and writing of the file, changing file size.

Among them, the file creation process is divided into two stages. In the first stage, the client needs to first access the directory to which the created file belongs, and the client needs to access the directory metadata node described in the first aspect to determine whether there is an file permissions. When it is determined that it is a legal creation request, enter the second stage, the client will send the request to the file metadata node, and the file node will use the string that combines the global unique identifier of the parent directory and the name of the file to go to the second stage. Indexing is performed on the file metadata node, and when it is determined that the file does not exist on the node, the file is successfully created.

For metadata operations on other files, you first need to find the file, then determine whether you have access rights, and if you have access rights, modify it accordingly. In these modifications, the modification of file reading and writing must wait until the data of the file has been written into the back-end storage system or has been successfully read from the back-end storage system before the data modification can be completed. This process needs to be defined as a transaction. process.

The metadata node is a network node based on RPC communication. When initializing, it needs to open the corresponding service port and start the database for storing directory metadata. For the processing of metadata operation requests, the requests are processed concurrently in a multi-threaded manner.

As shown in Figure 2, there are only a plurality of file metadata nodes logically in the entire system. The metadata of all files determines the location of the node where the file is stored by calculating the string composed of the unique identifier of its parent directory and the file name. This calculation method is guaranteed using a consistent hash algorithm. This method can ensure good scalability, and can also support dynamic expansion of metadata nodes of files.

As shown in Figure 2, its architecture also includes a distributed file system client with cache. On this client, applications can access distributed storage by directly accessing the libraries provided by the distributed file system. In the Linux system, the user can directly mount the file system to the disk on the client after calling this library through the user state file system (FUSE). When the client starts, a directory cache will be created on the client, and this cache is placed on the access library.

When the client accesses, it constructs different access policies for different access processes through the RPC protocol. For the metadata operation of the directory, including mkdir to create the directory, getattr to get the attributes of a file, directory, or folder (only the metadata of the directory is obtained here), chmod file/directory permission setting command, chown is a multi-user multi-tasking operating system , all files have their owners (Owner). The client will directly access the directory metadata node, and the client will give different parameters for different operations. After the node side receives the request, it directly processes it and returns it to the client side. For file metadata operations, including open, read, write, getattr, truncate, etc., it first needs to access the directory cache to check whether its parent directory has access rights. If there is no relevant information in the directory cache, the client will first access the directory The metadata node caches the metadata of the parent directory into the directory cache of the client, and then checks the permissions. After confirming that the permissions are granted, the client initiates a metadata operation request with the file metadata node. It should be pointed out that for the readdir (read directory content) operation, the client will initiate a request to read the directory content for all file metadata nodes and directory metadata nodes, and these nodes will return the directory content stored by them to the client On the client side, the client organizes these contents into an ordered list and returns it to the user. For the operation of rmdir (deleting a directory), the client needs to initiate a request to obtain directory content from all file metadata nodes and directory metadata nodes. When all returned requests are empty, it is confirmed that the deletion of the directory is legal. Only then can a delete directory request be initiated to the file metadata node.

Regarding the client's cache, its instance also needs to handle the following situations. Since there are some clients modifying the content of the directory during the running process, the expiration time of the cache needs to be set for the cached directory metadata, and the change of the directory permission of the directory metadata and the deletion of the directory need to wait for all the directory metadata The client's cache is invalid to continue the operation. Therefore, when a client sends these two kinds of requests, the request will not be executed immediately, but will wait until the cache expiration time set by the directory metadata node expires before updating and deleting the directory.

The client as shown in Figure 2 also needs a configuration file to store the global map. The global map includes the IP address of the directory metadata node and the file metadata node, and the globally unique number of each file metadata node. Based on this global map and number, the client can determine the location of the file node through the hash algorithm , and can exchange information with corresponding metadata nodes through the network. During the initialization process of the client, the client will read in the global map and establish a heartbeat link with the directory metadata node and the file metadata node, and will determine whether the directory and file metadata nodes are still working normally every time.

As shown in Figure 3, in the metadata organization process, the metadata of the file needs to be segmented. In this process, this example defines 4 metadata structures, including the metadata of the directory (d-inode), The content of the directory (d-entry), the metadata of the file (f-inode), the content of the file (f-content). In the traditional file system, d-inode can index to f-inode through d-entry, and f-inode can index to the specific content of the file. In this example, however, a new metadata organization method is proposed. This method of metadata sharding involves each node self-organizing its own directory content. As shown in Figure 2, the original d-entry is allocated to each d-inode through decoupling. This solves the strong coupling relationship that d-entry brings to files and directories. Specifically, in the implementation process, when creating a file, it first reads its parent directory to create a directory. As explained in the previous method, the metadata of its parent directory can be obtained through the client's directory cache. The client allocates files to a file metadata node for storage, and this allocation method is based on a consistent hash allocation method. While storing, the file is added to the directory content cache of the node where the file resides in the directory content cache of the metadata of the file, thereby avoiding interaction with other nodes and reducing access delay. When it is necessary to obtain directory content, it only needs to visit each node, and the decoupled directory content can be aggregated into a complete directory content and sent to the application.

As shown in Figure 2, during the execution of the instance, the metadata needs to be persisted through the key-value database. This example uses a method of storing metadata in the key-value database. The method uses the name of the file or directory or The globally unique identifier is used as the key, and the metadata of the file or directory is used as the value, which is stored in a key-value database.

In the directory metadata node, its key-value database uses the path of the file as the key for searching, and stores the subsequent metadata as the value in the directory metadata node.

In the file metadata node, the string composed of the global unique identifier of the parent directory and the file name is used as the key for searching, and other metadata is used as the value, which is stored in the file metadata node.

In the client of the distributed file system, metadata is cached using the same key-value method as the directory metadata node. The specific process is that when storing the metadata of a directory, the path of the directory is used as the key, and other metadata of the directory are stored as values. At the same time, a globally unique identifier of the directory is added to the back end of the key value. Identifiers are managed by the metadata node side of the catalog. When a file is created, the unique identifier and file name of its parent directory are used as the key, and the rest of the metadata of the file is stored as the value. At this time, since the files are distributed on each node, if a unique node is used to create and manage a unique identifier for the file, it will inevitably make this node a performance bottleneck. For this purpose, this instance uses the identifier of the node where the file is created and the unique identifier created for this file by the metadata node of the file creation on this node to form a globally unique file identifier. This flag will not change if the file is changed later, such as renaming or moving the path. When saving the metadata of the file, this identifier will be stored at the end of the key value.

In the internal data structure of the key-value store, this instance uses different key-value stores for different data services. Among them, the directory metadata node uses a B-tree-based key-value storage database, the file metadata node uses a hash-based key-value storage database, and the client of the distributed file system uses a memory-based database to store the metadata cache of the directory.

In terms of metadata storage optimization, this example uses variable-length keys and fixed-length values. This approach requires each metadata field to be deterministic. Wherein, each value of the metadata field of the directory metadata is of fixed length. Each value of the metadata field of the file metadata node is also fixed-length. When storing, a field with a certain length is directly stored in the value of the key-value store without serialization and deserialization. In addition, this method does not require an additional memory data structure to cache metadata in storage, and directly caches metadata in the cache of the key-value database.

In the implementation result of the present invention, all the metadata operations for the file visit the node twice at most, and in the case of directory metadata cache, only need to visit the node once, and its delay is only the RTT of one visit round trip, due to the use of Key-value storage, so the delay for metadata acquisition is very low, which is negligible in the RTT delay of Ethernet, so this method can effectively reduce the information between nodes when the distributed file system accesses metadata Interaction reduces the delay of metadata access. At the same time, by separating the content of the directory, the strong correlation between the file and the directory is decoupled, which can achieve high throughput, thereby improving the distributed file system for metadata. processing efficiency.

In addition, other components and functions of the method for decoupling and distributing distributed file system metadata in the embodiment of the present invention are known to those skilled in the art, and will not be repeated in order to reduce redundancy.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and modifications can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the invention is defined by the claims and their equivalents.

Claims (8)

1. A method for decoupling and distributing metadata of a distributed file system is characterized by comprising the following steps:

s1: separating metadata of the distributed file system to obtain metadata of a directory, metadata of a directory entry and metadata of a file;

s2: setting the metadata of the directory at a directory inode;

s3: dividing each directory entry according to the distribution condition of the files, storing the directory entries related to the directory entries in the nodes stored in the files, and establishing a reverse index pointing to the metadata of the directory;

further comprising: when a file or a directory is created, creating all directory entries containing a parent directory path of the file or the directory at a node where the file or the directory is created; if all or part of the directory entry has already been created at that node, the remaining directory entries are created.

2. The method of claim 1, wherein the directory operations include creating a directory, deleting a directory, reading a directory, obtaining all metadata of a directory, changing a user group in which the directory is located, and changing a user to which the directory belongs.

3. The method for decoupled distribution of distributed file system metadata according to claim 1, further comprising:

providing an identification of a globally unique certain file;

calculating the hash value of the global identification of the file to be accessed;

and positioning the nodes stored by the metadata according to the hash value.

4. The method of claim 3, wherein the identifier is a complete path of the file.

5. The method for decoupled distribution of distributed file system metadata according to claim 1, further comprising:

when a file is deleted, deleting the metadata of the node where the file is located and the directory entry metadata corresponding to the node where the file is located to point to the entry of the file.

6. The method for decoupled distribution of distributed file system metadata according to claim 1, further comprising:

when a read directory or delete directory operation is performed, all metadata nodes are accessed to obtain all directory entries under the read directory or delete directory.

7. The method for decoupled distribution of distributed file system metadata according to claim 1, further comprising:

providing a client cache, wherein the directory metadata cached by the client is used for determining whether the client has the authority of creating the file when the client creates the file;

when the client accesses the metadata of the file, the client accesses the metadata of the directory to acquire access authority;

when the client has access rights, metadata of the file is accessed.

8. The method for decoupled distribution of distributed file system metadata according to claim 7, further comprising:

and when the cache of the catalog metadata client fails, the permission of the catalog metadata is changed and the catalog is deleted.