CN106227839A - The expansion method of a kind of lustre file system and device - Google Patents

Abstract

The invention provides expansion method and the device of a kind of lustre file system, the method comprises determining that lustre file system to be expanded, and this lustre file system includes at least one client；At least one storage server is utilized to create RAID storage pool；The RAID storage pool of establishment is joined in described lustre file system, obtains the lustre file system after dilatation；By the lustre file system after each client carry dilatation.Therefore, the capacity of lustre file system can be expanded by this programme.

Description

A method and device for expanding the capacity of a lustre file system

technical field

The invention relates to the technical field of computers, in particular to a method and device for expanding the capacity of a lustre file system.

Background technique

The Luster file system is a Linux file system developed by the U.S. Department of Energy and Intel Corporation. It has the advantages of high concurrency and fast reading and writing. 90% of the world's top 500 supercomputers use this file system.

The Luster file system is a cluster-oriented storage architecture that can provide massive capacity storage, high concurrency, and high read/write solutions. However, after the Luster file system is used for a period of time, the capacity and performance usually cannot meet the computing needs.

Contents of the invention

The invention provides a method and device for expanding the capacity of a lustre file system, which can expand the capacity of the lustre file system.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

On the one hand, the present invention provides a method for expanding the capacity of a lustre file system. The luster file system to be expanded is determined, and the luster file system includes at least one client, and further includes:

S1: using at least one storage server to create a RAID (Redundant Arrays of Independent Disks, disk array) storage pool;

S2: adding the created RAID storage pool to the lustre file system to obtain the expanded lustre file system;

S3: Mount each client to the expanded lustre file system.

Further, the luster file system also includes at least one OSS (Object StorageServers, object storage server);

Adding the created RAID storage pool to the lustre file system includes: adding the created RAID storage pool to a queue of the OSS.

Further, after the S3, it also includes: checking data consistency, and using a predetermined test tool to detect the capacity increase and bandwidth increase corresponding to the expanded lustre file system.

Further, before the S1, it also includes: setting the luster file system to stop state, and executing the umount command on each of the clients to unmount the file system; determining at least one storage server, and Install the Linux operating system on the storage server, upgrade the lustre kernel, compile the lustre server end; set the upgrading scheme of the lustre file system;

The S1 includes: according to the upgrade solution, using the at least one storage server to create a RAID storage pool.

Further, the setting the upgrade plan of the luster file system includes: determining the role of each storage server, and planning the RAID array mode and space size.

On the other hand, the present invention provides a device for expanding the luster file system, including:

A determining unit, configured to determine the luster file system to be expanded, and the luster file system includes at least one client;

Create a unit for creating a RAID storage pool with at least one storage server;

A processing unit, configured to add the created RAID storage pool to the lustre file system to obtain the expanded lustre file system;

The mounting unit is used to mount each client to the expanded lustre file system.

Further, the luster file system also includes at least one OSS;

The processing unit is specifically configured to add the created RAID storage pool to the queue of the OSS.

Further, the expansion device of the luster file system further includes: a detection unit, which is used to check data consistency, and use a predetermined test tool to detect the corresponding capacity increase and bandwidth increase of the expanded lustre file system .

Further, the device for expanding the luster file system also includes: a setting unit, configured to set the luster file system to a stopped state, and execute the umount command on each of the clients to unmount the file system; determine at least one storage server, and install the Linux operating system, upgrade the lustre kernel, and compile the lustre server end on each of the storage servers; the upgrading scheme of the described lustre file system is set;

The creating unit is specifically configured to use the at least one storage server to create a RAID storage pool according to the upgrading scheme.

Further, the setting unit is specifically configured to set the upgrade plan of the lustre file system by determining the role of each of the storage servers, and planning the RAID array mode and space size.

The invention provides a method and device for expanding the capacity of a lustre file system. The luster file system to be expanded is determined, and the luster file system includes at least one client; at least one storage server is used to create a RAID storage pool; and the created RAID is stored The pool is added to the luster file system to obtain the expanded lustre file system; each client is mounted with the expanded lustre file system. Therefore, the present invention can expand the capacity of the luster file system.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are For some embodiments of the present invention, those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a flow chart of a method for expanding the capacity of a luster file system provided by an embodiment of the present invention;

Fig. 2 is a flow chart of another method for expanding the capacity of a luster file system provided by an embodiment of the present invention;

Fig. 3 is a schematic diagram of an expansion device of a luster file system provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of another device for expanding the luster file system provided by an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the protection of the present invention. scope.

As shown in Figure 1, the embodiment of the present invention provides a method for expanding the luster file system, which may include the following steps:

Step 101: Determine the luster file system to be expanded, and the luster file system includes at least one client.

Step 102: Create a RAID storage pool by using at least one storage server.

Step 103: Add the created RAID storage pool to the lustre file system to obtain the expanded lustre file system.

Step 104: Mount each client to the expanded lustre file system.

The embodiment of the present invention provides a method for expanding the capacity of a lustre file system, which determines the luster file system to be expanded, and includes at least one client in the luster file system; utilizes at least one storage server to create a RAID storage pool; and stores the created RAID The pool is added to the luster file system to obtain the expanded lustre file system; each client is mounted with the expanded lustre file system. Therefore, the embodiment of the present invention can expand the capacity of the luster file system.

In an embodiment of the present invention, in order to illustrate a way of expanding the luster file system, the luster file system also includes at least one OSS;

Adding the created RAID storage pool to the lustre file system includes: adding the created RAID storage pool to a queue of the OSS.

Specifically, when the created RAID storage pool is added to the OSS queue, the number of OSSs in the OSS queue increases accordingly.

In one embodiment of the present invention, in order to be able to determine whether the expansion operation of the luster file system is up to standard, after step 104, it further includes: checking the data consistency, and using a predetermined test tool to detect the expansion operation. The corresponding capacity increase and bandwidth increase of the Luster file system.

In detail, after the RAID storage pool is added to the resource pool of the original luster file system, it can be detected that the capacity and bandwidth of the original luster file system increase accordingly.

In one embodiment of the present invention, in order to illustrate a method of creating a RAID storage pool, before step 102, it further includes: setting the luster file system to stop state, and on each of the clients Execute the umount command to unload the file system; determine at least one storage server, and install the Linux operating system, upgrade the lustre kernel, and compile the lustre server end on each of the storage servers; set the upgrading scheme of the lustre file system;

The step 102 includes: according to the upgrade solution, using the at least one storage server to create a RAID storage pool.

In detail, the upgrade plan can be planned in advance according to the actual needs, and the preparations before the expansion can be done, so that the RAID storage pool can be created according to the planned upgrade plan, so that the created RAID storage pool can be used to support the lustre file system. To expand.

In detail, in one embodiment of the present invention, the setting the upgrade plan of the lustre file system includes: determining the role of each storage server, and planning the RAID array mode and space size.

As shown in Figure 2, an embodiment of the present invention provides another method for expanding the luster file system, which specifically includes the following steps:

Step 201: Determine the luster file system to be expanded, and the luster file system includes at least one client and at least one OSS.

In detail, a Luster file system can provide massive capacity storage, and high concurrency, high reading and writing solutions, but after a period of use, the capacity and performance usually cannot meet the computing needs, in order to alleviate or eliminate this situation , the luster file system can be expanded.

In detail, a luster file system usually includes three parts: client, OSS and MDS (Metadata Server, metadata server). Among them, MDS serves one MDT (Metadata Target, metadata target), one OSS serves at least one OST (Object Storage Target, object storage target), and the client uses the standard POSIX interface (Portable Operating System Interface, portable operating system interface) provides users with access to the file system.

MDS can store the basic information of the Luster file system, and is responsible for managing the namespace, directory structure, and user permissions of the entire system, and maintaining the consistency of file system data.

The client can read the metadata saved on the MDT through the MDS. When the client reads and writes files, it gets file information from MDS and data from OSS. In detail, the client can communicate with the MDS/OSS through the LNET protocol.

Step 202: Set the luster file system to stop state, and execute the umount command on each client to unmount the file system.

In detail, before performing the expansion operation of the luster file system, you can stop the existing luster file system and umount the file system on all clients.

Specifically, the umount command under the Linux system is used to unmount the file system currently mounted in Linux. Among them, the umount command can be: umount[-ahnrvV][-t<file system type>][file system].

Step 203: Determine at least one storage server, install a Linux operating system on each storage server, upgrade the lustre kernel, and compile the lustre server.

In the embodiment of the present invention, the storage server can be used to create a RAID storage pool and add the created RAID storage pool to the original luster file system to expand the original luster file system.

Therefore, a certain number of storage servers may be determined first, and corresponding preprocessing may be performed on the determined storage servers. In detail, the number of storage servers can be determined by staff according to actual needs.

Step 204: Determine the role of each storage server, and plan the RAID array mode and space size by setting the upgrade plan of the lustre file system.

In detail, before creating a RAID storage pool, a luster file system upgrade plan can be planned in advance, and a RAID storage pool can be created according to the upgrade plan.

Step 205: Create a RAID storage pool by using at least one determined storage server according to the set upgrade scheme.

Step 206: Add the created RAID storage pool to the OSS queue of the lustre file system to obtain the expanded lustre file system.

After the RAID storage pool is added to the resource pool of the original luster file system, the capacity of the luster file system can be expanded. For example, there can be 2 OSSs in the original Luster file system, and 6 OSSs in the expanded Luster file system.

In detail, the RAID storage pool can be directly added to the OSS queue of the luster file system to achieve capacity expansion.

In an embodiment of the present invention, the implementation code for adding the new storage server to the original luster file system can be: mkfs.lustre-ost--fsname=lustre--mgsnode=192.168.1.201@o2ib--index=10/dev /mapper/ost1.

Step 207: Mount each client to the expanded lustre file system.

In detail, the client can use the client end that comes with luster to mount the expanded lustre file system.

Step 208: Check the data consistency, and use a predetermined test tool to detect the corresponding capacity increase and bandwidth increase of the expanded lustre file system.

In detail, if the expansion of the luster file system is successful, the test result can be that the data is consistent, the capacity of the luster file system increases accordingly, and the bandwidth increases accordingly.

In the embodiment of the present invention, multiple storage server hard disks are used to create a RAID storage pool and add them to the original luster file system to realize the expansion of the original luster file system, and this expansion operation may not cause any damage to the original data. influences. By expanding the Lustre file system, it is possible to achieve: 1) High concurrency of writing data, effectively increasing the number of concurrent reads and writes of multi-node data to the file system; 2) Effectively increasing bandwidth to achieve high bandwidth; 3) Massive storage, which can Integrate the storage space of multiple newly added storage servers; 4) Unified management of data; 5) Global unified namespace.

As shown in Figure 3, an embodiment of the present invention provides a device for expanding the luster file system, including:

A determination unit 301, configured to determine a luster file system to be expanded, and the luster file system includes at least one client;

A creating unit 302, configured to use at least one storage server to create a RAID storage pool;

The processing unit 303 is configured to add the created RAID storage pool to the lustre file system to obtain the expanded lustre file system;

Mounting unit 304, configured to mount each client to the expanded lustre file system.

In one embodiment of the present invention, the luster file system further includes at least one OSS;

The processing unit 303 is specifically configured to add the created RAID storage pool to the queue of the OSS.

In an embodiment of the present invention, please refer to FIG. 4 , the device for expanding the luster file system may further include: a detection unit 401, configured to check data consistency, and use a predetermined testing tool to detect the expanded The capacity increase and bandwidth increase corresponding to the luster file system.

In an embodiment of the present invention, please refer to FIG. 4, the apparatus for expanding the capacity of the luster file system may further include: a setting unit 402, configured to set the luster file system to a stop state, and on each of the clients Execute the umount command to unload the file system; determine at least one storage server, and install the Linux operating system, upgrade the lustre kernel, and compile the lustre server end on each of the storage servers; set the upgrading scheme of the lustre file system;

The creating unit 302 is specifically configured to use the at least one storage server to create a RAID storage pool according to the upgrade solution.

In one embodiment of the present invention, the setting unit 402 is specifically configured to set the upgrade plan of the lustre file system by determining the role of each storage server, and planning the RAID array mode and space size.

The information exchange and execution process among the units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific content can refer to the description in the method embodiment of the present invention, and will not be repeated here.

In summary, each embodiment of the present invention has at least the following beneficial effects:

1. In the embodiment of the present invention, determine the luster file system to be expanded, and include at least one client in the luster file system; use at least one storage server to create a RAID storage pool; add the created RAID storage pool to the lustre file In the system, the expanded lustre file system is obtained; each client is mounted with the expanded lustre file system. Therefore, the embodiment of the present invention can expand the capacity of the luster file system.

2. In the embodiment of the present invention, multiple storage server hard disks are used to create a RAID storage pool and add them to the original luster file system, which can realize the expansion of the original luster file system, and this expansion operation may not cause damage to the original data. any impact. By expanding the Lustre file system, it is possible to achieve: 1) High concurrency of writing data, effectively increasing the number of concurrent reads and writes of multi-node data to the file system; 2) Effectively increasing bandwidth to achieve high bandwidth; 3) Massive storage, which can Integrate the storage space of multiple newly added storage servers; 4) Unified management of data; 5) Global unified namespace.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or sequence. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Those of ordinary skill in the art can understand that all or part of the steps to realize the above method embodiments can be completed by program instructions related hardware, and the aforementioned programs can be stored in a computer-readable storage medium. When the program is executed, the It includes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that the above descriptions are only preferred embodiments of the present invention, and are only used to illustrate the technical solution of the present invention, and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims (10)

1. A method for expansion of a lustre file system, characterized in that the luster file system to be expanded is determined, and at least one client is included in the luster file system, further comprising: S1: using at least one storage server to create a disk array RAID storage pool; S2: adding the created RAID storage pool to the lustre file system to obtain the expanded lustre file system; S3: Mount each client to the expanded lustre file system. 2. The method of claim 1, wherein, The luster file system also includes at least one object storage server OSS; Adding the created RAID storage pool to the lustre file system includes: adding the created RAID storage pool to a queue of the OSS. 3. The method of claim 1, wherein, After the S3, it further includes: checking data consistency, and using a predetermined test tool to detect the capacity increase and bandwidth increase corresponding to the expanded lustre file system. 4. The method according to any one of claims 1 to 3, characterized in that, Before the S1, it further includes: setting the luster file system to stop state, and executing the umount command on each of the clients to unmount the file system; determining at least one storage server, and Install the Linux operating system on the server, upgrade the luster kernel, compile the luster server side; set the upgrade plan for the luster file system; The S1 includes: according to the upgrade solution, using the at least one storage server to create a RAID storage pool. 5. The method of claim 4, wherein, The upgrade plan of setting the luster file system includes: determining the role of each of the storage servers, and planning the RAID array mode and space size. 6. A capacity expansion device for a lustre file system, comprising: A determining unit, configured to determine the luster file system to be expanded, and the luster file system includes at least one client; Create a unit for using at least one storage server to create a disk array RAID storage pool; A processing unit, configured to add the created RAID storage pool to the lustre file system to obtain the expanded lustre file system; The mounting unit is used to mount each client to the expanded lustre file system. 7. The expansion device of the luster file system according to claim 6, characterized in that, The luster file system also includes at least one object storage server OSS; The processing unit is specifically configured to add the created RAID storage pool to the queue of the OSS. 8. The expansion device of the luster file system according to claim 6, characterized in that, It further includes: a detection unit, which is used to check data consistency, and use a predetermined testing tool to detect the capacity increase and bandwidth increase corresponding to the expanded lustre file system. 9. The expansion device of the luster file system according to any one of claims 6 to 8, characterized in that, Further comprising: a setting unit, configured to set the luster file system to a stopped state, and execute a umount command on each of the clients to unload the file system; determine at least one storage server, and set the file system on each of the storage servers Install the Linux operating system, upgrade the luster kernel, and compile the luster server side; set the upgrade plan for the luster file system; The creating unit is specifically configured to use the at least one storage server to create a RAID storage pool according to the upgrading solution. 10. The expansion device of the luster file system according to claim 9, characterized in that, The setting unit is specifically configured to set the upgrade plan of the lustre file system by determining the role of each storage server, and planning the RAID array mode and space size.