CN108846136A - A kind of optimization method of distributed type assemblies, device, system and readable storage medium storing program for executing - Google Patents

Abstract

The invention discloses a kind of optimization methods of distributed type assemblies, by creating N number of target directory, N number of metadata pond and at least N number of data pool, metadata pond and data pool correspondence are bound, with for each metadata pond distribute Metadata Service cluster, form it is N number of using each target directory as mount point and with the one-to-one file system in metadata pond.And since N is the positive integer greater than 1, so, when receiving processing request, unified service can be externally provided by multiple file system, processing request can be effectively relieved waits in line situation, so that each processing request can be responded as early as possible, promotes the access efficiency of user.In addition, the invention also discloses optimization device, system and the computer readable storage medium of a kind of distributed type assemblies, effect is as above.

Description

An optimization method, device, system and readable storage medium of a distributed cluster

technical field

The present invention relates to the field of computer technology, in particular to a distributed cluster optimization method, device, system and readable storage medium.

Background technique

At present, although cluster performance can be optimized by setting up metadata service clusters for distributed clusters, from the perspective of the server, there is still only one file system for external services. For example, when it is necessary to manage user rights, samba combination is often used to map part of the directory, set permissions on the user on the directory, make the directory visible or invisible to the user, and have read-only or read-write access to the directory, etc. authority. Although, from the perspective of the client, each user can only see the content that he has permission to, but from the perspective of the server, the same file system provides unified services to the outside world. Once there is a large concurrency of multiple clients, since the server only has one file system for external services, a large number of processing requests will need to wait in line, and cannot be responded in time, which makes the user's access efficiency low and affects the user experience.

Therefore, it is a technical problem that those skilled in the art need to solve at present if the queuing and waiting situation of processing requests is alleviated to improve the user's access efficiency.

Contents of the invention

The purpose of the present invention is to provide an optimization method, device, system and readable storage medium of a distributed cluster, which can alleviate the queuing and waiting situation of processing requests, so as to improve the access efficiency of users.

In order to solve the above technical problems, the present invention provides a distributed cluster optimization method, including:

Create N target directories, N metadata pools and at least N data pools;

Correspondingly binding the metadata pool and the data pool, and assigning a metadata service cluster to each of the metadata pools to form N file systems;

Mounting the N file systems with each of the target directories as mount points;

Wherein, N is a positive integer greater than 1; one said file system corresponds to one said metadata pool.

Preferably, the creation of N target directories, N metadata pools and at least N data pools is specifically:

Create N target directories, N metadata pools and at least N data pools under the Linux local system;

Wherein, the Linux local system does not mount the cluster file system.

Preferably, the Linux local system has mounted the cluster file system, and the creation of N target directories, N metadata pools and at least N data pools is specifically:

Create N said target directories under the Linux local system;

Uninstall the cluster file system, and create N metadata pools and at least N data pools.

Preferably, the Linux local system has mounted the cluster file system, and before creating N target directories, N metadata pools and at least N data pools, it also includes:

Unmount the cluster file system.

Preferably, before creating N target directories, N metadata pools and at least N data pools, it further includes:

Obtain the concurrency degree of the client, and determine the value of N according to the concurrency degree;

Wherein, the degree of concurrency is positively correlated with the value of N.

In order to solve the above technical problems, the present invention provides a distributed cluster optimization device, including:

Create a module for creating N target directories, N metadata pools and at least N data pools;

A binding allocation module, configured to bind the metadata pools and the data pools respectively, and allocate metadata service clusters to each of the metadata pools to form N file systems;

a mount module, configured to mount N file systems using each of the target directories as a mount point;

Wherein, N is a positive integer greater than 1; one said file system corresponds to one said metadata pool.

Preferably, the creation module is specifically used for:

Create N target directories, N metadata pools and at least N data pools under the Linux local system;

Wherein, the Linux local system does not mount the cluster file system.

Preferably, it also includes:

The unloading module is used for unloading the mounted cluster file system of the Linux local system before triggering the creation module.

In order to solve the above technical problems, the present invention provides a distributed cluster optimization system, including:

memory for storing the optimization program;

A processor, configured to implement the steps of any one of the optimization methods for distributed clusters as described above when executing the optimization program.

In order to solve the above technical problems, the present invention provides a computer-readable storage medium, on which an optimization program is stored, and when the optimization program is executed by a processor, any one of the above-mentioned distribution The steps of the optimization method for clusters.

The optimization method of the distributed cluster provided by the present invention creates N target directories, N metadata pools and at least N data pools, binds the metadata pools and data pools correspondingly, and allocates metadata for each metadata pool. The data service cluster forms N file systems with each target directory as the mount point and one-to-one correspondence with the metadata pool. And because N is a positive integer greater than 1, when a processing request is received, a unified service can be provided externally through multiple file systems, which can effectively alleviate the queuing situation of processing requests, so that each processing request can be responded as soon as possible. Improve user access efficiency. In addition, the present invention also provides a distributed cluster optimization device, system and computer-readable storage medium, with the above effects.

Description of drawings

In order to illustrate the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. As far as people are concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Fig. 1 is a flowchart of a method for optimizing a distributed cluster provided by an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a cluster external storage system provided by an embodiment of the present invention;

Fig. 3 is a schematic composition diagram of a distributed cluster optimization device provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a distributed cluster optimization system provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The purpose of the present invention is to provide an optimization method, device, system and readable storage medium of a distributed cluster, which can alleviate the queuing and waiting situation of processing requests, so as to improve the access efficiency of users.

In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

FIG. 1 is a flowchart of a distributed cluster optimization method provided by an embodiment of the present invention. As shown in Figure 1, the optimization method of the distributed cluster provided by this embodiment includes:

S10: Create N target directories, N metadata pools and at least N data pools.

Among them, N is a positive integer greater than 1, and the specific value of N can be directly set in advance, such as directly setting a positive integer greater than 1 such as fixed values 2, 3, 4 and 5, or it can only be set in advance according to specific conditions. The calculation program for calculation, for example, can be set to calculate the specific value of N according to the concurrency degree of the client. The present invention does not limit the setting method of the specific value of N.

The target directory, metadata pool, and data pool can be created directly under the Linux local system that has not mounted the cluster file system before, or can be created under the Linux local system that has mounted the cluster file system (however, when the cluster file system has been mounted When creating metadata pools and data pools under the Linux local system of the cluster file system, it should be ensured that the original cluster file system has been uninstalled), that is, as long as the cluster file system is not currently mounted on the Linux local system, the implementation of this embodiment will not be affected.

In addition, in the Linux local system where the cluster file system is not currently mounted, the creation order of the target directory, metadata pool, and data pool is not in any order. Under the Linux local system where the cluster file system is currently mounted, the following two situations should be distinguished:

1. Create the target directory under the Linux local system first, and then uninstall the cluster file system. After the cluster file system is uninstalled, create a metadata pool and a data pool.

2. Uninstall the cluster file system first, and then create the target directory, metadata pool, and data pool after the cluster file system is uninstalled. Likewise, at this point, the creation order of the target directory, metadata pool, and data pool is in no particular order.

S11: Bind metadata pools and data pools correspondingly, and assign metadata service clusters to each metadata pool to form N file systems.

Wherein, a file system corresponds to a metadata pool, that is, there is a one-to-one correspondence between the file system and the metadata pool. In addition, the metadata pool and the data pool may have a one-to-one relationship or a one-to-many relationship. In specific implementation, the specific corresponding relationship between the metadata pool and the data pool may be determined according to the actual application scenario. The present invention There is no limit to this.

The metadata service cluster includes the activation metadata service and the backup metadata service, and the number of the activation metadata service and the backup metadata service matches the specific application scenario. Allocating a metadata service cluster to a metadata pool is actually allocating an activation metadata service and a backup metadata service to a metadata pool. The file system is used to provide external services when a processing request is received, including a metadata pool correspondingly bound to the data pool, and the data generated when responding to the processing request is automatically generated in the target directory.

S12: Mount N file systems using each target directory as a mount point.

The file system is mounted with each target directory as the mount point, that is, each target directory is used as the entry directory of the file system, similar to the C:, D:, E: and other drive letters used to access different partitions in Windows.

It can be seen that the optimization method for distributed clusters provided in this embodiment creates N target directories, N metadata pools, and at least N data pools, binds metadata pools and data pools correspondingly, and creates The pool allocates a metadata service cluster, forming N file systems with each target directory as a mount point and one-to-one correspondence with the metadata pool. And because N is a positive integer greater than 1, when a processing request is received, a unified service can be provided externally through multiple file systems, which can effectively alleviate the queuing situation of processing requests, so that each processing request can be responded as soon as possible. Improve user access efficiency.

In order to enable those skilled in the art to better understand the technical solution provided by the present invention, the present invention will be described in detail below by taking the formation of two file systems as an example in conjunction with the accompanying drawings.

FIG. 2 is a schematic structural diagram of a cluster external storage system provided by an embodiment of the present invention. As shown in FIG. 2, in this embodiment, two target directories, two metadata pools and two data pools are created. Among them, each target directory corresponds to a metadata pool and a data pool, and each metadata pool is assigned two activation metadata services and two backup metadata services, and finally the cluster external storage system 2 forms two file systems, respectively file system A 20 and file system B 21. Specifically, the file system A 20 includes a first metadata pool 201 and a first data pool 202, and the first metadata pool 201 is allocated with two first activation metadata services 2011 and two first backup metadata services 2012 The file system B 21 includes a second metadata pool 211 and a second data pool 212, and the first metadata pool 211 is allocated with two second active metadata services 2111 and two second backup metadata services 2112.

In a specific implementation, the nodes constituting the distributed cluster may be Linux that does not mount the cluster file system. In this case, in order to simplify this optimization method, based on the above-mentioned embodiment, as a preferred implementation mode, step S10 specifically for:

Create N target directories, N metadata pools and at least N data pools under the Linux local system;

Among them, the Linux local system does not mount the cluster file system.

In the optimization method for distributed clusters provided by this implementation, N target directories, N metadata pools, and at least N data pools are created directly under the Linux local system, compared to the cluster file system that has been mounted For creating N target directories, N metadata pools, and at least N data pools under the Linux local system, there is no need to unmount the original cluster file system again to ensure that the Linux local system is not currently mounted on the cluster when creating metadata pools and data pools The file system, the process of forming multiple file systems is simpler, and can play a role in simplifying the optimization method of the distributed cluster.

On the contrary, in specific implementation, there are also Linux whose nodes constituting the distributed cluster are mounted cluster file systems. In this case, based on the above-mentioned embodiment, as a preferred implementation mode, step S10 is specifically:

Create N target directories under the Linux local system;

Unmount the cluster file system, and create N metadata pools and at least N data pools.

In the distributed cluster optimization method provided in this embodiment, even if the cluster file system has been mounted on the Linux local system, N target directories can be created directly under the cluster file system, and then the cluster file system is uninstalled, and a new N metadata pools and at least N data pools complete the optimization of distributed clusters.

For the situation that the Linux local system has mounted the cluster file system, based on the above-mentioned embodiment, as another preferred implementation manner, before step S10, it also includes:

Unmount the cluster file system.

In the optimization method of the distributed cluster provided in this embodiment, even if the cluster file system has been mounted on the Linux local system, the cluster file system can be unmounted directly first, in the case of ensuring that the Linux local system is no longer mounted with the cluster file system Next, step S10 is executed again to complete the optimization of the distributed cluster.

In order to make the processing capabilities of each file system similar, based on the above embodiment, as a preferred implementation manner, the allocation of metadata service clusters for each metadata pool is specifically as follows:

Evenly distribute the active metadata service and backup metadata service in the metadata service cluster to each metadata pool.

In the distributed cluster optimization method provided in this embodiment, the activation metadata service and backup metadata service in the metadata service cluster are evenly distributed to each metadata pool, so that the processing capabilities of each file system can be similar. Of course, in practical applications, other allocation methods can be used to allocate the activation metadata service and backup metadata service in the metadata service cluster to each metadata pool according to the actual application scenario, for example, according to the ratio of 1:3 The activated metadata service and the backup metadata service in the metadata service cluster are allocated to each metadata pool, which is not limited in the present invention.

In order for this optimization method to better meet the individual needs of users, based on the above-mentioned embodiments, as a preferred implementation manner, before step S10, it also includes:

Obtain the concurrency level of the client, and determine the value of N according to the concurrency level.

In the distributed cluster optimization method provided in this embodiment, the concurrency degree of the client can be obtained, and the value of N can be determined according to the concurrency degree, so that the concurrency degree of the client is positively correlated with the value of N. That is, for the case where the client concurrency is relatively high on average, a larger N value can be determined to form more file systems and provide external services at the same time; and for the case where the client concurrency is low on average, a larger N value can be determined. With a smaller value of N, fewer file systems can be formed to meet user needs. It can be seen that the optimization method for distributed clusters provided by this embodiment can better meet the personalized needs of users.

The above describes in detail the embodiment of a distributed cluster optimization method provided by the present invention, and the present invention also provides a distributed cluster optimization device corresponding to the distributed cluster optimization method, because the device part The embodiment of the method and the embodiment of the method part refer to each other, so for the embodiment of the device part, please refer to the description of the embodiment of the method part, and details will not be repeated here.

FIG. 3 is a schematic composition diagram of a distributed cluster optimization device provided by an embodiment of the present invention. As shown in Figure 3, the optimization device for distributed clusters provided by this embodiment includes:

Create a module 30, for creating N target directories, N metadata pools and at least N data pools;

A binding allocation module 31, configured to bind the metadata pool and the data pool correspondingly, and allocate metadata service clusters for each metadata pool to form N file systems;

Mounting module 32, used for mounting N file systems with each target directory as a mount point;

Wherein, N is a positive integer greater than 1; one file system corresponds to one metadata pool.

The distributed cluster optimization device provided in this embodiment works collaboratively with the creation module, the binding allocation module, and the mount module, and finally creates N target directories, N metadata pools, and at least N data pools to store metadata Pools and data pools are bound correspondingly, and metadata service clusters are assigned to each metadata pool, forming N file systems with each target directory as a mount point and one-to-one correspondence with metadata pools. And because N is a positive integer greater than 1, when a processing request is received, a unified service can be provided externally through multiple file systems, which can effectively alleviate the queuing situation of processing requests, so that each processing request can be responded as soon as possible. Improve user access efficiency.

Based on the above embodiments, as a preferred implementation manner, the creation module 30 is specifically used for:

Create N target directories, N metadata pools and at least N data pools under the Linux local system;

Among them, the Linux local system does not mount the cluster file system.

Based on the above embodiments, as a preferred implementation manner, it also includes:

The unloading module is used for unloading the mounted cluster file of the Linux local system before triggering the creation module 30 .

The above has described in detail the embodiment of a distributed cluster optimization method provided by the present invention. The present invention also provides a distributed cluster optimization system corresponding to the distributed cluster optimization method. Since the system part The embodiment of the method and the embodiment of the method part refer to each other, so for the embodiment of the system part, please refer to the description of the embodiment of the method part, and details will not be repeated here.

FIG. 4 is a schematic structural diagram of a distributed cluster optimization system provided by an embodiment of the present invention. As shown in Figure 4, the optimization system for distributed clusters provided by this embodiment includes:

Memory 40, used to store the optimization program;

The processor 41 is configured to implement the steps of the method for optimizing a distributed cluster as provided in any one of the above embodiments when executing the optimization program.

The distributed cluster optimization system provided by this embodiment can realize the steps of the distributed cluster optimization method provided by any of the above-mentioned embodiments by calling the optimization program stored in the memory through the processor, so this optimization system has the same distribution as above. The same practical effect as the optimization method of the type cluster.

The present invention also provides a computer-readable storage medium. An optimization program is stored on the computer-readable storage medium. When the optimization program is executed by a processor, the steps of the optimization method for a distributed cluster as provided in any of the above embodiments are implemented. .

The computer-readable storage medium provided in this embodiment stores an optimization program. Since the optimization program is executed by a processor, the steps of the method for optimizing a distributed cluster as provided in any of the above-mentioned embodiments can be implemented, so the computer-readable storage medium has The same practical effect as the optimization method of the above-mentioned distributed cluster.

The optimization method, device, system and readable storage medium of the distributed cluster provided by the present invention have been introduced in detail above. Each embodiment in the description is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other.

It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

It should also be noted that in this specification, relative terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations Any such actual relationship or order exists between. Furthermore, the terms "comprises", "comprises" or any variations thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also includes elements not expressly listed. other elements of, or also elements inherent in, such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Claims (10)

1. a kind of optimization method of distributed type assemblies, which is characterized in that including：

Create N number of target directory, N number of metadata pond and at least N number of data pool；

The metadata pond and data pool correspondence are bound, and distribute Metadata Service collection for each metadata pond Group is to form N number of file system；

Using each target directory as the N number of file system of mount point carry；

Wherein, N is the positive integer greater than 1；One file system corresponds to a metadata pond.

2. the optimization method of distributed type assemblies according to claim 1, which is characterized in that the N number of target directory of the creation, N number of metadata pond and at least N number of data pool are specially：

N number of target directory, N number of metadata pond and at least N number of data pool are created under Linux local system；

Wherein, the non-carry cluster file system of the Linux local system.

3. the optimization method of distributed type assemblies according to claim 1, which is characterized in that Linux local system carry Cluster file system, N number of target directory, N number of metadata pond and at least N number of data pool of creating are specially：

N number of target directory is created under the Linux local system；

The cluster file system is unloaded, and creates N number of metadata pond and at least N number of data pool.

4. the optimization method of distributed type assemblies according to claim 1, which is characterized in that Linux local system carry Cluster file system further includes before the N number of target directory of the creation, N number of metadata pond and at least N number of data pool：

Unload the cluster file system.

5. the optimization method of distributed type assemblies according to claim 4, which is characterized in that in the N number of target mesh of creation Before record, N number of metadata pond and at least N number of data pool, further include：

The degree of concurrence of client is obtained, and determines the value of N according to the degree of concurrence；

Wherein, the size of the value of the height and N of the degree of concurrence is positively correlated.

6. a kind of optimization device of distributed type assemblies, which is characterized in that including：

Creation module, for creating N number of target directory, N number of metadata pond and at least N number of data pool；

Distribution module is bound, for binding the metadata pond and data pool correspondence, and is each metadata Metadata Service cluster is distributed to form N number of file system in pond；

Carry module, for using each target directory as the N number of file system of mount point carry；

Wherein, N is the positive integer greater than 1；One file system corresponds to a metadata pond.

7. the optimization device of distributed type assemblies according to claim 6, which is characterized in that the creation module is specifically used In：

N number of target directory, N number of metadata pond and at least N number of data pool are created under Linux local system；

Wherein, the non-carry cluster file system of the Linux local system.

8. the optimization device of distributed type assemblies according to claim 6, which is characterized in that further include：

Unload module, for before triggering the creation module, the group document system of unloading Linux local system carry System.

9. a kind of optimization system of distributed type assemblies, which is characterized in that including：

Memory is used for storage optimization program；

Processor, for realizing distributed type assemblies as described in any one in claim 1-5 when executing the optimization program The step of optimization method.

10. a kind of computer readable storage medium, which is characterized in that be stored with optimization journey on the computer readable storage medium Sequence, the optimization program realize the optimization side of distributed type assemblies as described in any one in claim 1-5 when being executed by processor The step of method.