CN104156216B - A kind of memory management system and method towards cloud computing - Google Patents

Abstract

The present invention provides a cloud computing-oriented heterogeneous storage management architecture system and method, the system includes a physical storage cluster module, a driver adaptation layer module, a data access scheduling module, a unified storage interface module and a client; the system and method By encapsulating different storage types in the back-end physical storage cluster module, a unified interface is provided for the front-end. For different drivers provided by different block device service vendors, the interface is unified. Using this method can greatly improve the scalability of the cloud service platform, realize the high cohesion and low coupling of the cloud service platform architecture, realize the flexible scalability in the cloud computing environment, and meet the construction of enterprise private cloud and public cloud .

Description

A cloud computing-oriented heterogeneous storage management system and method

technical field

The invention relates to computer application technology, in particular to a cloud computing-oriented heterogeneous storage management system and method.

Background technique

With the rise of cloud computing, the computer field is undergoing profound changes. my country's cloud computing service market is in its infancy, and cloud computing technology and equipment already have a certain foundation for development. The overall scale of my country's cloud computing service market is small, but the momentum of catching up is obvious. According to Gartner estimates, in 2011, my country accounted for less than 3% of the global cloud computing service market of about 90 billion US dollars, but the annual growth rate reached 40%. It is expected that the gap between my country and foreign countries in cloud computing will gradually narrow in the future.

To sum up, cloud computing has the following characteristics:

(1) Ultra-large scale: "Cloud" has a considerable scale. Google Cloud Computing already has more than 1 million servers, and Amazon, IBM, Microsoft, Yahoo and other "clouds" all have hundreds of thousands of servers. Enterprise private clouds generally have hundreds or thousands of servers. "Cloud" can give users unprecedented computing power.

(2) Virtualization: Cloud computing supports users to obtain application services at any location and using various terminals. The requested resource comes from the "cloud" rather than a fixed tangible entity. The application runs somewhere in the "cloud," but the user doesn't really need to know or worry about where the application is running. All we need is a laptop or a mobile phone, and everything we need can be realized through network services, even tasks such as supercomputing.

(3) High reliability: "Cloud" uses measures such as multi-copy data fault tolerance, isomorphic interchangeability of computing nodes and other measures to ensure high reliability of services. Using cloud computing is more reliable than using local computers.

(4) Versatility: Cloud computing is not aimed at specific applications, and it can construct ever-changing applications under the support of "cloud", and the same "cloud" can support the operation of different applications at the same time.

(5) High scalability: The scale of the "cloud" can be dynamically expanded to meet the needs of applications and user scale growth.

(6) On-demand service: "Cloud" is a huge resource pool that you can purchase on demand; cloud can be billed like tap water, electricity, and gas.

(7) Extremely cheap: Due to the special fault-tolerant measures of the "cloud", extremely cheap nodes can be used to form the cloud. Compared with the traditional system, the utilization rate of resources is greatly improved, so users can fully enjoy the low-cost advantages of "cloud". It usually only costs a few hundred dollars and can be completed in a few days. Previously, it took tens of thousands of dollars and several months to complete. task.

With the development of cloud computing, cloud management platforms launched at home and abroad have achieved rapid development in recent years in terms of resource management, data processing, operation monitoring and interactive processing. In terms of interactive processing of cloud management platforms, although the research on cloud management at home and abroad is still in its infancy and has not yet formed a systematic solution, major cloud computing vendors and open source communities have made intelligent cloud management technologies from different perspectives. active exploration.

Drawing on the existing relevant research results at home and abroad, combined with the advantages of the current cloud management platform, the research and development of an all-round, cross-platform, intelligent, large-scale cloud management platform will have great economic and social value. A cloud management platform that can support intelligent management and scheduling of large-scale cloud resources, intelligent management and processing of large-scale massive cloud data, intelligent monitoring and security of cloud operations, and intelligent interactive processing of cloud services is one of the main trends in the development of cloud computing.

In the cloud management platform, in the face of extremely elastic storage requirements and performance requirements, computing nodes need to be able to dynamically add virtual volumes. The virtual machine is mainly realized through the block device storage supported by the cloud management platform. There are different types of block devices, and the back-end storage can be DAS (Direct Attached Storage), NAS (NetworkAttached Storage), SAN (Storage Area Network), object storage or distributed file system. At this time, the backend of the cloud management platform only supports a single storage type, which is increasingly unable to meet the needs of enterprises.

Contents of the invention

In order to solve the above problems, the present invention proposes a cloud computing-oriented heterogeneous storage management architecture, so that the back end of the cloud management platform supports a variety of different storage types, greatly improving the scalability of the cloud service platform, and realizing the cloud service platform The high cohesion and low coupling of the architecture realizes the flexible scalability in the cloud computing environment and meets the needs of enterprise private cloud and public cloud construction.

In order to achieve the above purpose, the present invention proposes a cloud computing-oriented heterogeneous storage management system. The system includes: a physical storage cluster module, a driver adaptation layer module, a data access scheduling module, and a unified storage interface module.

The physical storage cluster module includes different types of storage systems, and supports the application network iSCSI constructed by the storage area network and the communication subsystem and the network file system NFS protocol to provide external storage space.

The driver adaptation layer module unifies the interface access of different storage type protocols in the form of driver adaptation; the driver adaptation layer provides the iSCSI and NFS protocols in the physical storage cluster module respectively The type of storage system performs interface adaptation access.

The data access scheduling module uses a message queue to schedule access to the accessed data input/output I/O requests; processes the tasks of the message queue, and selects an appropriate driver in the driver adaptation layer module according to a predetermined policy Adapting nodes to perform tasks, the data access scheduling module selects an active node with the least number of volumes to create volumes.

The unified storage interface module provides a standard expressive state transfer Rest application programming interface API access mode, is responsible for receiving and processing Rest requests, and puts the requests into the message queue of the data access scheduling module .

Preferably, the different types of storage systems in the physical storage cluster module include: direct attached storage DAS, network attached storage NAS, application network IP SAN constructed by storage area network and communication subsystem, distributed storage, object storage .

Preferably, the unified service for accessing interfaces of different storage type protocols by way of driver adaptation runs on storage nodes to manage storage space; there is one driver adaptation module on each storage node, and multiple The storage nodes are combined to form a storage resource pool.

Preferably, before the interface access of different storage type protocols is unified in the way of driver adaptation, each of the storage nodes generates a storage instance in the form of registration, and a network attached storage NAS drive device can add multiple storage instances. The NAS physical storage devices, each of the NAS physical storage devices will correspond to a storage instance after registration, and the information of the storage instance is recorded in the data access scheduling module.

Preferably, the data access scheduling module uses the message cluster queue to perform access scheduling on the accessed data I/O request, which further includes: recording the storage information of the registered storage instance of the driver adaptation layer module, and according to the application in the request content Select different storage instances for volume size and speed to send volume creation commands; the storage information of the storage instance includes the type of storage, the protocol IP address of the interconnection between the networks, the size of used and allocated storage space, and the size of storage speed .

The present invention also proposes a cloud computing-oriented heterogeneous storage management method, the method comprising:

The unified storage interface module receives and processes the Rest request in a standard Rest API access manner, and puts the request into the message queue of the data access scheduling module.

The data access scheduling module uses a message cluster queue to perform access scheduling on the accessed data IO request; processes the tasks of the message queue, and selects a suitable driver adaptation node in the driver adaptation layer module according to a predetermined policy to perform the task, The data access scheduling module selects an active node with the least number of volumes to create volumes.

The driver adaptation layer module unifies the interface access of different storage type protocols in the form of driver adaptation; the driver adaptation layer performs interface adaptation for different storage types that provide iSCSI and NFS protocols in the physical storage cluster module. with access.

The physical storage cluster includes different types of storage systems, and supports the application network iSCSI and network file system NFS protocol to provide external storage space through the storage area network and communication subsystem.

Preferably, the different types of storage systems in the physical storage cluster module include: DAS, NAS, IPSAN, distributed storage, and object storage.

Preferably, the unified service for accessing interfaces of different storage type protocols by way of driver adaptation runs on storage nodes to manage storage space; there is one driver adaptation module on each storage node, and multiple The storage nodes are combined to form a storage resource pool.

Preferably, before the interface access of different storage type protocols is unified in the way of driver adaptation, each of the storage nodes generates a storage instance in a registered manner, and a NAS drive device can add multiple NAS physical storage devices , each of the NAS physical storage devices registered corresponds to a storage instance, and the information of the storage instance is recorded in the data access scheduling module.

Preferably, the data access scheduling module uses the message cluster queue to perform access scheduling on the accessed data IO request, and further includes: recording the storage information of the registered storage instance of the driver adaptation layer module, and according to the requested volume size in the request content 1. Select a storage instance with a different speed and issue a command to create a volume; the storage information of the storage instance includes storage type, IP address, used and allocated storage space, and storage speed.

Compared with the prior art, the present invention provides a cloud computing-oriented heterogeneous storage management architecture system and method, the system includes a physical storage cluster module, a driver adaptation layer module, a data access scheduling module, a unified storage interface module and Client; the system and method provide a unified interface to the front end by encapsulating different storage types in the back-end physical storage cluster module. The backend storage type can be DAS, NAS, SAN, object storage or distributed file system. For different drivers provided by different block device service vendors, the interface is unified. Using this method can greatly improve the scalability of the cloud service platform, realize the high cohesion and low coupling of the cloud service platform architecture, realize the flexible scalability in the cloud computing environment, and meet the construction of enterprise private cloud and public cloud .

Description of drawings

The accompanying drawings in the embodiments of the present invention are described below. The accompanying drawings in the embodiments are used for further understanding of the present invention and are used together with the description to explain the present invention, and do not constitute a limitation to the protection scope of the present invention.

Figure 1 is a block diagram of cloud computing-oriented heterogeneous storage management architecture;

Figure 2 is a detailed structural diagram of cloud computing-oriented heterogeneous storage management architecture;

FIG. 3 is a flow chart of an embodiment of a process for a client to apply for a virtual volume.

detailed description

In order to facilitate the understanding of those skilled in the art, the present invention will be further described below in conjunction with the accompanying drawings, which cannot be used to limit the protection scope of the present invention.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

The implementation of the present invention will be described in detail below in conjunction with the accompanying drawings and examples, so as to fully understand and implement the process of how to apply technical means to solve technical problems and achieve technical effects in the present invention. The embodiments of the present application and the combinations of the various features in the embodiments without conflict are within the protection scope of the present invention.

First, as shown in Figure 1, the present invention proposes a cloud computing-oriented heterogeneous storage management system 01, which includes: a physical storage cluster module 05, a driver adaptation layer module 04, a data access scheduling module 03 and a unified Storage interface module 02.

The physical storage cluster module 05 includes different types of storage systems, and supports the application network iSCSI and network file system NFS protocol to provide external storage space through the storage area network and communication subsystem.

Preferably, the different types of storage systems in the physical storage cluster module 05 include: DAS (Direct Attached Storage), NAS (Network Attached Storage), IP SAN (storage area network and communication sub system-built application network), distributed storage, and object storage.

The driver adaptation layer module 04 unifies the interface access of different storage type protocols in the way of driver adaptation. The driver adaptation layer performs interface adaptation and access for the different types of storage systems in the physical storage cluster module that provide iSCSI and NFS protocols.

Preferably, the unified service for accessing interfaces of different storage type protocols in a driver-adapted manner runs on storage nodes to manage storage space; there is one driver adaptation module 04 on each of the storage nodes, A plurality of the storage nodes jointly form a storage resource pool.

Preferably, before the interface access of different storage type protocols is unified in the way of driver adaptation, each of the storage nodes generates a storage instance in a registered manner, and a NAS drive device can add multiple NAS physical storage devices After registration, each NAS physical storage device will correspond to a storage instance, and the information of the storage instance is recorded in the data access scheduling module 03.

The data access scheduling module 03 uses a message queue to schedule access to the accessed data input/output I/O request; processes the task of the message queue, and selects the appropriate one in the driver adaptation layer module 04 according to a predetermined policy The driver adaptation node is used to execute the task, and the data access scheduling module 03 selects an active node with the least number of volumes to create the volume. Preferably, the data access scheduling module 03 uses the message cluster queue to perform access scheduling on the accessed data I/O request, and further includes: recording the storage information of the registered storage instance of the driver adaptation layer module 04, according to the content of the request Choose a storage instance with a different size and speed for the requested volume and issue the command to create the volume.

Preferably, the storage information of the storage instance includes storage type, IP address, used and allocated storage space, and storage speed.

The unified storage interface module 02 provides a standard expressive state transfer Rest application programming interface API access mode, is responsible for receiving and processing Rest requests, and puts the requests into the message queue of the data access scheduling module 03 .

The client provides the management functions of locally creating volumes, querying volumes, deleting volumes, and unmounting volumes, and performs data transmission with the unified storage interface module 02 at the back end through the RestAPI access method.

The present invention also provides a cloud computing-oriented heterogeneous storage management method, the method mainly includes the following steps:

Step S100 , the client provides the management functions of locally creating volumes, querying volumes, deleting volumes, and unmounting volumes, and performs data transmission with the back-end unified storage interface module 02 through the Rest API access method.

Step S101 , the unified storage interface module 02 receives and processes the Rest request in a standard Rest API access mode, and puts the request into the message queue of the data access scheduling module 03 .

Step S102, the data access scheduling module 03 uses the message cluster queue to perform access scheduling on the accessed data IO requests.

Preferably, considering the storage scale, the message queue is generally a relatively large cluster message queue.

Preferably, the data access scheduling module 03 uses the message cluster queue to perform access scheduling on the accessed data I/O request, including: processing tasks in the task queue, and selecting an appropriate driver adaptation node to perform the task according to a predetermined strategy, the The data access scheduling module 03 selects an active node with the least number of volumes to create volumes.

Preferably, the data access scheduling module 03 uses the message cluster queue to perform access scheduling on the accessed data I/O request, and further includes: recording the storage information of the storage instance of the registered driver adaptation layer module 04, according to the application in the request content Choose a storage instance with a different volume size and speed to issue a command to create a volume.

Preferably, the storage information of the storage instance includes storage type, IP address, used and allocated storage space, and storage speed.

Step S103, the driver adaptation layer module 04 unifies the interface access of different storage type protocols in the form of driver adaptation; the driver adaptation layer performs interface adaptation access for different storage types that provide iSCSI and NFS protocols respectively .

Preferably, the different types of storage systems in the physical storage cluster module 05 include: DAS, NAS, IP SAN, distributed storage, and object storage.

Preferably, the unified service for accessing interfaces of different storage type protocols by way of driver adaptation runs on storage nodes to manage storage space; there is one driver adaptation module on each storage node, and multiple The storage nodes are combined to form a storage resource pool.

Preferably, before the interface access of different storage type protocols is unified in the way of driver adaptation, each of the storage nodes generates a storage instance in a registered manner, and a NAS drive device can add multiple NAS physical storage devices , each of the NAS physical storage devices registered corresponds to a storage instance, and the information of the storage instance is recorded in the data access scheduling module.

Step S104 , the physical storage cluster module 05 includes different types of storage systems, and supports the application network iSCSI and network file system NFS protocol to provide external storage space through the storage area network and communication subsystem.

The cloud computing-oriented heterogeneous storage management architecture system and method provided by the present invention provide a unified interface for the front end by encapsulating different storage at the back end. The back-end storage can be DAS, NAS, SAN, object storage or distributed file system. Different block device service vendors realize interface unification by providing different drivers. Using this method can greatly improve the scalability of the cloud service platform, realize the high cohesion and low coupling of the cloud service platform architecture, realize the flexible scalability in the cloud computing environment, and meet the construction of enterprise private cloud and public cloud .

An embodiment of the present invention is introduced below, the process of a user applying for a virtual volume through a client.

Fig. 3 is a flowchart according to one embodiment of the present invention.

As shown in Fig. 3, the cluster server deployment method includes the following steps.

In step S110, the client applies for creating a virtual volume, and the client initiates a request, the content of which includes the size of the virtual volume, the speed level, and the IP of the client.

Step S111, sending the request to the unified storage interface in the form of the Rest interface.

Step S112, the data access scheduling module selects the appropriate driver layer according to the request, and the data scheduling module records the storage information of the registered driver adaptation layer storage instance, which includes the storage type, IP address, used and allocated storage space size, storage speed size. The data access scheduling module selects different storage instances according to the volume size and speed applied for by the requested content and issues the command to create a volume.

In step S113, the driver adaptation layer invokes protocols such as iSISC to notify the physical storage cluster to create a volume. Here, the corresponding storage command is researched according to the selected driver configuration layer.

In step S114, the physical storage cluster creates a real volume and returns the result, which is mainly realized through commands of different physical storages.

In step S115, the data access scheduling layer drives the adaptation layer to return a result and record information, mainly recording the corresponding relationship between the physical volume and the virtual volume, which is used for mapping during data access.

In step S116, the unified storage interface returns the created volume information to the client.

Step S117, sending the request to the unified storage interface in the form of the Rest interface.

Additionally, the steps shown in the flowchart of FIG. 3 may be performed on a computer system, such as a set of computer-executable instructions. Also, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be implemented using a computer program flow, the computer program can be stored in a computer-readable storage medium, and the computer program can be run on a corresponding hardware platform (such as system, device, device, device, etc.), and when executed, includes one or a combination of the steps of the method embodiment.

Optionally, all or part of the steps in the above embodiments can also be implemented using integrated circuits, and these steps can be fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module accomplish. As such, the present invention is not limited to any specific combination of hardware and software.

The devices/functional modules/functional units in the above embodiments can be realized by general-purpose computing devices, and they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices.

When each device/functional module/functional unit in the above-mentioned embodiments is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like.

It should be noted that the above-described embodiments are only for the convenience of those skilled in the art to understand, and are not intended to limit the protection scope of the present invention. Any obvious replacements and improvements made by the invention are within the protection scope of the present invention.

Claims (8)

1. A cloud computing-oriented heterogeneous storage management system, characterized in that the system comprises: a physical storage cluster module, a driver adaptation layer module, a data access scheduling module, and a unified storage interface module; The physical storage cluster module includes different types of storage systems, and supports the application network iSCSI constructed by the storage area network and the communication subsystem and the network file system NFS protocol to provide external storage space; The driver adaptation layer module unifies the interface access of different storage type protocols in the form of driver adaptation; the driver adaptation layer provides the iSCSI and NFS protocols in the physical storage cluster module respectively Adaptive access to the interface for the type of storage system; The data access scheduling module uses a message queue to schedule access to the accessed data input/output I/O requests; processes the tasks of the message queue, and selects an appropriate driver in the driver adaptation layer module according to a predetermined policy Adapting nodes to perform tasks, the data access scheduling module selects an active node with the least number of volumes to create volumes; The unified storage interface module provides a standard expressive state transfer Rest application programming interface API access mode, is responsible for receiving and processing Rest requests, and puts the requests into the message queue of the data access scheduling module ; The data access scheduling module uses a message queue to schedule access to the accessed data I/O request and further includes: recording the storage information of the registered storage instance of the driver adaptation layer module, according to the volume size and speed applied for in the request content Select a different storage instance to issue a command to create a volume; the storage information of the storage instance includes the type of storage, the protocol IP address of the interconnection between the networks, the size of the used and allocated storage space, and the size of the storage speed. 2. The system according to claim 1, wherein the different types of storage systems in the physical storage cluster module include: direct attached storage DAS, network attached storage NAS, storage area network and communication subsystem construction The application network IP SAN, distributed storage, object storage. 3. The system according to claim 1, wherein the unified service for accessing interfaces of different storage type protocols in the form of driver adaptation runs on storage nodes to manage storage space; each of the storage There is one driver adaptation layer module on a node, and multiple storage nodes are combined to form a storage resource pool. 4. The system according to claim 3, wherein before the interface access of different storage type protocols is unified by means of driver adaptation, each of the storage nodes generates a storage instance by means of registration, A network-attached storage (NAS) drive device adds multiple NAS physical storage devices, and each NAS physical storage device corresponds to a storage instance after registration, and the information of the storage instance is recorded in the data access scheduling module. 5. A cloud computing-oriented heterogeneous storage management method, characterized in that the method comprises: The unified storage interface module receives and processes the Rest request in a standard Rest API access mode, and puts the request into the message queue of the data access scheduling module; The data access scheduling module uses a message queue to schedule access to the accessed data IO request; processes the task of the message queue, and selects a suitable driver adaptation node in the driver adaptation layer module according to a predetermined policy to perform the task, so The data access scheduling module selects an active node with the least number of volumes to create volumes; The driver adaptation layer module unifies the interface access of different storage type protocols in the form of driver adaptation; the driver adaptation layer performs interface adaptation for different storage types that provide iSCSI and NFS protocols in the physical storage cluster module. with access; The physical storage cluster includes different types of storage systems, and supports the application network iSCSI constructed by the storage area network and the communication subsystem and the network file system NFS protocol to provide external storage space; The data access scheduling module uses a message queue to schedule access to the accessed data IO request and also includes: recording the storage information of the registered storage instance of the driver adaptation layer module, and selecting different volumes according to the volume size and speed applied for in the request content. The storage instance issues a command to create a volume; the storage information of the storage instance includes storage type, IP address, used and allocated storage space, and storage speed. 6. The method according to claim 5, wherein the different types of storage systems in the physical storage cluster module include: DAS, NAS, IP SAN, distributed storage, and object storage. 7. The method according to claim 5, wherein the unified service for accessing interfaces of different storage type protocols in a driver-adapted manner runs on storage nodes and manages storage space; each of the storage There is one driver adaptation layer module on a node, and multiple storage nodes are combined to form a storage resource pool. 8. The method according to claim 7, characterized in that, before the interface access of different storage type protocols is unified by means of driver adaptation, each of the storage nodes generates a storage instance by means of registration, One NAS drive device adds multiple NAS physical storage devices, and each NAS physical storage device corresponds to a storage instance after registration, and the information of the storage instance is recorded in the data access scheduling module.