CN102652423B - Method and system for cluster selection and cooperative replication - Google Patents

Abstract

Disclosed are apparatus, systems, and methods for creating cluster families for cluster selection and collaborative replication. Clusters are grouped into family members of a cluster family based on their relationships and roles. Members of a cluster family determine which family member is best positioned to obtain replication information and become cumulatively consistent within their cluster family. Once the cluster family becomes cumulatively consistent, data is shared within the cluster family so that all copies within the cluster family are consistent.

Description

Method and system for cluster selection and cooperative replication

technical field

The present invention relates to data storage in relation to data storage systems, and more particularly to clustering in storage systems.

Background technique

A storage system may include multiple tape devices for accessing multiple tapes using a library manager. The magnetic tape may be arranged in an audio cassette. The controller may instruct the actuator to transfer the tape cartridge from the storage area to the tape drive, thereby accessing data written on the tape and/or writing data to the tape.

Storage systems may be located at multiple sites including multiple geographically distinct sites. Storage systems can communicate over one or more networks. Each storage system can include multiple clusters. Each cluster can include multiple tape drives. Mounts a tape to a tape drive, reading data from the tape and writing data to the tape.

Each tape can be organized into one or more logical volumes, referred to here as volumes. Volumes can appear to hosts as different storage devices. Volumes can be logically "mounted" on virtual tape drives. As used herein, a virtual tape drive is a logical construct that appears to a host as a tape drive.

US 2009/0132657 (Sutani, MR, et al.) discloses data partitioning across clusters in a distributed architecture, where dynamic replication of cache nodes is based on the concept of buddy replication. Buddy replication allows data to be replicated by a limited number of nodes within a cluster and provides reduced network replication traffic.

US 2009/0030986A1 (Bstes, J W) discloses a remote asynchronous data replication process implemented in a replication cluster, which realizes point-to-point data replication. Point-to-point topology allows primary storage at a local site to distribute data to remote sites by replicating bi-directional data between sites on the network.

In a multi-cluster configuration, each cluster is equally independent of all other clusters. Without some classification of relationship awareness, clusters cannot operate in the most efficient manner based on their role and/or distance from other clusters. In typical grid configurations, this lack of awareness of relationships greatly affects the means by which a cluster is selected as a source for a volume during the mount process and the cluster's ability to honor volume replication. For example, a grid may prefer to select a global remote source cluster for installation and/or copy processing over a metro remote cluster. Global remote clusters are much less efficient due to the network distance between clusters. Although real-time latency checks can be introduced to detect such distances, the irregularities and randomness of wide area networks (WANs) make it very difficult to reliably measure relative distances. Further, self-replication across two or more cluster distances in a group can be more efficient if the groups work together and cumulatively replicate data into the group and then each other.

Therefore, there is a need in the prior art to solve the above-mentioned problems.

Contents of the invention

Methods, apparatus and systems are provided for creating a cluster family, selecting a cluster family member or cluster families, and collaborative replication between family members and different families. For example, cluster-based relationships group clusters into family members of cluster families. Members of the cluster family determine which family member is in the best position to obtain and become cumulatively consistent with the external data objects within their cluster family. Once a cluster family becomes cumulatively consistent, data objects are shared within the cluster family such that all clusters within the cluster family have a known copy of each external data object.

Viewed from one aspect, the invention provides a computer program product comprising a computer-usable medium comprising a computer-readable program. The computer readable program, when executed on a computer, causes the computer to: group a plurality of clusters into cluster members; determine which family member is in the best position to obtain external data from a source; select one or more of the cluster clusters family members to obtain data; copy data to a cluster family; achieve cumulative consistency within a cluster family of at least two external data objects; and share data within a cluster family such that all clusters within a cluster family have each external data A consistent copy of the object.

Viewed from another aspect, the invention provides a method for cooperative replication of multiple clusters. The method includes arranging a plurality of clusters into family members of a cluster family; negotiating among the family members which family member is best positioned to obtain data from a source; selecting one or more family members of the cluster family, obtain data; cooperatively copy the data into the cluster family; achieve cumulative consistency within the cluster family; and share data within the cluster family so that all copies of the data within the cluster family are consistent.

Viewed from another aspect, the present invention provides an apparatus for creating cluster families and family members to perform cooperative replication. The apparatus includes a plurality of modules configured to functionally perform the steps of: creating a cluster family and family members; applying collaborative replication; and selecting a cluster family and family members based on cluster relationships. These modules in the described embodiments may include a relationship module, a creation module, a collaborative replication module, an installation processing module, a communication module, and a policy module or any combination thereof.

The relationship module includes a computer readable program that executes on the processor and maintains factors that define roles, rules, and relationships between cluster families and family members. The clusters communicate over a network. Each cluster includes at least one cache, eg, a virtual volume cache.

The creation module includes a computer readable program that executes on the processor and creates a family of clusters for cluster selection and cooperative replication. In a preferred embodiment of the invention, the creation module creates cluster families by grouping clusters into cluster members based on cluster relationships and roles. In an alternative embodiment, the creation module configures clusters into families. In an alternative embodiment, the creation module creates relationships between different clusters and groups clusters into families.

The cooperative replication module includes a computer readable program executing on a processor and cooperatively replicating data across cluster family members in the cluster family and across different cluster families.

The installation processing module executes a computer readable program on a processor and supports family members within a cluster family for production purposes more than other cluster families.

Viewed from another aspect, the present invention provides a system for cooperative replication of multiple clusters. The system includes a network; a plurality of sites in communication over the network, each cluster including at least one host computer and a storage system, the storage system including a plurality of clusters, each cluster including a at least one tape drive, at least one tape volume cache, and a cluster manager configured to execute a computer readable program using a processor and memory, wherein the software readable program includes: a creation module configured to establish a cluster group and A cluster group is arranged into the family members of the cluster family; and a cooperative replication module is configured to select the family members to cooperatively replicate the external data object into the cluster family and achieve cumulative family consistency.

Description of drawings

The invention will be described below, by way of example only, with reference to a preferred embodiment as shown in the accompanying drawings, in which:

Figure 1 is a schematic block diagram illustrating a preferred embodiment of a distributed site according to the present invention;

2A and 2B are schematic block diagrams illustrating a preferred embodiment of a storage system according to the present invention;

Figure 3 is a schematic block diagram illustrating a preferred embodiment of the cluster of the present invention;

Figure 4 is a schematic block diagram illustrating a preferred embodiment of the cluster cluster device of the present invention;

Figure 5 is a schematic flow diagram illustrating a preferred embodiment of the collaborative replication method and cluster family selection of the present invention; and

6A and 6B are schematic flowcharts illustrating a preferred embodiment of the collaborative replication method and cluster family selection of the present invention.

Detailed ways

Reference in this specification to features, advantages or similar language does not imply that all or all of the features and advantages that can be achieved with the invention should be in any single embodiment of the invention. Rather, language referring to the features and advantages is understood to mean that a particular feature, advantage, or characteristic described in connection with a preferred embodiment is included in at least one embodiment of the present invention. Thus, discussions of the features, advantages, and similar language throughout this specification may, but do not necessarily, refer to the same embodiment.

Furthermore, the above-described features, advantages, and characteristics of the invention may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize that the invention can be practiced without one or more of the specific features or advantages of a particular embodiment.

The present invention is described in the following illustrative embodiments with reference to the accompanying drawings, in which like reference numerals designate like or similar elements. While the invention has been described in terms of the best mode for carrying out the objects of the invention, those skilled in the art will appreciate that changes may be made in light of these teachings without departing from the scope of the invention.

Many of the functional units introduced in this specification are marked as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising conventional very large scale integration (VLSI) or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Modules may also be implemented in programmable hardware devices, such as field programmable gate arrays (FPGAs), programmable array logic, programmable logic devices, and the like. Modules may also be implemented in software that is executed by various types of processors. An identified module as an executable module, for example, comprises one or more blocks of physical or logical computer instructions which may, for example, be organized as an object, procedure or function. However, executable identified modules need not be physically located together, but may include disparate instructions stored at different locations which, when logically linked together, comprise the modules and implement the functions for the modules. stated purpose.

Indeed, a module of an executable node may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, actionable data can be identified and described in modules here. The operational data collection can be as a single collection of data, or can be distributed across different locations including different storage devices.

Reference in this specification to "the preferred embodiment," "preferred embodiment" or similar language means that a particular feature, structure, or characteristic described in connection with this embodiment is included in at least one preferred embodiment of the invention. Thus, appearances of the phrases "in a preferred embodiment," "in a preferred embodiment" and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

Furthermore, the described features, structures or characteristics of the invention may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide embodiments of the invention. a thorough understanding. One skilled in the art will recognize, however, that the invention may be practiced without one or more of the specific details, or other methods, components, materials, or the like. In other instances, well-known structures, materials, or operations are not described or shown in detail to avoid obscuring aspects of the invention.

Fig. 1 is a schematic block diagram illustrating a preferred embodiment of a distributed site 100 according to the present invention. Distributed site 100 includes multiple sites 105 . Each site 105 may communicate with other sites 105 over network 110 . Network 110 may be the Internet, a local area network (LAN), a wide area network (WAN), a private network, a combination of networks, or the like.

Each site 105 may include one or more storage systems, as will be described hereinafter. Additionally, each site 105 may include bridges, routers, etc. to connect the storage systems to the network 110 .

2A and 2B are schematic block diagrams illustrating a preferred embodiment of a storage system 200 according to the present invention. One or more storage systems 200 may be embodied in each site 105 of FIG. 1 .

Storage system 200 may store data on various physical media including, but not limited to, storage cartridges, disk drives, solid state disks (SSDs), disk direct access storage devices (DASDs), tape drives, libraries, and disk drive arrays, such as Redundant array or cluster of independent disks (RAID). Examples of storage cartridges are cassette tapes, which include rewritable magnetic tape wound on pivoted reels, and cassette memory. One example of a cassette includes a cassette based on Linear Tape Open (LTO) technology. Linear Tape Open LTO and the LTO logo are trademarks of HP, IBM Corporation and Quantum in the US or other countries.

Storage system 200 may store data in different forms, such as logical or virtual data. Here, data may be organized in any of various forms, referred to as "volumes" or "objects", data selected without reference to any particular size or arrangement of the data.

As shown in FIGS. 2A and 2b , a storage system 200 provides storage for a plurality of host systems 210 . For example, the storage system 200 includes multiple hosts 210 , multiple clusters 220 , and a network 215 . Although two (2) hosts 210a, 210b, four (4) clusters 220a, 220b, 220c, 220d, and one (1) network 215 are shown in FIG. 2A for purposes of simplicity, any number of hosts may be used 210 , cluster 220 and network 215 . Therefore, any number of clusters 220 may be included in the storage system 200 .

As shown in FIG. 2A, the storage system 200 may utilize four (4) clusters 220a, 220b, 220c, 220d connected by a network 215, each cluster 220 including a virtual node for emulating a tape drive or tape library for the host 210a (“VN”) 260 and storage device 230 . In a preferred embodiment, the clusters 220a, 220b, 220c, 220d are virtual tape server clusters.

Each cluster 220 includes hierarchical storage nodes (“HSNs”) 250 for locally moving and/or passing data between storage devices 230 and libraries 240 . In a preferred embodiment, the storage system 200 includes a disk storage 230 and a disk library 240 . In a preferred embodiment, library 240 is an automated disk library ("ATL"). HSN 250 may be used to remotely transfer data between local disk storage 230 and remote disk storage 230. For example, disk storage 230 may include one or more disk drives arranged in RAID, JBOD, SSD, or any combination thereof.

Each cluster 220 includes a library manager 370 with tapes as shown in FIG. 3 and described below. Host 210 may initiate or run a task or job, such as a tape job, in which data is read from and written to tape in cluster family 280 and/or family members 220 . Host 210 may be a mainframe computer, a server, or the like. Host 210 may have the capability to run or support multiple operating systems. For example, host 210 may run or may support multiple operating systems, such as wait. Linux is a registered trademark of Linus Toralds in the United States and/or other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries. Each of the hosts 210 of the storage system 200 can function as a single mainframe computer, one or more servers, or multiple virtual machines. Host 210 can provide three levels of virtualization: Logical Partitions (LPARs) through Processor Resource/System Manager (PR/SM) tools; operating system virtual machines; and operating systems, especially those with key-protected address spaces and goal-oriented workload scheduling IBM, z/VM and z/OS are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.

Host 210 may communicate with cluster 220 over network 215 to access multiple tape drives, disk drives, or other storage devices through cluster family members 220 as described below. For example, first host 210a may communicate over network 215 to access storage devices and tapes through first cluster 220a.

Each cluster 220 may include a tiered storage controller, such as tiered storage nodes 315 , as shown in FIG. 3 . The cluster 220 can provide a single point of management for being read and stored, gathers storage tools that can easily allocate storage to different hosts 210, expand the storage system 200 by adding storage or storage control nodes, and for implementing A platform for advanced features such as write caching, point-in-time copy, transparent data migration, and remote copy.

Cluster 220 may follow an "in-band" approach. The in-band approach may result in all input/output (I/O) requests and all management and configuration requests being processed by cluster family members 220 .

Each of clusters 220 may be connected among themselves and may be connected to hosts 220 through network 215 to access data written on and/or write data to tape. Multiple clusters 220 may form domain 205 of storage system 200 . Domain 205 may represent multiple cluster or grid configurations. Domain 205 may include two or more clusters 220 .

The network 215 of the storage system 200 may be a storage area network (SAN), a token ring network, a local area network (LAN), a wide area network (WAN), the Internet, a dedicated network, a combination of networks, and the like. A SAN may comprise a "fabric" through which hosts 210 may communicate with clusters 220 over network 215 . Fabrics can include Fiber Channel networks, Ethernet, and more. All elements cannot share the same fabric for communication. The first host 210a may communicate with the first cluster 220a through one configuration. Additionally, the first host 210a may communicate with the third cluster 220c through another configuration.

Each storage system 200 may include a cluster family 280 . Cluster family 280 may include a plurality of cluster family members 220 arranged, configured, organized, and/or grouped into cluster family 280 . For example, as shown in FIG. 2B, storage system 200 includes cluster family 280(1) and cluster family 280(2). Cluster family 280(1) includes a plurality of clusters 220(a), 220(b) grouped into cluster members of cluster family 280(1). Cluster family 280(2) includes a plurality of cluster family members 220(b), 220(c) grouped into the cluster members of cluster family 280(2). Cluster family 280(1) and cluster family 280(2) communicate with each other via a network (eg, networks 110, 215). Each cluster family 280 may be assigned or assigned a name. For example, cluster family 280(1) may be named City A, and cluster family 280(2) may be named City B.

Although for purposes of simplicity, FIG. 2B shows storage system 200 with two cluster families 280 . Any number of storage systems 200, cluster families 280, and cluster family members 220 may be used.

An example of storage system 200 is an IBM TS7700 virtual tape server.

Figure 3 is a schematic block diagram illustrating a preferred embodiment of a cluster 220 of the present invention. For example, cluster 220 may represent cluster family member 220 of cluster family 280 of FIGS. 2A and 2B . The description of the cluster 220 refers to elements of FIGS. 1-2, like numbers denoting like elements. Cluster 220 may include virtualization nodes 310 , tiered storage nodes 315 , volume cache 365 , and librarian 370 .

For example, storage device 230 may include one or more disk drives arranged as a redundant array of independent disks (RAID) or a bunch of disks (JBOD), or a solid state disk (SSD), or the like. Storage device 230 may include volume storage 365 . Volume cache 365 may function as a virtual volume cache and/or a tape volume cache (TVC).

For example, storage device 230 includes virtual volume cache 365 . Virtual volume cache 365 can be used as a TVC, where a TVC includes a fast access memory device, such as a hard drive. In a preferred embodiment, cluster 220 is used to cache data to TVC 365.

TVC 365 can cache data read from logical volumes and/or cache data to be written to logical volumes. Host 210 may repeatedly write to the logical volume. The TVC 365 can store written data on the hard disk drive 230 without writing the data to tape of the logical volume. At a later time, TVC 365 may write the cached data to tape within tape library 240. Thus, operations such as read operations and write operations for virtual tape drives that mount logical volumes can be routed through the TVC 365.

Host 210 may initiate and run tasks and/or jobs on cluster 220 . For example, first host 210a access may cause the transport of librarian 370, controlled by physical tape manager 335, to transfer the tape cartridge from the storage area to the tape drive to access the data written on the tape and/or transfer the data Write to tape and/or TVC 365.

Virtualization node 310 may be a stand-alone processor-based server with multiple connections to network 215 . Virtualization node 310 may include a battery backup unit (BBU) and/or may have access to an uninterruptible power supply (UPS). Virtualization node 310 may include a watchdog timer. A watchdog timer can ensure that a failed virtualized node 310 that cannot and/or takes a long time to recover can be restarted.

Virtualization node 310 may include one or more tape daemons 312 . The tape daemon 312 can emulate the tape drives of the cluster 220 to the host 210 as virtual tape drives. Tape daemon 312 may operate files on TVC 365, and/or may operate files in a remote TVC 365 of another cluster 220 through remote file access.

Hierarchical storage node 315 may include cluster manager 320 , remote file access 325 , data mover 330 , physical tape manager 335 , cache manager 340 , callback manager 345 , database 350 , management interface 355 , and media manager 360 . Cluster manager 320 may coordinate operations among multiple clusters or multiple clusters 220 in a mesh topology.

Cluster manager 320 may use the token to determine which cluster 220 has the current copy of the data. The tokens may be stored in database 350 . Cluster manager 320 may also coordinate copying data between clusters 220 . Cluster manager 320 may include one or more processors configured to execute computer-readable programs known to those skilled in the art.

Remote file access 325 may be a server, one or more processors, or the like. Remote file access 325 may provide a link to TVC 365 for access by any remote cluster 220. Cluster manager 320 may include a computer readable program.

Data mover 330 may control the actual data transfer operations for the copies performed between clusters 220 and may also transfer data between physical tape media and TVC 365. Data mover 330 may include a computer readable program.

Physical tape manager 335 may control physical tapes in cluster 220 . The physical tape manager 335 can manage physical tapes in multiple pools, revamp, borrow volumes from and return volumes to a common scratch pool, and transfer tapes between pools. Physical tape manager 335 may include a computer readable program.

Cache manager 340 may control the copying of data from TVC 365 to physical tape, and the subsequent removal of redundant copies of data from TVC 365. Cache manager 340 may provide control signals to flow data between the various components and TVC 365. Cache manager 340 may be a computer readable program.

Callback manager 345 may queue and control callbacks of data from physical media to TVC 365 for virtual tape drives or copies requested by cluster manager 320. Callback manager 345 may comprise a computer readable program.

Database 350 may be a structured collection of records stored on a hard drive. A record may include the location of data on tape. Host 210 may use the database address to write data to and/or access data from tape in cluster 220 to provide the data to users.

Management interface 355 may provide information related to cluster 220 to users. Likewise, management interface 355 may allow a user to control and configure cluster 220 . Management interface 355 may include a computer cathode ray tube (CRT), liquid crystal display (LCD) screen, keyboard, etc., or exist as a web-based interface.

Media manager 360 may manage the physical handling of tapes of cluster 220 . Likewise, media manager 360 may manage error recovery for tapes of cluster 220 . Media manager 360 can diagnose the error and can determine whether the error is caused by the physical tape drive or by the physical tape media. Additionally, media manager 360 can take appropriate actions for error recovery.

Librarian 370 may include multiple physical tape drives, robotic accessors, and multiple physical tape media. The robotic accessor of the librarian 370 can deliver the tapes to the tape drives assigned to the TVC 365. A virtual tape drive may be a logical structure that appears to host 210 as a physical tape drive. As known to those skilled in the art, data may be read from or written to the tape of the tape drive through the read/write channel.

Each tape drive in number of clusters 220 may use one or more tapes to store data. Magnetic tape may be used as a storage medium for storing data in system 200 . Cluster 220 may use any number of tape drives and tapes. For example, storage system 200 may use two (2) tape drives and two hundred and fifty six (256) virtual drives.

TVC 365 can contain data from tape volumes being manipulated and can store additional volume data for fast access. Operations such as read operations and write operations of the mounted volume's virtual tape drive can be routed through the TVC 365. Thus, select cluster 220 may select a TVC 365 of the cluster. All tapes of a tape drive can be organized into one or more logical volumes. Volumes in the TVC 365 may be managed using first-in-first-out (FIFO) and/or most recently used (LRU) algorithms.

TVC 365 may be a fast access memory device. For example, the TVC 365 may be a hard drive or the like with a memory capacity of fifty-four hundred gigabytes (5400GB). In storage system 200, tape drives may cache data read from logical volumes to TVC 365 and/or may cache data to be written to logical volumes. For example, host 210 may repeatedly write to a virtual tape drive. TVC 365 can store written data on the hard drive without writing the data to virtual tape. At a later time, cache manager 340 may write the cached data to cluster 220 tape.

A virtualization node 310 that accesses a volume may be referred to as a mount point. Selecting a remote cluster TVC 365 for the nearest mount point of a logical volume can improve access to the volume. The high availability, fast write memory of the TVC 365 allows the host 210 to write data to the TVC 365 without having to wait for the data to be written to the physical disk.

In a preferred embodiment, each site 105 includes a storage system 200 . Each storage system 200 includes two or more cluster family members 220 that are grouped together to create a cluster family 280 . For example, cluster family 280(1) includes the set of cluster family members 220(a) and 220(b), and cluster family 280(2) includes the set of cluster family members 220(c) and 220(d). Cluster family 281(1) may be used for production purposes, and cluster family 280(2), for example, may be used for disaster (DR) or archival purposes. Thus, cluster families 280 may fulfill different roles in relation to other cluster families 28 . In addition, cluster family members 220 of cluster family 280 may fulfill different roles within cluster family 280 that are related to each other. Thus, cluster family members 220 of cluster family 280 may fulfill different roles than non-cluster members.

In a preferred embodiment, cluster clusters 280 may be configured on global distances, metropolitan distances, or a combination thereof. Similarly, cluster family members 220 may be configured on global distances, metropolitan distances, or a combination thereof. Furthermore, in cluster family 280, cluster family members 220 may have different distance ratings from each other. Similarly, cluster families 280 may have different distance classes from one another. While distance hierarchy can be used as a factor in defining roles and relationships between cluster family 820 and cluster family members 220 , it is only a factor that brings about a sense of relationship between cluster family members 220 and base station families 280 . Accordingly, arranging or grouping clusters 220 into cluster family members of cluster family 280 is not limited by distance.

Furthermore, since each storage system 200 includes a cluster family 280 created by grouping two or more clusters 220 into family members, each storage system 200 or a combination of storage systems 200 may represent a multi-cluster configuration or grid.

Furthermore, clusters 220 of storage systems 200 may form a distributed storage configuration. For example, the second cluster 220(b) may create a second instance of the volume. The second instance may be synchronized with the first copy on the first cluster 220(a), wherein the second copy is updated at any time the first copy is updated. The second instance may be stored in another cluster family 280 at the remote site 105 to ensure data availability in the event the first instance becomes unavailable. Future mount point accesses may select the second copy as the first copy. Transparent data migration can be used when adding, removing and/or rebalancing data to tape.

Although a preferred embodiment of the present invention is discussed with reference to Figures 1-2, this is for illustration purposes only. Those skilled in the art will appreciate that the present invention is not limited to any particular grid configuration and may be implemented in any multi-cluster or grid configuration. For example, one or more clusters 220 from site 105(a) may be grouped with one or more clusters 220 from a different site 105 (site 105(b)) to create a first cluster family 280 . Likewise, one or more clusters 220 from site 105(c) and site 105(a) may be grouped into family members to create a second cluster family 280. Thus, any combination of clusters 220 may be grouped into group members to create a cluster family 280 .

FIG. 4 is a schematic block diagram illustrating a preferred embodiment of a cluster cluster device 400 of the present invention. Apparatus 400 may be embodied in host 210 and/or cluster 220 . In a preferred embodiment, the apparatus 400 is embodied in the cluster manager 320 . The description of the apparatus 400 refers to elements of FIGS. 1-3, like numerals referring to like elements. Apparatus 400 may include relationship module 405, creation module 410, collaborative replication module 415, installation processing module 420, communication module 425, and policy module 430, or any combination thereof.

Relationship module 405 includes a computer readable program that executes on a processor, such as the processor of cluster manager 320 . Additionally, cluster relationship module 405 includes factors that define relationships and roles between cluster families 280 and family members 220 . For example, factors related to which family members belong to which families, distance grading between adjacent family and/or group members, and which family members are used for production purposes and which are used for DR (disaster recovery) and/or fulfillment Purpose.

Cluster family members 220 communicate over a network such as network 110 and/or network 215 . Each cluster family member 220 may include a library manager 370 having at least one tape drive configured to access volumes on tape and at least one TVC 365.

Creation module 410 includes executing a computer readable program on a processor, such as a processor of cluster manager 320 . Creation module 410 selects and arranges clusters 220 into family members of cluster family 280 by grouping clusters 22 together to operate by a common set of criteria, rules, and/or objectives.

Creation module 410 groups clusters 220 into cluster families 280 to allow family members 220 to follow a common set of rules or guidelines. This allows groups of clusters, such as families 280(1), 280(2) to work together to more efficiently accomplish a particular task, or allows different groups of clusters 220 and/or families 280 to have different purposes within the grid.

Creation module 410 may be used to allow customizable behavior of family members 220 within family 280 through configuration properties. For example, referring to FIG. 2B, a group of cluster family members 220(a), 220(b) may be allowed to act as a production family 280(1) following a set of rules beneficial to production workloads. Another set of cluster family members 220(c), 220(d) in domain 205 may be allowed to act as archive or disaster recovery family 280(2), with family members 220(c), 220(d) in slave production (1) Rules to operate more efficiently when copying data.

Furthermore, the creation module 410 manages the relationships of the cluster members 220 of the clusters 280 as well as the relationships between different cluster clusters 280 . For example, creation module 410 can manage cluster family members 220 based on their relationships and roles. In a preferred embodiment, relationship module 405 may provide such information to creation module 410 . Cluster family members 220 will coordinate with each other to determine which family member 220 is in the best position to obtain external data from multiple clusters outside of family 280 based on the relationship and/or roles of the family members and neighboring families. Creation module 410 may also use this information to support members 220 of family 280 as TVC clusters, or to allow access restrictions or other special case behavior on families 280 with respect to cluster-only or full-mesh.

The creation module 410 can use the management interface 355 to display a page where a user (eg, customer) can create a cluster family with a character name, such as an 8 character name. The user can then use the creation module 410 to add one or more clusters to the family. Creation module 410 may store this information within the cluster's persistent vital product data so that all clusters in multiple clusters or grid configurations know their cluster's role and the family in which it resides. Creation module 410 may determine that a cluster being selected for a family has already been selected for another family. To avoid having any one cluster appear in two families at the same time, the creation module 410 can notify the user that the selected cluster already exists in another family member. In addition, creation module 410 can use rule sets to prevent a cluster from being selected into two families at the same time.

Policy module 430 includes a computer readable program executing on a processor, such as the processor of cluster manager 320 . In a preferred embodiment, policy module 430 may include specific policies regarding which cluster family member 220 should be used for production and which family member should be used for DR/archive purposes. These policies may include data collection that manages data replication. A user may enter policies for managing multiple cluster families 280 and family members 220 via management interface 355 .

2A and 2B, cluster family creation module 410 may be used to create cluster family 280(1) named "City A" and to create cluster family 280(2) named "City B". Cluster family 280(1) may include a set of cluster family members 220(a), 220(b), and cluster family 280(2) may include a set of cluster family members 220(c), 220(d). Additionally, the creation module 410 may be used to add or remove cluster members 220 to or from cluster clusters 280 , as well as to regroup cluster cluster members into different cluster clusters 280 .

Since the creation module 410 builds the clusters and arranges the clusters into family groups based on their relationships and/or roles to each other during the creation of the clusters, all clusters in a grid or multi-cluster configuration respect each other's roles and the families in which they reside . Accordingly, creation module 410 may alert or notify the user via management interface 355 that cluster 220(d) added to family 280(1) is currently, for example, a family member of another family 280(2). The user can then deselect 220(d) from family 280(2), and add or reselect 220(d) for family 280(1). Thus, the creation module 410 allows all clusters 220 in the domain 205 (e.g., a grid) to understand their own roles and relationships with the families they reside in, with other family members, and with Relationships among non-ethnic members.

In a preferred embodiment, the creation module 410 may assign a name to the cluster family. For example, during configuration, a user may use management interface 355 to assign a name to a cluster family.

Cooperative replication module 415 includes a computer readable program that executes on a processor, such as the processor of cluster manager 320 . In addition, collaborative replication module 415 enhances the management of existing copies to enable groups of clusters 220 belonging to cluster family 280 to more efficiently work together to achieve consistency between.

The collaborative replication module 415 allows two or more cluster family members 220 within a family 280 (eg, a DR or archive family) to share inbound replication workloads. Thus, the family 280 of DR/archive cluster family members 220 using the cooperative replication module 415 can benefit from improved TVC selection when a source cluster is selected for replication.

The collaborative replication module 415 allows cluster family members to share copy workloads among other cluster family members belonging to the same family. For example, in a preferred embodiment, domain 205 includes Y clusters 220 , where Y represents the number of clusters 220 included in domain 205 . Clusters are grouped into cluster families 280 having N (two or more) cluster family members 220 . Thus, domain 205 consists of Y clusters 220 , with some of clusters 220 grouped into N cluster family members of cluster family 280 .

For example, referring to FIG. 2B, there are four clusters 220(a), 220(b), 220(c), and 220(d) in domain 205, so Y represents 4 clusters (Y=4). The two clusters 220(a) and 220(b) are grouped into the N=2 cluster family members of the first cluster family 280(1), and the two clusters 220(c) and 220(d) are grouped into the first cluster family 280(1). Among the N=2 cluster cluster members of a cluster cluster 280(2). In this grid configuration, domain 205 consists of Y(4) clusters, with a subset of N=2 clusters grouped into cluster family 280 cluster members. Therefore, N=2 as the number of cluster cluster members in the cluster.

The cooperative replication module 45 cooperatively replicates clusters of base stations by serializing the replication of either volume when the cluster is first brought into the cluster. For example, cooperative replication module 415 instructs each cluster member in family 280(2) to replicate 1/N of the outer volume, where N is the number of clusters in the family that need to be copied. Once all external volumes are replicated into a family 280(2) and the family 280(2) is cumulatively consistent, then non-consistent clusters within the same family 280(2) share the external data between each.

For example, without the present invention, each cluster 220 works independently of each other, since the clusters 220 do not know the relationships and roles between each other, if possible, from the microcode level. For example, if we assume that cluster 220(a) includes 20 volumes that need to be replicated to clusters 220(c) and 220(d). Since clusters 220(c) and 220(d) work independently of each other, each of clusters 220(c), 220(d) can pull 20 volumes of raw data on network 215.

Referring now to FIGS. 2A and 2B , in a preferred embodiment, for example, there are four clusters, where two clusters 220 ( a ), 220 ( b ) are grouped into clusters 280 ( 1 ) by creation module 410 and two Clusters 220(c), 220(d) are grouped into families 280(2). All family members 220 know each other and the family 280 they belong to, and know all volumes that need to be replicated to adjacent clusters in the family.

For example, cluster family members 220(c) and 220(d) know each other and there are 20 volumes from non-family member 220(a) within a different cluster family 280(1) that need to be replicated to their family 280(2) . Using collaborative replication module 415, family member 220(c) pulls 10 unique volumes and family member 220(d) pulls the other 10 unique volumes. That is, each cluster family member 220(c), 220(d) pulls 1/N of the volume, where N = the number of cluster family members in the cluster. Since in this example there are two cluster family members 220(c), 220(d) belonging to cluster cluster 280(2), each family member pulls 1/2 of the volumes (e.g., each pulls 10 unique rolls) to obtain a total of 20 rolls. Cluster family members 220(c), 220(d) then share 10 unique volumes with each other.

By cooperatively replicating via the cooperative replication module 415, the cluster family 280(2) or DR location can become cumulatively consistent N times faster since any one volume is only pulled over the remote link 110/235 once instead of N times. Thus, cluster family members 220 may more quickly become consistent with each other in availability due to their relative distances. Thus, by having each cluster 220 independently originate from the same remote production cluster 220, the overall time to become DR consistent and highly available (HA) consistent can be greatly enhanced.

Thus, copy throughput can be optimized and overall time improved to achieve volume consistency within the cluster family 280 . For example, in a bandwidth-constrained system or a grid with multiple archival sites, the collaborative replication module 415 allows each family member 220 in a family 280 to participate in the replication process for all inbound copies without duplicating any effort. Once a group (family member) of clusters 220 within a family 280 reaches an aggregate consensus state, a consistent copy in each cluster 220 within the family 280 is shared among peer clusters within the same family.

Additionally, the collaborative replication module 415 handles ongoing replication source awareness by deferring replication. For example, a cluster member 220 with a consistent source may be notified to keep the source volume in cache, making it readily available to other family members 220 for peer-to-peer replication. A cluster with a consistent source inherits the role of the original install source cluster or the cluster containing the original copy created/modified by the host. Once a cluster in the family replicates one of its 1/N volumes, the cooperative replication module 415 first notifies the original installation source cluster, stating the reason for all other clusters in its family (including itself) and the production cluster can de-represent the target cluster. The cluster's own role. This frees the production cluster to organize volumes out of backend tape (assuming there are no other families or production clusters that need to be copied), thus providing more cache availability. Second, the DR family cluster that initiated the replication for the volume inherits the role and remembers which clusters within its family still need to be copied. Through this inheritance, a volume can be backed in cache until all its peer clusters have completed the copy.

In a preferred embodiment, the collaborative replication module 415 may use a cascaded copy requirement flag. For example, as cluster families become consistent, cooperative replication module 415 moves ownership of a copy requirement flag from one cluster family to another. By cascading copying demand tokens, the collaborative replication module 415 can allow the benefits of tokens to be moved from one family to another, thus releasing the original TVC it participated in. By inheriting the copy demand flag from the TVC, for example, once a family member gets a copy, it can allow the TVC cluster to migrate volumes and allocate space in cache for other new workloads.

An instance could be a domain that includes a product or a default family connected to a DR/Archive family. A TVC cluster can be a member of the production or default family and can start managing copy demand flags. Once a member of the DR/archive family has obtained a copy from the TVC cluster, the DR/archive family may notify the TVC cluster to clear all copy required flags associated with the member of the DR/archive family. In conjunction with this, the DR/Archive family can inherit responsibility for managing these copy requirement flags for its family members.

For example, in another embodiment of storage system 200 (not shown), domain 205 may include: a first family cluster 280(1) including cluster family members 220(a), 220(b), 220(c ); a second cluster 280(2) comprising cluster cluster members 220(d), 220(e), 220(f); and a third cluster cluster 280(3) comprising cluster cluster members 220(g) , 220(h), 220(i). Each family 280 includes three cluster family members 220 and each family member represents a bit. Since there are three families and each family has three family members (3 bits), there are a total of 9 bits in the bit set. For example, cluster 280(1) includes original data objects that need to be copied into clusters 280(2), 280(3).

It is possible that cluster 220(a) may hold the volume in cache until all nine clusters 220 have pulled copies across network 110 or 215 . For example, cluster 220(a) may include a 9-bit set and, when each cluster 220 pulls a copy, cluster 220(a) may clear bits in its mask. Since cluster 220(a) holds copies for all nine clusters 220 in its cache, cluster 220(a) cannot allocate space for the additional workload.

By allowing each cluster family 280 to inherit responsibility for managing these copy demand flags for its family members, the cluster 220(a) can clear the remaining 6 bits for these cluster families 280(2), 280(3), and Only keeps in its cache for its own copy of the two family members 220(b), 220(c) residing in the family 280(1). Once its own family members 220(b) and 220(c) have copies, cluster 220(a) can then clear its mask to allocate room for more workloads.

In this example, cluster 220(d) of family 280(2) pulls a copy on network 215 and informs cluster 220(a) of family cluster 280(1) that since 220(d) will keep the copy in its cache until Its family member 220(e), 220(f) receives the copy location, it no longer needs to hold a copy for family 280(2) in cache. This eliminates the need for cluster 220(a) to hold copies in cache for all cluster members of cluster family 280(2). Similarly, family member 220(g) belonging to family 280(3) indicates that cluster 220(a) will maintain a copy of its family members 220(h), 220(i) in its cache. Thus, family 220(a) is exempt from holding copies in its cache for all family members belonging to 280(3).

Furthermore, the cooperative replication module 415 can increase performance in low-bandwidth environments and improve the overall time for intra-cluster clusters to become consistent by using more links in the domain to perform the copy rather than relying primarily on the copy from the TVC . For example, families 280 using collaborative replication module 415 cooperate to achieve cross-family consistency. Upon receiving a family-wide consensus, family members 220 can then work together to share data among the family members to bring each respective member to the family's level of consistency.

Installation processing module 420 includes a computer readable program executing on a processor, such as a processor of cluster manager 320 . The mount processing module 420 supports and selects cluster family members within its own family on clusters external to its family when a mount occurs using the cluster's logical volumes. For example, an installation to a production cluster may support another production cluster in the same family 280(1) by being primarily used for DR or an e-vault remote cluster. When production data needs to remain highly available with local and fast replication, the installation processing module 420 can be used to support availability through disaster recovery, and thus select family members within the production family by the DR family.

The installation processing module 420 can improve control and performance by supporting cluster family members when remote installation is required. A family and/or family members may be configured (eg, using creation module 410 ) to prefer a particular cluster over other base stations when selecting a remote TVC. This is beneficial in differentiating the set of production clusters from non-production clusters. Preferably, family members within the same product family can maintain TVC selection within a production cluster, rather than potentially selecting clusters that are distant in distance for DR or archival purposes.

Furthermore, since cluster family members 220 within a cluster family 280 are targeted for installation and are supported in the same cluster family 280, the TVC selection process may be improved.

In a preferred embodiment, storage system 200 may include multiple clusters, wherein two or more subsets of clusters 220 are grouped into a first cluster family 280, and two or more subsets of clusters 220 are grouped into In the second cluster family 280 . The grouping of clusters may be based on the roles, relationships, and/or distances of the cluster members to each other and/or to other non-clan member clusters. Each cluster family member of cluster family 280 is aware of their relationship to each other. This awareness of relationships among family members allows groups to work efficiently together to cumulatively replicate data into the group and then among each other.

A site 105 may include a cluster family 280 or a combination of cluster families 280 . For example, site 105(a) may include a first cluster family 280 and a second cluster cluster family 280 . The first cluster family 280 may include production clusters 220(a), 220(b), and the second cluster cluster 280 may include production clusters 220(c), 220(d). Additionally, clusters 220 may be selected from a combination of sites 105 to create a cluster family 280 . For example, cluster family 280 may be created by selecting clusters 220 at different sites 105 (e.g., 105(a) and 105(b) where clusters 220(a), 220(b) are used for production purpose and the clusters 220(c), 220(d) at site 105(b) are used for DR and/or archival purposes.

In a preferred embodiment, clusters 220(c), 220(d) are used for archiving data. In a preferred embodiment, clusters 220(c), 220(d) are used for DR. In another embodiment, one cluster, such as cluster 220(c), is used for DR, and another cluster, such as cluster 220(d), is used for archiving.

Generally, the following exemplary flow diagrams detail logical flow diagrams. As such, the depicted order and labeled steps are indicative of the preferred embodiment of the method. Other steps and methods which are equivalent in function, logic, or effect may be conceived as one or more steps of the described method, or portions thereof. Additionally, the format and symbols used are provided to explain the logical steps of the methods and are understood not to limit the scope of the methods. Although various arrow types and line types may be used in the flowchart, they are not to be construed as limiting the scope of the corresponding method. In fact, some arrows and other connectors can be used to just indicate the logical flow of the method. For example, an arrow may indicate a waiting or monitoring period of no specific duration between enumerated steps of the depicted method. Furthermore, the order in which a particular method occurs may or may not be limited to the order of the corresponding illustrated steps attached.

Fig. 5 is a schematic flowchart showing a preferred embodiment of the cluster family selection and cooperative replication method of the present invention. The method 500 essentially includes performing the steps presented above in relation to the operation of the apparatus and method presented in the presentation of FIGS. 1 to 4 . In a preferred embodiment, the method is implemented by means of a computer program product comprising a computer readable medium with a computer readable program. A computer readable program may be integrated into a computer system, such as the cluster manager 320 and/or the host computer 210 , wherein the program associated with the computing system is capable of performing the method 500 .

Method 500 begins and at step 510 a set of clusters is arranged into cluster members of cluster families. For example, grouping clusters based on their relationship to each other and to other clusters in the domain. Cluster families are created based on various factors and/or functions, including role (eg, production source, DR, archive, etc.), scope, distance (eg, distance ratio between families), and the like. Additionally, users can assign character names to cluster families. For example, as shown in FIG. 2B, a family of clusters may be created using the character name ("City A") and another family may be created using the character name "City B".

In a preferred embodiment, the creation module 410 is used to create a cluster family, wherein the user can use the management interface 355 to create a cluster family during configuration to create a cluster family name, add one or more clusters to a family, Assign roles and/or distance ratios, and use configuration properties to teach the cluster. For example, the creation module 410 may use these persistent settings to bring awareness of relationships to one or more cluster or family members as well as related attributes between families, such as distance.

In a preferred embodiment, relationship module 405 maintains these persistent settings for cluster families and family members.

In addition, autonomous functions can be used to detect relationships between roles and clusters. For example, in the creation module 410 an autonomous function cluster is performed.

In step 515, the family members negotiate among themselves to determine which family member of the family is in the best position to obtain the external data object. For example, as shown in FIG. 2B , a cluster family 280 ( 1 ) including two or more cluster members 220 ( a ), 220 ( b ) may be configured at urban distances and may be used for production purposes. The cluster family 280(2), comprising two or more family members 220(c), 220(d), may be configured with reference to the family 280(1) at global distances and may be used for DR purposes. Cluster family 280(1) may communicate with cluster family 280(2) via network 110, 215 ready to copy data objects. Since the cluster members of each family, as well as the cluster itself, know each other's role and relationship to each other, the family members 220(c), 220(d) can negotiate among themselves to determine which of the family 280(2) Family members are in the best position to obtain a copy of the data object's exterior.

In a preferred embodiment, for example, cluster family members 220 belonging to cluster family 280 work in FIFO order using a common copy work queue. Before working on the copy, each cluster family member 220 first ensures that no other cluster family member in the cluster family 280 is already copying or has copied. If not, one or more cluster family members 220 may perform the copy. One or more cluster family members may move the copy to the delay queue if another family member is already copying or if another family member has already taken the copy. After some time, after all the active production content has been copied into the cluster family 280, the family members start working on the delay queues where they should share content with each other. If the peer family member that originally obtained the copy is not present, it can still obtain the copy from an external cluster or another family member.

In step 520, one or more cluster family members obtain and replicate information or source volumes. For example, one or more cluster family members 220 belonging to the cluster family 280 are selected to pull data or source volumes on the remote network 110 , 215 , copy/duplicate the data or source volumes, and bring them into the cluster family 280 . For example, family member 220(c) of family 280(2) pulls an external data object over network 110, 215 to family 280(2). Family member 220(c) now has a consistent source and may ask family member 220(c) to keep the source volume in cache (e.g., TVC 365) so that it is readily available for peer-to-peer replication.

In step 525, the source volume is collaboratively replicated among family members of the family. For example, when a cluster is first brought into a family, the cluster's family groups collaborate by serializing the replication of any one volume. Each cluster in the family can be used to replicate 1/N volumes, where N is the number of clusters within the family that need to be replicated.

By replicating cooperatively, a cluster family or DR location can become cumulatively consistent N times faster due to pulling any one volume only once rather than multiple times across long-distance links. Then, due to their relative distance, the clusters can become consistent with each other for availability more quickly. The overall time to become DR consistent and HA (High Availability) consistent can be greatly increased on each cluster, independent of sources from the same remote production cluster.

At step 530, the cluster family achieves cumulative consistency. That is, complete all volumes outside the cluster family that need to be replicated to the cluster family. The cluster family as a whole is consistent with all external data objects. Cluster family members can now be shared among each other such that each individual family member within the cluster family has its own copy.

At step 535, after all volumes are replicated into the cluster and the cluster is cumulatively consistent, inconsistent clusters within the same cluster then share volumes (ie, data objects) with each other.

Thus, the implementation method 500 of the present invention performs replication cooperatively, where a cluster family or DR location can become cumulatively consistent N times faster due to pulling any one volume across long-distance links only once rather than multiple times. Then, due to their relative distance, the clusters can become consistent with each other for availability more quickly. The overall time to become DR consistent and HA (High Availability) consistent can be greatly increased on each cluster, independent of sources from the same remote production cluster.

Furthermore, method 500 uses a family of more efficient methods of replicating to X clusters (where X represents the number of clusters) when customers only need N copies and those N copies are far from each other. This allows clients to spread copies across distances/clusters without specifying which clusters receive the copies. For example, a user may not care which cluster contains a copy, as long as there are N copies (where N is less than X); and a client requires that N copies all exist in a separate family. Thus, all base stations in the domain can cooperate to ensure that at least one member from each family replicates the volume, and then the remaining clusters can meet their replication requirements. As such, one can end up with N copies in N families without having too many of the N copies in any one region.

The steps of method 500 may be used in any combination during the installation process. For example, with the cluster family configured in step 510, using steps 515 through 535, method 500 may support clusters of its own family on clusters external to its family. For example, an installation to a production cluster may support another production cluster (in a system family) on a remote cluster that was originally used for disaster recovery (electronic vault). Since users may tend to want production data to maintain local and fast-replicated high availability (to support availability through disaster recovery), it is more efficient to get a production cluster based on short-term goals and still not compromising long-term goals.

Referring to FIGS. 6A and 6B , there are schematic flowcharts illustrating a preferred embodiment of the cluster family selection and cooperative replication method of the present invention. The method 600 essentially includes steps to perform the functions presented above with respect to the operation of the devices and systems described above with respect to FIGS. 1 to 4 . In a preferred embodiment, the method is implemented by means of a computer program product comprising a computer readable medium with a computer readable program. A computer readable program may be integrated into a computer system, such as integrated manager 320 and/or host 210 , where the program associated with the computing system is capable of performing method 600 .

Method 600 begins and at step 605, the copy process begins. For example, an external data object in city A needs to be replicated in city B (eg, Figure 2B).

In step 610, control determines whether the cluster receiving the copy request is a cluster family member. If not, at step 615, the volume is copied and no collaborative copy is performed. For example, collaborative replication module 415 can manage copy requests without delay or priority changes.

Additionally, the collaborative replication module 415 can select at least one family member in the family to pull data across distant links or networks. Selection can be performed once it is determined that the cluster is a family member and no other family members are pulling data across the network.

If this is a cluster family member, at step 620 control determines whether one of the other family members has finished copying the volume. If so, at step 625, since one of the other family members has already copied the volume, the copy volume is given lower priority and the volume is copied back to the queue.

If one of the family members has not completed copying the volume, in step 630, control determines whether another family member actively includes the volume. If so, at step 635, the priority for copying this volume is lowered and there is a delay before returning to the queue. The delay before sending the copy request back to the queue ensures, for example, that another family member actively copying the volume does not encounter any problems with the copy volume.

In step 630, if there are no other family members actively copying the volume, then in step 640, control determines whether another family member is also ready to copy the volume, but not actively copying at this time. If not, in step 645, control determines whether such other family members that are not actively copied at this time should inherit the copy requirement flag. If so, at step 645, the cluster lowers the copy priority and delays back to the queue.

If no in step 645, method 600 moves to step 655 and the family member wins the connection breaker between the two cluster members and inherits the copy flag. Accordingly, in step 645, control determines which family member is to be designated to inherit the copy flag. Non-designated family members lower the copy priority and delay back to the queue (eg, step 650).

Returning to step 640, if another family member is not ready to copy the volume, then in step 655, control determines that only one family member in the cluster family is ready to copy the volume and designates the family member as the cluster to inherit the copy flag And complete the copy.

It should be noted that at step 640, control may determine that there is another cluster that is ready to copy and is not actively copying at this time, but as shown in step 645, determines that the other clusters will not inherit the copy flag. Therefore, in step 640, the cluster will inherit the copy flag, as shown in step 655.

At step 660 , the specified cluster that inherited the copy flag in step 655 completes the copy.

In step 670, control clears the copy required flag at the source cluster and cooperates to cumulatively make the family consistent by setting the copy required flag for the family members of the cluster family.

In step 675, the other family members of the cluster member complete their copy and reset their copy required flags set in step 655 within the cluster assigned the integrated copy flag.

Figures 1 to 3 may be indicative of a multi-cluster configuration. In a multi-cluster configuration or (grid configuration), each cluster may be unaware of its relationship and role to itself and to other clusters from a microcode perspective, and thus works equally and independently from all others. For example, when two or more clusters are configured remotely globally from one or two production clusters, they can be replicated individually by "pulling" data over the remote network. Since clusters are not relationship aware, they cannot operate in the most efficient manner based on their role and/or distance from other clusters.

Furthermore, in a multi-cluster configuration, due to this unawareness of the relationship, the device that selects the cluster to obtain the volume during installation and the ability of the cluster to obtain volume replication is largely affected. For example, a production cluster may select a global remote source cluster over a city remote cluster for installation and/or copy processing. Globally remote clusters are much less efficient due to the network distance between clusters.

Implementations of the present invention are able to address these issues by introducing awareness of family members and relationships between families in a multi-cluster or grid configuration. Furthermore, implementing the present invention can improve the performance, efficiency and optimization of data copying and/or replication. For example, cooperatively replicating to the family, thereby achieving cumulative family consistency N times faster and reducing the overall time to become DR consistent and HA consistent by using only 1/N of cumulative network throughput can improve efficiency and performance.

Referring to Figures 1 to 6, implementation of the present invention may involve software, firmware, microcode, hardware, and/or any combination thereof. Implementations may take the form of code or logic implemented in a medium, such as the memory, storage, and/or circuitry of hierarchical storage node 315, where the medium may include hardware logic (e.g., an integrated circuit chip, a programmable gate array [ PGA], application-specific integrated circuit [ASIC], or other circuits, logic, or devices), or computer-readable storage media, such as magnetic storage media (e.g., electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, semiconductor or Solid State Memory, Magnetic Tape, Removable Computer Tape, and Random Access Memory [RAM], Read Only Memory [ROM], Hard Disk and Optical Disk, Compact Disk - Read Only Memory [CD-ROM], Compact Disk - Read/Write [CD -R/W] and Digital Video Disk (DVD)).

Those skilled in the art will readily appreciate that changes may be made to the manner discussed above, including changes to the order of the steps. Furthermore, those skilled in the art will appreciate that specific component arrangements may be used other than those shown here.

Although preferred embodiments of the present invention have been described in detail herein, it should be understood that various modifications can be made to these embodiments by those skilled in the art without departing from the scope of the invention as set forth in the following claims. modifications and changes.

Claims (25)

1. A method for collaborative replication of a plurality of clusters, said method comprising the steps of: arranging at least a subset of the plurality of clusters into family members of cluster families; negotiating among cluster family members to determine which cluster family member is in the best position to obtain at least one external data object from at least one cluster external to the cluster; Select a family member of the cluster family to obtain the external data object; achieve cumulative consistency with external data objects within the cluster family; and The external data object is shared among cluster family members such that each cluster within the cluster family is consistent with the external data object. 2. The method of claim 1, further comprising the step of creating relationships between cluster families based on at least one of cluster relationship factors and role factors. 3. The method of claim 1, further comprising the steps of: Family members with consistent sources maintain volumes in cache so that the volumes are readily available to other family members for peer-to-peer replication. 4. The method of claim 3, further comprising the steps of: The family member maintains the volume in cache for other family members, freeing the external cluster from maintaining a copy in the external cluster's cache. 5. The method of claim 1, further comprising the steps of: Each of the N family members in the family replicates 1/N volumes of external data objects. 6. A method according to any one of claims 3 to 5, further comprising the step of cooperatively serializing all replicas into cluster families. 7. The method of claim 1, further comprising the steps of: The first of N family members replicating 1/N volumes of the external data object informs the external cluster that the first family member will keep the volume for the cluster family member and exempts the external cluster from maintaining the volume in the external cluster cache. 8. The method of any one of claims 1 to 5, further comprising the step of providing a plurality of cluster families, wherein at least one family member from each family replicates a volume. 9. The method of claim 1, further comprising the step of providing a domain comprising a plurality of clusters, wherein the clusters cooperate to ensure that at least one family member from each family replicates a volume and that the remaining clusters obey replication requirements. 10. The method of claim 1, further comprising the steps of: receiving a copy request to copy the volume to the first cluster; determining whether the first cluster is a family member of a cluster family; In response to the first cluster being a first family member, determining which family member in the specified cluster family inherits the copy request; In response to the determination, executing a copy request and cooperatively replicating the volume into the cluster family; achieve cumulative consistency within cluster families; and The volume is shared within the cluster family such that all copies of the volume within the cluster family are consistent. 11. The method of claim 10, wherein, in response to the first cluster being a first family member, determining which family member of the specified cluster family inherits the copy request comprises the steps of: determining whether another family member has finished copying the volume, In response to another family member not having copied the volume, the first family member is designated to inherit the copy request. 12. The method of claim 10, wherein, in response to the first cluster being a first family member, determining which family member of the specified cluster family inherits the copy request comprises the steps of: determining whether the second family member is actively copying the volume; and In response to the second family member actively copying the volume, the second family member is designated to inherit the copy request. 13. The method of claim 12, wherein, in response to the first cluster being a first family member, determining which family member of the specified cluster family inherits the copy request further comprises the step of: Determine whether the second family member is ready to copy, but is not actively copying at this time; In response to the second family member being ready to copy and not actively copying at this time, the copy priority of the second family member is lowered and the copy request is delayed. 14. The method of claim 10, further comprising the steps of: In response to determining that the second family member inherits the copy request, lowering the copy priority of the first family member and delaying the copy request. 15. The method of claim 10, further comprising the steps of: Designate members of the first family to inherit the copy request as the source cluster of the cluster family; Copies are made at members of the first family; Clear the copy request flag at the first family member; setting a copy request flag at the first family member for other family members; Other family members inherit the copy request flag; other family members complete the copy; and Each family member resets the copy request flag. 16. A system for collaborative replication of multiple clusters, the system comprising: network; a plurality of sites in communication over a network, each site comprising at least one host and a storage system comprising a plurality of clusters, each cluster comprising at least one tape drive configured to access volumes stored on tape, at least one tape volume cache, and a cluster manager comprising: create modules for establishing cluster families and grouping clusters into cluster family members; and A cooperative replication module for selecting family members to cooperatively replicate external data objects into cluster families and achieve cumulative consistency. 17. The system of claim 16, wherein the collaborative replication module is further for sharing replication among each family member. 18. The system of claim 17, wherein the collaborative replication module is further for instructing each of the N family members of the family to replicate 1/N volumes of the external data object. 19. The system of any one of claims 16 to 18, wherein the creation module is further configured to create a cluster family based on at least one of an inter-cluster relationship factor and a role factor. 20. An apparatus for collaborative replication of multiple clusters, the apparatus comprising: Create a module for creating a cluster family from multiple clusters, where the clusters communicate over a network and each cluster includes a cache; and A cooperative replication module for cooperatively replicating at least one external data object to the cluster family and achieving cumulative consistency. 21. The apparatus of claim 20, wherein the creation module is further for establishing cluster families and grouping clusters into cluster family members based on at least one of a cluster relationship factor and a role factor. 22. The apparatus according to any one of claims 20 to 21, wherein the collaborative replication module is further configured to handle persistent replication source awareness by delaying replication. 23. The apparatus according to any one of claims 20 to 21, wherein the cooperative replication module is further configured to instruct each of the N family members in the cluster family to replicate 1/N volumes of the external data object. 24. The apparatus according to any one of claims 20 to 21, wherein the cooperative replication module is further configured to: select a family member of the cluster family to obtain an external data object, and serve as the source cluster; and making the source cluster responsible for achieving cumulative consistency within the cluster family before sharing external data objects. 25. Apparatus according to any one of claims 20 to 21, further comprising: A relationship module that holds factors that define roles, rules, and relationships between cluster families and family members.