HK1096170B - Emulated storage system - Google Patents

Description

Emulated storage system

Background

Technical Field

Aspects of the present invention relate to data storage and, more particularly, to methods and apparatus for emulating a tape storage system to provide the equivalent of using an existing full back-up and a later incremental full back-up and to enable an end user to restore data from the back-up.

Prior Art

Many computer systems include one or more hosts and one or more storage systems that store data used by the hosts. These hosts and storage systems are typically connected together by a network, such as a fibre channel, ethernet, or other type of communications network. Fibre channel is a standard that combines the speed of channel-based transport mechanisms and the flexibility of network-based transport mechanisms and allows multiple initiators to communicate over a network with multiple targets, which may be any device coupled to the network. Fibre channel is typically implemented using fast transmission media, such as fiber optic cables, and is a popular choice in storage networks that transmit large amounts of data.

An embodiment of a typical networked computing environment includes several host computers and a backup storage system as shown in FIG. 1. One or more application servers are coupled to a plurality of user computers 104 via a local area network 103 (LAN). Both the application server 102 and the user computer 104 may be considered "hosts". The application server 102 is coupled to one or more primary storage devices 106 via a storage area network 108 (SAN). The primary storage device 106 may be, for example, a disk array, available from companies such as EMC, IBM, and the like. Alternatively, a data transfer bus (not shown) or other network connection may provide interconnection between the application server and the primary storage system 106. The connection of the data transfer bus and/or the fibre channel network may operate using a protocol such as the Small Computer System Interface (SCSI) protocol, which controls the transfer of packets in a format between a host (e.g., application server 102) and storage system 106.

It will be appreciated that the networked computing environment illustrated in FIG. 1 is typical of a large system, such as one used by a large financial institution or large corporation. It is to be appreciated that many networked computing environments need not include all of the components listed in FIG. 1. For example, a smaller networked computing environment may be reduced to include a storage system connected directly to a host computer, or through a LAN. Additionally, although FIG. 1 illustrates a single user computer 104, application server 102, and media server 114, these functions may be combined into more than one computer.

In addition to the primary storage device 106, many networked computing environments include at least one secondary or backup storage system 110. The back-up storage system 110 is typically a tape library, although other large capacity, reliable secondary storage systems may be used. Typically, these secondary storage systems are slower than the primary storage systems, but include some type of removable media (e.g., magnetic tape, disk, or disc) that can be removed and stored off-site.

In an illustrative embodiment, the application server 102 may be capable of communicating directly with the backup storage system 110 via, for example, an Ethernet or other communication connection 112. However, the connection may be relatively slow and take up resources, such as processor time or network bandwidth. Thus, the illustrated system may include one or more media servers 114, the media servers 114 providing a communication link between the SAN108 and the backup storage system using, for example, fibre channel.

The media server 114 may run software that includes a backup/restore application that controls the transfer of data between a host computer (e.g., user computer 104, media server 114, and/or application server 102), primary storage device 106, and backup storage system 110. Backup/restore applications are available from companies such as Veritas, Legato, and others. For protection of data, data from various hosts and/or primary storage devices may be periodically backed up in a networked computing system to a backup storage system 110 through a backup/restore application, as is well known in the art.

Of course, as discussed above, it is to be appreciated that many networked computing environments may be smaller and include fewer components than the exemplary networked computing environment illustrated in FIG. 1. Thus, it will be appreciated that the media server 114 is actually connected to the application server 112 in a separate host computer, and that the backup/restore application may be executed on any host computer coupled (directly or indirectly, e.g., via a network) to the backup storage system 110.

One example of a typical back-up storage system is a tape library that includes a plurality of tape cartridges and at least one tape drive, and a robot that controls loading and unloading of the tape cartridges into and out of the tape drives. The backup/restore application provides instructions to the robot to determine the location of a particular tape cartridge, such as tape number 0001, and to load the tape cartridge into the tape drive so that data can be written to the tape. The backup/restore application may also control the format of data written to the tape. Typically, the back-up/restore application may use SCSI commands, or other standardized commands, to instruct the robot and control the tape drive to write data to the tape and restore written data from the tape.

Conventional tape library backup systems suffer from several problems including speed, reliability, and fixed capacity. Many large companies need to back up terabytes of data weekly. However, even expensive, high-end tape is typically only capable of reading/writing data at 30-40 megabytes/second (MB/s), which translates to about 50 gigabytes/hour (GB/hr). Thus, backing up one or two terabytes of data into a tape backup system may require at least 10-20 hours of continuous data transfer time.

In addition, most tape manufacturers are not guaranteed the possibility of storing (or retrieving) data to/from tape if the tape is lost (which may occur relatively frequently in a typical tape library due to human manipulation or robotic mechanisms during a move or load operation) or if the tape is exposed to non-ideal environments, such as extreme temperatures or humidity. Therefore, a great deal of effort is required to store tapes in a controlled environment. Moreover, the complex mechanics of the tape library (including the robotic mechanisms) are expensive to maintain, and individual tape cartridges are relatively expensive and have a limited lifespan.

Disclosure of Invention

Embodiments of the present invention provide a back-up storage system that overcomes or alleviates some or all of the problems of conventional tape library systems and may provide more flexibility than conventional tape library systems.

In summary, aspects of embodiments of the present invention provide a random access based storage system that emulates a conventional tape back-up storage system such that the back-up/restore application recognizes the same devices and media as a real tape library. The storage system of the present invention uses software and hardware to emulate physical tape and replace it with one or more random access disk arrays to convert tape format, linear, sequential data to data suitable for storage on disks.

In accordance with certain aspects of embodiments of the present invention, apparatus is provided for decoding existing backup data sets and storing metadata (e.g., data representing user information) in a queryable metadata cache, which allows querying and/or reading of files or objects in the metadata cache, and which downloads such files or objects from data stored by existing backup policies via network connections and operation of typical backup software. It is also possible to include means for authenticating the user by means of existing authentication means and means for restricting the view of the metadata cache to the basis of the credentials of the current user.

In accordance with other aspects and embodiments of the present invention, means are provided for performing a logical merge of multiple cartridge representations in a metadata cache, and means for appropriately marking and barcoding a new synthetic cartridge so that the cartridge may be accepted by backup/restore software as an efficient data set. Moreover, in accordance with a further aspect of an embodiment of the present invention, means are provided for storing both multiple copies of data elements representing synthetic tapes and for storing pointers only in existing data in the metadata cache.

According to one embodiment, a method includes the steps of creating a complete backup data set comprising a first plurality of data files, creating at least one incremental backup data set comprising a second plurality of data files, determining a most recent copy of each of the first and second plurality of data files, storing a plurality of indicators, each indicator identifying a storage location of the most recent copy of each of the first and second plurality of data files in one of the complete backup data set and the at least one incremental backup data set, and creating a synthetic complete backup data set corresponding to each of the first and second plurality of data files based on the plurality of indicators.

In one embodiment, the method further comprises the step of storing metadata associated with each of the first and second pluralities of data files in a searchable metadata cache. In another embodiment, the step of determining the most recent copy includes analyzing metadata associated with each data file of the first and second pluralities of data files to determine the most recent copy of each data file of the first and second pluralities of data files. According to another embodiment, the method further comprises the step of providing an interface to allow a user to locate a searchable metadata cache to locate a most recent copy of at least one data file in one of the first and second pluralities of data files, wherein the step of providing an interface may also include the step of providing an interface to allow a user to download and retrieve the most recent copy of the at least one file data using the interface.

In other embodiments, the method further comprises the step of analyzing one of the existing full back-up data set and the incremental data set or analyzing to provide a list of each of the first and second pluralities of data files. In one embodiment, the first plurality of data files includes a first data file, and wherein the second plurality of data files includes a modified copy of the first data file. In another embodiment, the second plurality of data files includes at least one data file whose version is not included in the first plurality of data files.

According to another embodiment, a backup storage system includes a backup storage medium for storing a first and second plurality of data files, and a controller including at least one processor configured to execute a set of instructions to perform an embodiment of the method described above. In another embodiment, a computer-readable medium is encoded with a plurality of instructions that execute the above-described embodiments on at least one processor (e.g., a processor of a storage system).

In another embodiment, the synthetic full back-up data set includes a plurality of indicators, each indicator identifying a storage location of a most recent copy of the first plurality of data files in one of an existing full back-up data set and at least one incremental back-up data set, wherein the existing full back-up data set includes a stored copy of at least one of the plurality of data files, and wherein the at least one incremental data set includes a stored copy of a retained portion of the plurality of data files.

In accordance with another embodiment, a method is provided for creating a synthetic full back-up data set from an existing full back-up data set including a first plurality of data files and an incremental back-up data set including a second plurality of data files. The method includes determining a most recent version of each of the first and second pluralities of data files, storing a plurality of indicators, each indicator identifying a storage location of a most recent copy of each of the first and second pluralities of data files in one of an existing full back-up data set and an incremental back-up data set, and creating a synthetic full back-up data set corresponding to the most recent copy of each of the first and second pluralities of data files, the data files created based on the plurality of indicators.

Drawings

The figures are not drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

FIG. 1 is a block diagram of one embodiment of a large networked computing environment including a backup storage system;

FIG. 2 is a block diagram of one embodiment of a networked computing environment including a storage system in accordance with aspects of the present invention;

FIG. 3 is a block diagram of one embodiment of a storage system in accordance with aspects of the present invention;

FIG. 4 is a block diagram illustrating a virtual design of one embodiment of a storage system in accordance with aspects of the present invention;

FIG. 5 is a schematic diagram of one embodiment of a system file in accordance with aspects of the present invention;

FIG. 6 is one embodiment of a tape directory in accordance with aspects of the present invention;

FIG. 7 is a diagrammatic illustration of one embodiment of a method for creating a synthetic full backup in accordance with aspects of the present invention;

FIG. 8 is a schematic diagram of one embodiment of a series of backup data sets including synthetic full backups in accordance with aspects of the present invention;

FIG. 9 is a diagram of one embodiment of a structure of a metadata cache;

FIG. 10 is a diagram of one embodiment of a virtual cartridge storing a synthetic full back-up data set; and

FIG. 11 is a diagram of another embodiment of a virtual cartridge storing a synthetic full back-up data set.

Detailed Description

Various embodiments and aspects are described in more detail below with reference to the accompanying drawings. It is to be understood that the invention is not limited in its application to the details of construction and the arrangements of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

As used herein, the term "host" refers to any computer having at least a processor, such as a personal computer, workstation, mainframe, network client and server, etc., that is capable of communicating with other devices, such as a storage system or other host. Hosts may include media servers and application servers (as described above in connection with FIG. 1) as well as user computers (which may be user workstations, personal computers, mainframes, etc.). In addition, in the disclosed document, the term "networked computing environment" includes any computing environment in which multiple hosts are connected to one or more shared storage systems, in which manner the storage systems may communicate with each host. Fibre channel is one example of a communication network that may be used in embodiments of the present invention. However, it is to be appreciated that the networks described herein are not limited to fibre channel, and that the various network components may communicate with each other via any network connection, for example, Token Ring or ethernet, rather than, or in addition to, fibre channel, or via a combination of different network connections. Moreover, aspects of the present invention may also be used in data transfer bus topologies, such as SCSI or parallel SCSI.

According to various embodiments and aspects of the present invention, a virtual removable media library backup storage system is provided that may use one or more disk arrays to emulate removable media based on the storage system. Using embodiments of the present invention, data can be backed up to the disk array using a similar back-up/restore application, as is used to back-up data to removable media (e.g., tapes, disks, compact disks, etc.), eliminating the need for the user to make any modifications or adjustments to existing back-up programs or to purchase a new back-up/restore application. In one embodiment, the removable media is emulated as tape, and the back-up storage system of the present invention emulates a tape library system comprising tape and a robot for handling the tape in a conventional tape library system, as described in detail herein.

A storage system according to aspects of the present invention includes hardware and software that interfaces with a host computer (running a backup/restore application), and a backup storage medium. The storage system may be designed to emulate tape, or other types of removable storage media, such that the back-up/restore application views the device as a physical tape library and converts linear, sequential data, tape format data into data suitable for storage on a random access disk. In this manner, the storage system of the present invention may provide increased functionality (e.g., allowing a user to query individual backed-up user files, as discussed below) without requiring new backup/restore applications or policies.

Referring to FIG. 2, there is illustrated in block diagram form one embodiment of a networked computing environment including a back-up storage system 170 in accordance with aspects of the present invention. As illustrated, the host 120 is coupled to the storage system 170 through a network connection 121. The network connection 121 may be, for example, a fibre channel connection to allow high speed transfer of data between the host 120 and the storage system 170. It is to be appreciated that host 120 can be, or can include, one or more application servers 102 (see FIG. 1) and/or media servers 114 (see FIG. 1) and can facilitate the backing up of data from either any computer present in the networked computing environment or from primary storage 106 (see attachment 1). In addition, one or more user computers 136 may be coupled to the storage system 170 through additional network connections 138, such as an Ethernet connection. As discussed in detail below, the storage system may enable a user of the user computer 136 to read and selectively restore backed-up user files from the storage system.

The storage system includes a backup storage medium 126, which backup storage medium 126 may be, for example, one or more disk arrays, as explained in more detail below. The backup storage media 126 provides the actual storage space for backup data from the host computer 120. However, the storage system 170 may also include software and additional hardware that emulates a removable media storage system, such as a tape library, such that the back-up/restore application is run on the host computer 120 such that data appears to be backed-up to conventional removable storage media. Thus, as explained in FIG. 2, the storage system 170 may include "emulated media" 134, representing, for example, virtual or emulated removable storage media (e.g., tape). The "emulated media" 134 is presented to the host computer by the storage system software and/or hardware, and is presented to the host computer 120 as physical storage media. Further interfaces between the emulated media 134 and the actual back-up storage media 126 may be a storage system controller (not shown) and a switching network 132, the switching network 132 receiving data from the host computer 120 and storing the data on the back-up storage media 126, as discussed in more detail below. In this manner, the storage system "emulates" a conventional tape storage system to the host computer 120.

According to one embodiment, the storage system may include a "logical metadata cache" 242, the logical metadata cache 242 storing metadata relating to user data backed up from the host 120 to the storage system 170. As used herein, the term "metadata" refers to information representing user data, and data describing attributes of real user data. The logical metadata cache 242 represents a queryable data set that enables a user and/or software application to randomly locate a backup user file, compare the computer user file to another file, and access and process the backup user file. Two embodiments of software applications that may use the data stored in the logical metadata cache 242 include a synthetic full back-up application 240 and an end-user restore application 300, which will be discussed more fully below.

In general terms, the synthetic full back-up application 240 has the ability to create a synthetic full back-up data set from an existing full back-up data set and one or more incremental back-up data sets. Synthetic full backups may eliminate the need to complete periodic (e.g., weekly) full backups, thus saving considerable time and network resources. Details of the synthetic full back-up application 240 are discussed further below. End user restore application 300, as will be discussed further below, enables an end user (e.g., an operator of user computer 136) to browse, query, read, and/or restore user files previously backed up from storage system 170.

As discussed above, the storage system 170 includes hardware and software for the interface between the host computer 120 and the back-up storage media 126. Hardware and software incorporating embodiments of the present invention may emulate a conventional tape library backup system in which data is apparently backed up to tape from the perspective of the host computer 120, but is actually backed up to another storage medium, such as, for example, a plurality of disk arrays.

Referring to FIG. 3, one embodiment of a storage system 170 in accordance with aspects of the present invention is illustrated in block diagram form. In one embodiment, the hardware of the storage system 170 includes a storage system controller 122 and a switching network 132 that connects the storage system controller 122 to the back-up storage media 126. The storage system controller 122 includes a processor 127 (which may be a single processor or multiple processors) and a memory 129 (e.g., RAM, ROM, PROM, EEPROM, flash memory, etc., or combinations thereof) that may run all or part of the storage system software. The memory 129 may also be used to store and store metadata in the backup storage media 126. Software (including program code to implement embodiments of the present invention) is typically stored in a readable/writable non-volatile storage medium such as RAM, ROM, optical disk, magnetic disk or tape, etc., and then copied into the memory 129 where it is executed by the processor 127. The program code may be written in any of a number of programming languages, such as Java, Visual Basic, C, C #, or C + +, Fortran, Pasca1, Eiffel, Basic, COBAL, and combinations thereof, as the present invention is not limited to a particular programming language. Typically, in operation, the processor 127 causes data (e.g., code that implements an embodiment of the invention) to be read from the non-volatile recording medium into another form of memory, such as RAM, allowing the processor's information to be accessed at a faster rate than the non-volatile recording medium.

As shown in fig. 3, the controller 122 also includes a series of interface adapters that connect the controller 122 and the host 120 to the switching network 132. As illustrated, host 120 is coupled to the storage system through an interface adapter 124a, which interface adapter 124a may be, for example, a fibre channel interface adapter. Through the controller 122 of the storage system, the host computer 120 backs up data to the backup storage media 126 and can restore the data backed up from the backup storage media 126.

In an exemplary embodiment, the switch network 132 may include one or more fibre channel switches 128a, 128 b. The storage system controller 122 includes a plurality of fibre channel interface adapters 124b and 124c to couple the storage system controller to the fibre channel switches 128a, 128 b. Through the fibre channel switches 128a, 128b, the controller 122 of the storage system allows data to be backed up to the backup storage media 126. As illustrated in FIG. 3, the switch network 132 may further include one or more Ethernet switches 130a, 130b, the Ethernet switches 130a, 130b coupled to the controller 122 of the storage system via Ethernet interface adapters 125a, 125 b. In one embodiment, the storage system controller 122 further includes another Ethernet interface adapter 125c that may be coupled to, for example, the LAN103 to facilitate communications between the storage system 170 and a host (e.g., a user computer), as discussed below.

In the embodiment illustrated in FIG. 3, storage system controller 122 is coupled to backup storage media 126 through a switching network that includes two fibre channel switches and two Ethernet switches. Each type of switch provided in at least two storage systems 170 emulates any single point of system failure. In other words, even if one switch (e.g., fibre channel switch 128a) fails, the storage system controller 122 will still be able to communicate with the backup data media 126 through the other switch. The arrangement is advantageous in terms of stability and speed. For example, as discussed above, stability is improved by providing a backup element and eliminating a single point of failure. Moreover, in some embodiments, the storage system controller is able to back up data to the back-up storage media 126 using some or all of the fibre channel switches in parallel, thereby providing a total number of back-ups. However, it is to be appreciated that the system is not required to include two or more switches of each type, nor is the switch network required to include both fibre channel and ethernet switches. Further, in embodiments of the back-up storage media 126 comprising a single disk array, no switch is required at all.

As discussed above, in one embodiment, the backup storage media 126 may include one or more disk arrays. In a preferred embodiment, the backup storage media 126 includes a plurality of ATA or SATA disks. The disks are "for sale" items that are less expensive than conventional storage array items made by manufacturers such as EMC, IBM, etc. Moreover, while the cost of removable media (e.g., tape) is a factor and the life of the media is limited, the disks are comparable in cost to conventional tape-based backup storage systems. In addition, the disk reads/writes data substantially faster than the tape. For example, over a single fiber optic network connection, data may be backed up to disk at a speed of at least about 150MB/s, with 150MB/s transitioning to about 540GB/hr, significantly faster than the tape backup speed (e.g., via disk order). In addition, several fibre channel connections can be performed in parallel, thereby further increasing the speed. According to embodiments of the present invention, the backup storage system media may be aggregated to perform any one of a RAID (redundant array of inexpensive disks) sequence. For example, in one embodiment, the backup storage media may perform RAID-5 tasks.

As discussed above, embodiments of the present invention emulate a conventional tape library backup system by using disk arrays in place of tape cartridges as the actual backup storage media, thus providing a "virtual tape library". Real tape cartridges appear in traditional tape libraries and are replaced by the term "virtual cartridge" as used herein. It is to be appreciated that for purposes of this disclosure, the term "virtual tape library" refers to an emulated tape library that may be implemented in software and/or physical hardware, for example, as one or more disk arrays. It is further to be appreciated that although the discussion primarily refers to emulated tape, the storage system may also emulate other storage media, for example, a CD-ROM or DVD-ROM, and that the term "virtual cartridge" generally refers to emulated storage media, for example, emulated tape or emulated CD. In one embodiment, the virtual cartridges actually correspond to one or more hard disks.

Thus, in one embodiment, a software interface is provided that emulates the tape library to a backup/restore application, looking as if the data were backed up to tape. However, the actual tape library is replaced by one or more disk arrays, so that the data is actually backed up to those disk arrays. It is to be appreciated that other types of removable media restocking systems may be emulated, and that the present invention is not limited to emulation of tape library storage systems. The following discussion will explain the operation of various aspects, features and software included in the storage system 170.

It will be appreciated that although software may be depicted as being "included" in the storage system 170 and possibly being executed by the processor 127 of the storage system controller 122 (see FIG. 3), it is not required that all of the software be executed in the storage system controller 122. Software programs, such as a synthetic full back-up application and an end-user restore application, may run on the host and/or user computers, some of which may be distributed across all or some of the storage system controllers, host and user computers. Thus, it will be appreciated that the storage system controller is not required to comprise a physical entity, such as a computer. The storage system 170 communicates with software stored in hosts such as, for example, the media server 114 and the application server 102. In addition, a storage system may contain several applications that can run and be retained on the same or different hosts. Moreover, it is to be appreciated that the storage system 170 is not limited to discrete pieces of equipment, although in certain embodiments, the storage system 170 may be embodied as discrete pieces of equipment. In one embodiment, the storage system 170 may be provided as a self-contained unit that functions as a "suspend and start" (e.g., modifications that need not be made to existing backup processors and policies) replacement for a conventional tape library backup system. The storage system unit may be used in a networked computing environment including a conventional backup system to provide redundancy or additional storage capability.

As discussed above, according to one embodiment, a host 120 (which may be, for example, an application server 102 or a media server 114 in fig. 1) may back up data to a back-up storage media 126 via a network connection 121 (e.g., a fiber optic connection), the network connection 121 coupling the host 120 to a storage system 170. It will be appreciated that while the following discussion will preferentially refer to the backup of data in an emulated media, the principles also apply to the re-saving of backup data from an emulated media. The flow of data between the host computer 120 and the emulated media 134 may be controlled by the backup/restore application, as discussed above. From the perspective of the backup/restore application, it is apparent that the data is actually backed up into a real version of the emulated media.

Referring to FIG. 4, the storage system software 150 may include one or more logical abstraction layers representing emulated media and providing an interface between the backup/restore application 140 fixed in the host computer 120 and the backup storage media 126. The software 150 accepts tape format data from the backup/restore application 140 and translates the data into data suitable for storage on a random access disk (e.g., hard disk, optical disk, and the like). In one embodiment, the software 150 runs in the processor 127 of the storage system controller 122 and may be stored in the memory 129 (see FIG. 3).

According to one embodiment, the software 150 may include layers, such as the Virtual Tape Library (VTL) layer 142 referred to herein, that may provide emulation of SCSI tapes, tape drives, and robots for converting tapes to and from tape drive formats. The backup/restore application 140 may communicate (e.g., backup or read data into emulated media) with the VTL142 using, for example, SCSI commands (represented by arrow 144). Thus, the VTL may form a software interface between the other storage system software and hardware and the backup/restore application in which the emulated storage system media 134 (FIG. 2) is present, and allow the emulated media to appear as conventional removable backup storage media in the backup/restore application.

A second software layer, referred to herein as the file system layer 146, may provide an interface between the emulated storage medium (represented in the VTL) and the real back-up storage medium 126. In one embodiment, the file system 146 functions to fine tune the communication between the system and the back-up storage media 126, for example, using SCSI commands, represented by arrows 148, to read and write data from the back-up storage media 126.

In one embodiment, the VTL provides general tape library support and may support any SCSI media changer. The emulated tape devices may include, but are not limited to, IBM LTO-1 and LTO-2 tape devices, Quantum SuperDLT320 tape devices, Quantum P3000 tape library systems, or Storage TekL180 tape library systems. In a VTL, each virtual cartridge is a file that grows dynamically as data is stored. This is in contrast to conventional tape cartridges having fixed dimensions. One or more virtual cartridges may be stored in the system file, as described further below with reference to FIG. 5.

In FIG. 5, one example of a data structure in the file system software 146 is illustrated listing a system file 200 in accordance with one embodiment of the present invention. In the depicted embodiment, system file 200 includes header 202 and data 204. The header 202 may include information identifying each virtual cartridge stored in the system file. Regardless of whether the virtual cartridge is write protected, the header may include information such as new/modified data for the virtual cartridge, and the like. In one embodiment, the header 202 includes information that uniquely identifies each of the cartridges and distinguishes each of the cartridges from other virtual cartridges stored in the storage system. For example, the information may include a name and an identification code of the virtual cartridge (corresponding to a code in which a bar code is typically represented on a real cartridge so that the cartridge may be identified by a robotic device). The header 202 may also include additional information such as, for example, the capacity of each virtual cartridge, the most recently modified data, etc.

In accordance with an embodiment of the present invention, the size of the header 202 may be leveraged to reflect the type of data stored (e.g., virtual cartridges representing data backed up from one or more host systems) and the apparent set of such data (e.g., virtual cartridges) that the system may track. For example, data that is typically backed up to a tape storage system is typically characterized by a larger data set, representing digital system and user files. Because the data set is very large, the series of discrete data files being tracked may be relatively small.

Accordingly, in one embodiment, the size of the header 202 is selected based on a compromise between not being able to efficiently keep track of (e.g., the header is too large) due to storing too much data and not having space to store an adequate number of indications of the encoder tape (e.g., the header is too small). In an exemplary embodiment, the header 202 utilizes the first 32MB of the system file 200. However, it will be appreciated that the head 202 may have different sizes based on system requirements and characteristics, and that one may select different sizes for the head 202 depending on the needs and capacity of the system.

It is to be appreciated that from the perspective of the backup/restore application, virtual cartridges with all the same attributes and characteristics appear as real cartridges. In other words, for backup/restore applications, the virtual cartridge appears essentially as a write tape. However, in a preferred embodiment, the data stored in the virtual cartridges is not stored in a sequential format to backup storage media 126, but rather, the data that is ostensibly written to the virtual cartridges is actually stored as randomly accessible, disk-formatted data in a file in the storage system. The metadata is used to link the storage data to the virtual cartridges so that the backup/restore application can read and write in cartridge format.

Thus, in general, from a preferred embodiment, user and/or system data (referred to herein as "file data") is received by the storage system 170 from the host computer 120 and stored in the disk array of the supplemental back-up storage media 126. The software 150 (see FIG. 4) and/or the storage system hardware writes this file data to the back-up storage media 126 in the format of a system file, as described in more detail below. Metadata extracted from the backed-up file data by the storage system controller is used to track that attributes of user and/or system files are backed up. For example, the metadata for each file may include the file name, date of creation or most recent modification of the file, any cryptographically encoded information about the file, and other information. In addition, metadata is created for each file by the storage system that connects the file to the virtual cartridge. Using said metadata, the software provides emulation of the tape cartridge to the host; however, in reality, the file data is not stored in a tape format but in a system file, as described below. Storing data in system files, rather than in a sequential encoder tape format, may be advantageous to allow fast, efficient, and random access to individual files without requiring scanning of sequential data to find a particular file.

As discussed above, according to one embodiment, file data (e.g., user and/or system data) is stored in the backup storage medium as system files, each system file including a header and data, the data being actual user and/or system files. The header 202 of each system file 200 includes a tape directory 206, which tape directory 206 includes metadata that connects user and/or system files to virtual cartridges. The term "metadata" herein refers to neither user nor system file data, but data that describes attributes of actual user and/or system data. According to one embodiment, a tape project may define, down to the byte stage, the layout of data in a virtual cartridge. In one embodiment, the tape directory 206 has a table structure, as shown in FIG. 6. The table includes a column 220 for the type of information stored (e.g., data, file flag FM, etc.), a column 222 indicating the size of the disk bytes used in the bytes, and a column 224 that counts the number of disk bytes storing the file data. Thus, the tape directory allows the controller to randomly access (in reverse order) any data files stored in the back-up storage media 126. For example, referring to FIG. 6, data file 226 may be located quickly in a virtual cartridge because the tape directory indicates that the data of file 226 begins in the block diagram of the initial stage of system file 200. Since there is no size in response to the file marker, one block diagram is depicted. The file marker is not stored in the system, e.g., the file marker corresponds to zero data. The tape directory includes file markers because they are used by conventional tapes and backup/restore applications, and therefore, file markers are written along data files and are intended to be seen when viewing virtual cartridges. Thus, file markers are tracked in the tape directory. However, the file markers do not represent any data and are therefore not stored in the data portion of the system file. Thus, the data of file 226 begins at the beginning of the data portion of the system file, represented by arrow 205 (see FIG. 5), and is 1024 bytes in length (e.g., one disk byte is 1024 bytes). It is to be appreciated that other file data may be stored in bytes other than 1024 bytes, depending on the total amount of data, e.g., the size of the data file. For example, a larger data file may be stored using larger disk bytes for efficiency.

In one embodiment, the tape directory may be included in a "file descriptor" associated with each data file backed up to the storage system. The file descriptor contains metadata associated with the data file 204 stored in the storage system. In one embodiment, the file descriptor may be implemented according to a standard format, for example, the tape archive file format (an extension to a compressed file) used by most UNIX based systems (multi-user computer operating systems). Each file descriptor may include, for example, information of the name of the corresponding user file, data of the new/modified user file, the size of the user file, any access restrictions of the user file, and the like. The additional information stored in the file descriptor may further include information describing a directory structure from which the data may be copied. Thus, the file descriptor may include queryable metadata about the corresponding data file, as discussed in more detail below.

From the perspective of the backup/restore application, any virtual cartridge may include a plurality of file data and corresponding file descriptors. From the perspective of the storage system software, the data files are stored in system files that may be linked, for example, for special backup jobs. For example, a backup executed by a host at a particular time may generate a system file corresponding to one or more virtual cartridges. The virtual cartridges may thus be of any size, and may grow dynamically as more user files are stored in the virtual cartridges.

Referring again to FIG. 2, as described above, the storage system 170 may include a synthetic full back-up software application 240. In one embodiment, the host computer 120 backs up data to the emulated media 134, forming one or more virtual cartridges. In some computing environments, a "full backup," for example, where backup copies of all data are stored in the primary storage system of a network (see FIG. 1), may be performed periodically (e.g., weekly). This process is typically very long, since a large amount of data is copied. Thus, in many computing environments, additional backups, the incremental backups, may be performed between successive full backups (e.g., daily). Incremental backup is a process in which only data is changed since the most current backup is performed (whether incremental or full) is saved. Typically, the changed data is a backup in a repository of files, even though a large portion of the data in the files is not changed frequently. Thus, incremental backups are typically smaller and thus can be completed faster than full backups. It is to be appreciated that while in many environments full backups are typically performed once a week and incremental backups are performed daily throughout the week, the use of a schedule is not required. For example, certain environments may require incremental backups several times a day. The principles of the present invention apply to any environment that uses full backups (and random incremental backups) regardless of the frequency of execution.

During a full backup process, the host computer may create one or more virtual cartridges containing backup data comprising a plurality of data files. For clarity of explanation, the following discussion will assume that a full backup produces only one virtual cartridge. However, it is to be appreciated that a full backup may generate more than one virtual cartridge, and the principles of the present invention may be applied to any number of virtual cartridges.

In accordance with one embodiment, a method is provided for creating a synthetic full back-up data set from an existing full back-up data set and one or more incremental back-up data sets. The method may avoid the need to perform a full backup periodically (e.g., weekly), thereby saving the user a significant amount of time and network resources. Further, as is known to those of ordinary skill in the art, restoring data based on a full backup and one or more incremental backups is a time consuming process because, for example, if the most recent version of a file exists in an incremental backup, the backup/restore application will typically store the file based on the most recent full backup and then apply any changes from the incremental backup. Thus, providing a synthetic full backup may have the additional advantage of allowing a backup storage application to save data files more quickly on a synthetic full backup basis without having to complete multiple saves from a full backup and one or more incremental backups. It is to be appreciated that the term "latest version" as used herein generally refers to the latest copy of a data file (e.g., the data file was last saved time) regardless of whether the file has a new version number. The term "version" as used herein generally refers to a copy of the same file, which may be modified in some manner or may be saved multiple times.

Referring to FIG. 7, a diagrammatic depiction of a synthetic full back-up program is illustrated. The host computer 120 may perform the full backup 230 at a first time, for example, on a weekend. The host computer 120 may perform successive incremental backups 232a, 232b, 232c, 232d, and 232e, for example, each day of the week. The storage system 170 may create a synthetic full back-up data set 234 as described below.

According to one embodiment, the storage system 170 may include a software application, referred to herein as a synthetic full back-up application 240 (see FIG. 3). The synthetic full back-up application 240 may run in the storage system controller 122 (see FIG. 2) or in the host computer 120. The synthetic full back-up application includes software commands and the interfaces necessary to create the synthetic full back-up data set 234. In one embodiment, the synthetic full back-up application may perform a logical merge of the metadata representing each of the full back-up data set 230 and the incremental back-up data set 232 to generate a new virtual cartridge containing the synthetic full back-up data set 234.

For example, referring to FIG. 8, an existing full back-up data set may include user files F1, F2, F3, and F4. The first incremental backup data set 232a may include modified versions of user files F2 ', F2, and F3', F3. The second incremental backup data set 232b may include user files F1', a modified version of F1, and F2 ", a further modified version of F2, and a new user file F5. Thus, the synthetic full back-up data set 234 formed from the logical union of the full back-up data set 230 and the two incremental data sets 232a and 232b includes the latest version of each of the user files F1, F2, F3, F4, and F5. As shown in FIG. 8, the synthetic full back-up data set herein includes user files F1 ', F2 ", F3', F4 and F5.

Referring again to FIGS. 3 and 4, the file system software 146 may create a logical metadata cache 242 storing metadata associated with each user file stored on the emulated media 134. It is to be appreciated that the logical metadata cache is not required to be a real data cache, but may be substituted for the queryable collection data stored in the storage medium 126. In another example, the logical metadata cache 242 may be implemented as a database. The metadata is stored in a database and conventional database commands (e.g., SQL commands) may be used to complete the logical merge of the full back-up data set and one or more incremental back-up data sets to create a synthetic full back-up data set.

As discussed above, each data file stored on the emulated media 134 may include a file descriptor that contains metadata associated with the data file, including the location of the file in the backup storage media 126. In one embodiment, a backup/restore application running in the host computer 120 maintains data in tape stream format in the emulated media 134. An embodiment of the data structure 250 represents the tape format illustrated in fig. 9. As discussed above, the system file data structure includes a header that may contain information about the data file, such as a file descriptor for the data file, new and/or modified file data, security information, a directory structure of the host system from which the file originated, and other information linking the file to the virtual cartridge. The header is associated with data 254, which are the actual user and system files that have been backed up from the host, primary storage system, etc. The system file data structure may also include pads 256 randomly, and the pads 256 may properly align the next header to the region boundary.

As shown in FIG. 9, in one embodiment, the header data is placed in a logical metadata cache 242 to allow fast querying and random access to a continuous tape data format. The use of a logical metadata cache, accomplished using the file system software 146 stored in the system controller 122, allows translation of the linear, continuous tape data format to be stored in the emulated media 134 into a random access data format stored in a physical disk that supplements the back-up storage media 126. The logical metadata cache 242 stores a header 252 that includes a file descriptor for the data file, security information that is used to control access to the data file, as discussed in more detail below, and an indicator 257 for the real location of the data file in the virtual cartridge and backup storage media 126. In one embodiment, the data stored by the logical metadata cache relates to all of the data files of each of the data in the full back-up data set 230 and the incremental data set 232.

According to one embodiment, the synthetic full back-up application 240 uses the information stored in the logical metadata cache to create a synthetic full back-up data set. The synthetic full back-up data set is linked to a virtual cartridge that is newly created by the synthetic full back-up application 240. For the backup/restore application, the synthetic full back-up data set appears to be stored in the synthetic virtual cartridge. As discussed above, the synthetic full back-up data set may be created by performing a logical merge of the existing full back-up data set and the incremental back-up data set. The logical merge may include comparing each data file contained in each of the existing full back-up dataset and the incremental back-up dataset and creating a composite of the most recently modified versions of each user file, as discussed with reference to FIG. 8.

According to one embodiment, synthetic virtual cartridges 260 include indicators that indicate the location of data files in other virtual cartridges, notably including existing full back-up data sets and incremental back-up data sets, as shown in FIG. 10. In view of the previous embodiment shown in FIG. 8, the synthetic virtual cartridge 260 includes an indicator 266 that indicates (indicated by arrow 268) the location of the user file F4 (since the existing full back-up data set includes the latest version of F4) in the existing full back-up data set in virtual cartridge 262 and, for example, the location of the user file F3' in the incremental data set 232a in virtual cartridge 264.

The composite virtual cartridge may also include a list 270 containing the identification codes (and optionally names) of all virtual cartridges that include the data indicated by the indicator 266. The attached cartridge list 270 is important for tracking real data and preventing the attached virtual cartridges from being demagnetized. In the depicted embodiment, the synthetic full back-up data set does not include any actual user files, but the additional set of indicators indicate the locations of the user files in the back-up storage media 126. Therefore, it is desirable to prevent real user files (stored in other virtual cartridges) from being deleted. This may be accomplished, in part, by keeping a record of the virtual cartridges that contain the data (the auxiliary cartridge directory 270), and protecting each of the virtual cartridges from over-writing or deletion. The synthetic virtual cartridges may also include cartridge data 272, such as the size of the synthetic virtual cartridges, the location in the backup storage media 126, and the like. In addition, the resultant virtual cartridge has an identification code and/or name 274.

According to another embodiment, the synthetic virtual cartridge may also include a combination of indicators and actual stored user files. Referring to FIG. 11, in one embodiment, the synthetic virtual cartridge includes an indicator 266, the indicator 266 indicating the location of the data file (latest version, as discussed with reference to FIG. 9) in the existing full back-up data set 230 of the virtual cartridge 262. The synthetic virtual cartridge may also include data 278, data 278 comprising a real data file copied from the incremental data set 232, as indicated by arrow 280. In this manner, after the synthetic full backup data set 276 is newly created, the incremental backup data set may be deleted, thereby conserving storage space. The resultant virtual cartridge is relatively small because it includes all or part of the indicator rather than a copy of all of the user files.

It is to be appreciated that the synthetic full back-up may include any combination of indicators and stored file data and is not limited to the embodiments presented above. For example, a synthetic full backup may include pointers to data files for some files stored in some incremental and/or full backup, and storage of data files copied from other existing full and/or incremental backups. Also, as an alternative, a synthetic full backup may be created based on an existing full backup and any associated incremental backups that do not include any indicators, but include the latest version of the real data files copied from the preferred full and/or incremental backups.

In one embodiment, the synthetic full back-up application may include a differencing operation that is capable of comparing user and system file metadata for each existing full back-up data set and incremental data set to determine the location of the latest version of each data file. For example, the differencing operation may be used to compare new and/or modified data, version numbers (if available), etc., different versions of the same data file in different backup sets to select the most recent version of the data file. However, the user can open the user file and save the file (thereby changing the modified data) without actually changing any data in the file. Thus, the system can perform more complex differencing operations and can analyze the data in the system or user file to determine if the data was indeed modified. The transformation of the differential operation and other comparable algorithm types are well known in the art. Additionally, as discussed above, when metadata is stored in a database format, database commands, such as SQL commands, may be used to perform logical merges. The present invention may employ any of the described differencing operations to ensure that the most recent or most recent version of each user file is selected from the total set of data compared to the existing backup so as to suitably produce a synthetic full backup data set.

Those of ordinary skill in the art will appreciate that synthetic full back-up applications are capable of creating full back-up data sets and are available without the need for the host computer to perform a true full back-up. Not only does it avoid imposing the expense of a host and processor to convert data to a backup storage system, but in embodiments a synthetic full backup application is executed in the storage system, which can significantly reduce network bandwidth utilization. As shown in fig. 7, a further synthetic full back-up data set may be created using the first synthetic full back-up data set 234 and the subsequent incremental back-up data set 236. Among the significant time advantages provided is that files or objects are not modified often, and are copied frequently. In practice, the synthetic full back-up data set may retain an indicator in the file that was just copied.

As discussed with reference to fig. 3, the storage system may include a software application that relates to the end-user restore application 300. Thus, according to another embodiment, a method is provided for an end user to determine and restore backup data without the invention of an IT workgroup and without requiring any changes to existing backup/restore processors and/or policies. In a typical backup storage system, the backup/restore application running on the host computer 120 is controlled by the IT workgroup and IT is impossible or very difficult for an end-user to access the backed-up data without the IT workgroup's invention. In accordance with aspects of embodiments of the present invention, storage system software is provided that allows an end user to locate and restore files through, for example, a network-based or other interface to the back-up storage media 126.

It is to be appreciated that as a result of using the synthetic full back-up application 240, the end-user restore application 300 may run on the controller 122 of the storage system (see FIG. 2) or on the host computer 120. The terminal restore application includes software commands and interfaces necessary to allow authorized users to query the logical metadata cache, randomly restore, and backup files from the backup storage media 126.

According to one embodiment, the software provided includes a user interface installed on the user computer 136 and/or executed on the user computer 136. The user interface may be any type of interface that allows a user to determine the location of files in the backed up storage media. For example, the user interface may be a drawn user interface, may be web-based, or may be a text interface. The user computer is coupled to the storage system 170 through a network connection 138, which network connection 138 may be, for example, an Ethernet connection. An operator of user computer 136 may access data stored in storage system 170 via network connection 138.

In one embodiment, the end-user restore application 300 includes user authentication and/or authorization features. For example, a user may register with a username and password through a user interface on the user's computer. The user computer may communicate the username and password to the storage system (e.g., to the end user recovery application) may use a preferred user authorization means to determine whether the user has accessed the storage system. Some embodiments of the user authorization device may include, but are not limited to, a Microsoft active Directory Server, a Unix "yellow pages" Server, or a standard Directory Access protocol. The registration/user authorization means may communicate with the end-user restore application to exchange the privileges of the user. For example, some users may be allowed to query only files that are created by themselves, or files that possess certain privileges or are identified as owners. Other users, such as system operators or authorized personnel, may access all of the backed-up files, etc.

According to one embodiment, the end-user restore application uses the logical metadata cache to obtain information about all data files backed-up to the back-up storage media. The terminal restore application appears to the user via the user interface with a hierarchical directory structure of the user's file storage, such as backup time/data, user name, initial user computer directory structure (obtained when the file was backed up), or other file characteristics. In one embodiment, the directory structure that appears in front of the user may change depending on the user privileges. The end-user restore application may receive a request to browse (e.g., via a user interface, a user may browse a directory structure to a desired file location) or the user may query the file by name, date, etc.

According to one embodiment, a user may restore a backed-up file from a storage system. For example, once the user has located the desired file, as discussed above, the user may download the file from the storage system via the network connection 138. In one embodiment, the downloading program may be downloaded in contrast to any network-based download, as known to those of ordinary skill in the art.

By allowing end users to access files that allow browsing/downloading, and by facilitating access through a user interface (e.g., web-based termination), the end user restore application may enable users to query and restore their own files without changing any backup policies or procedures.

It is to be appreciated that although aspects of the present invention, such as the synthetic full back-up application and the end-user restore application, are described herein primarily in terms of software, it will be appreciated that software, hardware or firmware, or any combination thereof, may be altered. Thus, for example, embodiments of the present invention may include any computer-readable medium (e.g., computer memory, floppy disk, compact disk, and magnetic tape, etc.) and computer program code (e.g., a plurality of instructions) that, when executed on a processor of a storage system, at least partially executes to perform the functions of a synthetic full back-up application and/or an end-user restore application as described in detail above.

Generally speaking, embodiments and aspects of the present invention thus include storage systems and methods that emulate conventional tape backup systems, but may provide enhanced functionality, for example, enabling the creation of synthetic backups and allowing end users to view and restore backed-up files. However, it is to be appreciated that aspects of the present invention may be used for, rather than, backup of computer data. Because the memory system of the present invention can be used to economically store large amounts of data, and the stored data can be randomly accessed in reverse order at hard disk access times, embodiments of the present invention may find application outside of conventional backup memory systems. For example, embodiments of the present invention may be used to store video or audio data representing more selections of movies and music, and to implement video and/or audio as desired.

Having thus described several aspects of at least one embodiment of this invention, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the scope of the invention. Accordingly, the foregoing description and drawings are by way of example only.

Claims (14)

1. A method comprising the steps of:

creating a complete backup data set comprising a first plurality of data files;

creating at least one incremental backup data set comprising a second plurality of data files;

determining a most recent copy of each of the first and second pluralities of data files;

storing a plurality of indicators, each indicator identifying a storage location of a most recent copy of each of the first and second plurality of data files in one of the full back-up data set and the at least one incremental data set; and

a synthetic full back-up data set is created that corresponds to the most recent copy of each of the first and second plurality of data files based on the plurality of indicators without copying data from the most recent copy of each of the data files.

2. The method of claim 1, further comprising the step of storing metadata associated with each of the first and second pluralities of data files in a searchable metadata cache.

3. The method of claim 2, wherein the step of determining the most recent copy comprises analyzing metadata associated with each of the first and second pluralities of data files to determine the most recent copy of each of the first and second pluralities of data files.

4. The method of claim 2, further comprising the step of providing an interface to allow a user to search a searchable metadata cache to locate a most recent copy of at least one data file in one of the first and second pluralities of data files.

5. The method of claim 4, wherein the step of providing an interface comprises the step of providing a user interface that allows the user to download and restore the most recent copy of the at least one data file.

6. The method of claim 1, further comprising the step of analyzing an existing full back-up data set to provide a first list of each data file of the first plurality of data files.

7. The method of claim 1, further comprising the step of analyzing the at least one incremental backup data set to provide a second list of each data file in the second plurality of data files.

8. A method as claimed in claim 1, wherein the first plurality of data files comprises a first data file, and wherein the second plurality of data files comprises a modified copy of the first data file.

9. A method as claimed in claim 1, wherein the second plurality of data files includes at least one data file whose version is not included in the first plurality of data files.

10. The method of any of claims 1-9, wherein the method comprises creating the synthetic full back-up data set by a computer, wherein the computer comprises a processor, wherein the method is implemented by a program stored on a computer readable medium, and wherein the method is performed by the processor.

11. The method of any of claims 1-9, wherein the method creates the synthetic full back-up data set by a storage system, wherein the storage system comprises a processor, wherein the method is implemented by a program stored on a computer readable medium, and wherein the method is performed by the processor.

12. The method of claim 1, wherein the synthetic full back-up data set is a first synthetic full back-up data set, the method further comprising:

a second synthetic full back-up data set is created corresponding to the first synthetic full back-up data set, the second synthetic full back-up data set including a plurality of first indicators of a plurality of indicators, each first indicator identifying a respective storage location at which a most recent copy of each of the first and second plurality of data files is stored in the full back-up data set and identifying copy data of the most recent copy of each of the data files in the at least one incremental back-up data set.

13. The method of claim 1, wherein the synthetic full back-up data set is a first synthetic full back-up data set, the method further comprising:

a second synthetic full back-up data set is created corresponding to the first synthetic full back-up data set, the second synthetic full back-up data set including copy data of the most recent copy of each of the first and second pluralities of data files stored in the full back-up data set and copy data of the most recent copy of each of the first and second pluralities of data files stored in the at least one incremental back-up data set.

14. A backup storage system comprising:

a backup storage medium for storing a first and second plurality of data files;

a controller comprising at least one processor;

means for creating a full back-up data set including a first plurality of data files;

means for creating at least one incremental backup data set comprising a second plurality of data files;

means for determining a most recent copy of each of the first and second pluralities of data files;

means for storing a plurality of indicators, each indicator identifying a storage location of a most recent copy of each of the first and second plurality of data files in one of the full back-up data set and the at least one incremental data set;

means for creating a synthetic full back-up data set corresponding to the most recent copy of each of the first and second plurality of data files based on the plurality of indicators without copying data from the most recent copy of each of the data files.