CN1185581C - Evolution storage system and its evolution method - Google Patents
Evolution storage system and its evolution method Download PDFInfo
- Publication number
- CN1185581C CN1185581C CNB031190219A CN03119021A CN1185581C CN 1185581 C CN1185581 C CN 1185581C CN B031190219 A CNB031190219 A CN B031190219A CN 03119021 A CN03119021 A CN 03119021A CN 1185581 C CN1185581 C CN 1185581C
- Authority
- CN
- China
- Prior art keywords
- storage
- data
- memory device
- performance
- storage device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明公开了一种进化存储系统及其进化方法。该系统包括存储库和处理器,存储库划分为运行、待机和禁用存储池;处理器设置有:存储服务装置、数据I/O及分布特征分析装置、决策分析装置、存储设备管理装置和用户管理装置。本发明能在进化过程中根据存储系统环境及自身状态的变化,系统自动选择最适合当前环境的系统组织方案,保持系统的动态平衡。系统不会因为存储设备的过时而导致系统不能适应当前应用的需求,整个系统的安装、配置和扩展也非常简单而容易,从而呈现给用户的外在特征是系统越用越好,性能不会随着时间的推移而下降,反而在不断地提升,并具有极高的可用性和良好的可维护性。是海量存储领域中很好的存储管理解决方案。
The invention discloses an evolution storage system and an evolution method thereof. The system includes a storage library and a processor, and the storage library is divided into running, standby and disabled storage pools; the processor is provided with: a storage service device, a data I/O and distribution characteristic analysis device, a decision analysis device, a storage device management device and a user management device. The invention can automatically select the most suitable system organization scheme for the current environment according to the change of the storage system environment and its own state during the evolution process, and maintain the dynamic balance of the system. The system will not be unable to adapt to the needs of current applications due to outdated storage devices, and the installation, configuration, and expansion of the entire system are also very simple and easy, so the external characteristics presented to users are that the system gets better as it is used, and its performance will not Instead of declining over time, it is constantly improving, and has extremely high availability and good maintainability. It is a very good storage management solution in the field of mass storage.
Description
技术领域technical field
本发明属于数据存储技术领域,具体涉及一种进化存储系统及其进化方法,即一种在海量存储环境中管理存储资源的技术。The invention belongs to the technical field of data storage, and in particular relates to an evolutionary storage system and an evolutionary method thereof, that is, a technique for managing storage resources in a mass storage environment.
背景技术Background technique
用于网络环境下的企业级海量存储系统面临如下挑战:数字化信息爆炸性增长、数据的重要性和安全性日益增加、大数量的用户群和多媒体的应用对存取性能产生巨大压力、24×7的服务需求要求极高的可用性和可维护性。迎接这种挑战的主要技术是高性能磁盘阵列、附网存储(NAS,NetworkAttached Storage)、存储区域网(SAN,Storage Area Network)、iSCSI(internet SCSI)和存储管理软件等。The enterprise-level mass storage system used in the network environment faces the following challenges: the explosive growth of digital information, the increasing importance and security of data, a large number of user groups and multimedia applications have huge pressure on access performance, 24×7 The service requirements of the company require extremely high availability and maintainability. The main technologies to meet this challenge are high-performance disk arrays, network attached storage (NAS, NetworkAttached Storage), storage area network (SAN, Storage Area Network), iSCSI (internet SCSI) and storage management software.
虽然上述技术发展日新月异,它还存在以下几个方面的问题:Although the above-mentioned technology is developing rapidly, it still has the following problems:
第一、现有系统有一个普遍存在的矛盾,即元部件的飞速发展和整体结构相对固定的矛盾。以最新的磁盘阵列技术为例,它的构成元部件是磁盘,一旦某个磁盘出故障,它会在备份的新盘上重构丢失的数据。然而,无论新盘在速度上和容量上比其它旧盘高出多少倍,重构后的磁盘阵列总体性能不会有丝毫提高。目前结构的存储系统不会由于更新了部件而得到整体性能的提高。因而,随着技术的进步,这种性能停滞不前的系统很快就会过时,用户不得不购买新的系统来满足应用新的需求;First, there is a common contradiction in the existing system, that is, the contradiction between the rapid development of components and the relatively fixed overall structure. Take the latest disk array technology as an example. Its components are disks. Once a disk fails, it will reconstruct the lost data on a new backup disk. However, no matter how many times the speed and capacity of the new disk are higher than other old disks, the overall performance of the reconfigured disk array will not be improved at all. The storage system with the current structure will not improve the overall performance due to the updated components. Therefore, with the advancement of technology, this system with stagnant performance will soon become obsolete, and users have to purchase new systems to meet the new requirements of applications;
第二、目前数据固定的组织方式使得数据分布不适应变化,系统整体性能呈现一种随使用而退化的趋势。例如,磁盘的性能随着反复改写的次数增多而性能逐步下降,这是众所周知的碎片问题。单机上的磁盘碎片整理十分耗时,整理时不继续使用此磁盘,磁盘阵列上更难使用整理程序;Second, the current fixed data organization method makes the data distribution unadaptable to changes, and the overall performance of the system presents a tendency to degrade with use. For example, the performance of a disk gradually degrades as the number of repeated rewrites increases, which is a well-known fragmentation problem. Disk defragmentation on a single machine is very time-consuming, and the disk will not continue to be used during defragmentation, and it is more difficult to use the defragmentation program on the disk array;
第三、系统结构的组织方式较为固定,不能适应应用的多样性和动态性。例如,磁盘阵列的级别(RAID level)一般设定好后就固定下来,难以适应各种不同性质的应用,如对事务处理设定了RAID5而具有较高的性能,但对流媒体的应用就不大合适。Third, the organization mode of the system structure is relatively fixed, which cannot adapt to the diversity and dynamics of applications. For example, the level of the disk array (RAID level) is generally fixed after being set, and it is difficult to adapt to various applications of different natures. For example, RAID5 is set for transaction processing and has high performance, but it is not suitable for streaming media applications. Great fit.
上述问题存在的原因在于现有存储系统物理和逻辑的组织是一种静态的结构,而静态组织结构模型不能很好地刻画处于不断变化之中的系统。这种结构往往适合于特定的应用需要,而且缺少适应不断变化的存储要求的机制。The reason for the above problems is that the physical and logical organization of the existing storage system is a static structure, and the static organizational structure model cannot well describe the constantly changing system. This structure is often suitable for specific application needs, and lacks a mechanism to adapt to changing storage requirements.
在此之前,已有一些相关技术被提出来解决对存储资源的优化和管理。Prior to this, some related technologies have been proposed to solve the optimization and management of storage resources.
例如,惠普公司开发的AutoRAID技术,见HP whitepaper,《HP AutoRAID:High-Performance Storage for the High-Availability NT Environment》,http://www.hp.com.cn/prodserv/server/pcserver/whitepaper/DOCS/au toraid.pdf,该技术是综合了不同RAID优点的多级RAID阵列技术,它将最近使用的数据放在按RAID 0/1方式存储的快速高性能的磁盘中,将不太常用的数据放在RAID5方式存储的经济高效的磁盘中。AutoRAID技术可以使系统的安装、配置和扩展变得容易,该技术不再需要将数据转移到阵列中的其他磁盘上,只需将新磁盘安装好,AutoRAID就可以自动地判断磁盘大小,并将它加入磁盘阵列中;系统马上就可以利用新磁盘的空间,并将更多的数据按RAID 0/1方式存储,以提高系统的性能和存取速度。此外,AutoRAID能够管理由不同容量磁盘组成的磁盘阵列。动态数据转移是HPAutoRAID的一大特点,它是系统内部的一套控制机构,能够连续监视磁盘阵列的性能,并决定是否将数据保持在缓存中、是否将数据按RAID1存储、是否将数据写入RAID5阵列。这些控制机构能够适应不同的数据情况,并保持系统的高性能。对于工作环境不断变化的场合,这一特点极为有用。For example, for the AutoRAID technology developed by Hewlett-Packard, see HP whitepaper, "HP AutoRAID: High-Performance Storage for the High-Availability NT Environment", http://www.hp.com.cn/prodserv/server/pcserver/whitepaper/ DOCS/au toraid.pdf , this technology is a multi-level RAID array technology that combines the advantages of different RAIDs. It puts the recently used data in the fast and high-performance disks stored in RAID 0/1, and saves less commonly used data. Data is placed on cost-effective disks stored in RAID5. AutoRAID technology can make the installation, configuration and expansion of the system easy. This technology no longer needs to transfer data to other disks in the array. Just install the new disk, and AutoRAID can automatically determine the size of the disk, and It is added to the disk array; the system can immediately use the space of the new disk and store more data in RAID 0/1 mode to improve system performance and access speed. In addition, AutoRAID can manage disk arrays composed of disks of different capacities. Dynamic data transfer is a major feature of HPAutoRAID. It is a set of control mechanisms inside the system that can continuously monitor the performance of the disk array and decide whether to keep the data in the cache, whether to store the data as RAID1, and whether to write the data. RAID5 array. These control mechanisms are able to adapt to different data situations and maintain high performance of the system. This feature is extremely useful in situations where the work environment is constantly changing.
但是,该技术仅仅根据应用对数据使用频度的不同,在数据的动态分布的层次上对存储资源加以优化,因此效果还十分有限。However, this technology only optimizes storage resources at the level of dynamic distribution of data according to the frequency of use of data by applications, so the effect is still very limited.
发明内容Contents of the invention
本发明的目的在于提供一种能克服上述缺陷的进化存储系统,该系统可使存储系统的容量和性能随时间进化,并且易于实现和管理;本发明还提供了一种用于上述系统的进化方法。The purpose of the present invention is to provide an evolutionary storage system that can overcome the above-mentioned defects, the system can make the capacity and performance of the storage system evolve with time, and is easy to implement and manage; the present invention also provides an evolutionary storage system for the above-mentioned system method.
为实现上述发明目的,一种进化存储系统,包括存储库和管理存储库的处理器,所述存储库中的物理存储设备分为运行、待机和禁用三种状态,并划分为运行存储池、待机存储池和禁用存储池;所述处理器设置有:存储服务装置,用于响应计算机节点发出的数据I/O请求,提供数据存储服务,并对I/O数据进行采样,传送给数据I/O及分布特征分析装置;数据I/O及分布特征分析装置用于对I/O数据的使用频率及该数据在存储库中的物理位置分布特征进行统计分析,将分析结果传送给决策分析装置;决策分析装置用于综合从数据I/O及分布特征分析装置传送来的I/O数据的使用频率和分布特征结果,以及从装置传送来的存储设备工作性能结果,在进化规则库中选择进化规则,得到该规则中制定的控制信息,将控制命令发送到存储设备管理装置;存储设备管理装置根据决策分析装置发送过来的操作命令,对存储库中的数据重新进行分布或对存储设备进行再生处理;并对存储设备进行管理,维护存储设备的逻辑设备单元标识与位置号的映射关系表,将存储设备的运行状态信息进行实时采样分析,将性能分析结果反馈给决策分析装置;该系统还包括用户管理装置,用于接收数据I/O及分布特征分析装置输出的系统I/O及分布特征分析信息,及存储设备管理装置输出的各存储设备的运行状态信息,并与决策分析装置及进化规则库相连。In order to achieve the purpose of the above invention, an evolutionary storage system includes a storage library and a processor that manages the storage library. The physical storage devices in the storage library are divided into three states: running, standby and disabled, and are divided into running storage pools, A standby storage pool and a disabled storage pool; the processor is provided with: a storage service device for responding to a data I/O request sent by a computer node, providing a data storage service, and sampling the I/O data, and sending it to the data I /O and distribution feature analysis device; the data I/O and distribution feature analysis device is used to perform statistical analysis on the frequency of use of I/O data and the distribution characteristics of the physical location of the data in the storage library, and transmit the analysis results to decision analysis device; the decision analysis device is used to synthesize the use frequency and distribution characteristic results of the I/O data transmitted from the data I/O and distribution characteristic analysis device, and the storage device work performance result transmitted from the device, in the evolutionary rule base Select the evolution rule, obtain the control information formulated in the rule, and send the control command to the storage device management device; the storage device management device redistributes the data in the storage library or redistributes the storage device according to the operation command sent by the decision analysis device Perform regeneration processing; manage the storage device, maintain the mapping relationship table between the logical device unit identifier and the location number of the storage device, perform real-time sampling and analysis of the operation status information of the storage device, and feed back the performance analysis result to the decision analysis device; The system also includes a user management device for receiving the system I/O and distribution characteristic analysis information output by the data I/O and distribution characteristic analysis device, as well as the operation status information of each storage device output by the storage device management device, and communicates with decision analysis The device and the evolution rule base are connected.
一种用于上述进化存储系统的进化方法,其步骤为:An evolutionary method for the above-mentioned evolutionary storage system, the steps of which are:
(1)系统实时对每个存储设备进行监控;(1) The system monitors each storage device in real time;
(2)判断存储设备的状态:(2) Determine the status of the storage device:
(2.1)当存储设备处于运行态时,检测其运行性能,若性能不符合要求,系统告警,并从待机存储池选取性能和空间满足要求的存储设备提升至运行存储池中,并将原设备上的数据在该存储设备上重构;否则,继续运行使用;(2.1) When the storage device is in the running state, check its operating performance. If the performance does not meet the requirements, the system will alarm, and select the storage device with performance and space that meets the requirements from the standby storage pool to the running storage pool, and replace the original device. The data on the storage device is reconstructed on the storage device; otherwise, continue to run and use;
(2.2)当存储设备处于待机态时,检测其运行性能,若性能低于指定值,转入禁用存储池,否则,根据需要转入运行存储池或保持原有状态;(2.2) When the storage device is in the standby state, detect its operating performance, if the performance is lower than the specified value, transfer to the disabled storage pool, otherwise, transfer to the operating storage pool or maintain the original state as required;
(2.3)当存储设备处于禁用态时,检测其物理性能,若性能不满足要求,用新存储设备替换,否则对存储设备进行格式化处理;再将新的或格式化后的存储设备转入待机存储池。(2.3) When the storage device is in a disabled state, check its physical performance, if the performance does not meet the requirements, replace it with a new storage device, otherwise format the storage device; then transfer the new or formatted storage device into Standby storage pool.
与现有的几种主流的存储技术,如RAID、DAS、NAS、SAN、iSCSI等仅仅从存储设备的物理连接方式和I/O协议着手静态地提高存储性能不同,本发明借鉴生物系统进化和耗散结构原理,提出进化存储系统的设计思想。这是一种能在进化过程中根据存储系统环境及自身状态的变化,系统自动选择最适合当前环境的系统组织方案,保持系统的动态平衡。它不仅使上述问题迎刃而解,由于系统可以根据环境的变化自动调整系统的管理与组织策略,并可以动态的不断加入新型的存储设备,不会因为存储设备的过时而导致系统不能适应当前应用的需求,整个系统的安装、配置和扩展也非常简单而容易,从而呈现给用户的外在特征是系统越用越好,性能不会随着时间的推移而下降,反而在不断地提升,并具有极高的可用性和良好的可维护性。是海量存储领域中很好的存储管理解决方案。Unlike several existing mainstream storage technologies, such as RAID, DAS, NAS, SAN, iSCSI, etc., which only statically improve storage performance from the physical connection mode and I/O protocol of storage devices, the present invention draws lessons from biological system evolution and Based on the principle of dissipative structure, the design idea of evolutionary storage system is proposed. This is a system that can automatically select the most suitable system organization scheme for the current environment according to changes in the storage system environment and its own state during the evolution process to maintain the dynamic balance of the system. It not only solves the above problems, because the system can automatically adjust the management and organization strategies of the system according to changes in the environment, and can dynamically add new storage devices continuously, so that the system will not be unable to adapt to the needs of current applications due to outdated storage devices , the installation, configuration, and expansion of the entire system are also very simple and easy, so the external characteristics presented to users are that the system gets better and better with the use of time, and the performance will not decline over time, but will continue to improve, and has extremely High availability and good maintainability. It is a very good storage management solution in the field of mass storage.
总之,本发明所提出的进化存储方法及其控制系统具有容量和性能随时间进化的特点,又具有极高的可用性和易管理性,是一种新的存储系统概念。它满足了现代企业级存储系统对性能、容量、可用性、可维护性和可扩展性等多方面的要求,具有十分广阔的应用前景。In a word, the evolutionary storage method and its control system proposed by the present invention have the characteristics of capacity and performance evolving with time, and have extremely high availability and easy management, and are a new storage system concept. It meets the performance, capacity, availability, maintainability and scalability requirements of modern enterprise-level storage systems, and has a very broad application prospect.
附图说明Description of drawings
图1为采用本发明方法的一种进化存储系统的结构示意图;Fig. 1 is the structural representation of a kind of evolution storage system adopting the method of the present invention;
图2为进化存储系统的控制系统的结构示意图;2 is a schematic structural diagram of a control system of an evolutionary storage system;
图3为图2中存储设备管理装置的一种具体实现方式的结构示意图;FIG. 3 is a schematic structural diagram of a specific implementation of the storage device management device in FIG. 2;
图4为用于阐述性地显示进化存储系统的存储单元逻辑组织结构的原理图;4 is a schematic diagram for illustratively showing the logical organization structure of storage units of the evolutionary storage system;
图5为当进化存储系统中某一存储设备发生故障,系统自动将该存储设备对应的LDU标识切换到待机存储池中的某一存储设备上的示意图;Fig. 5 is a schematic diagram showing that when a certain storage device in the evolutionary storage system fails, the system automatically switches the LDU identifier corresponding to the storage device to a certain storage device in the standby storage pool;
图6为一种用于进化存储系统的进化方法的流程图。Fig. 6 is a flowchart of an evolution method for evolving a storage system.
具体实施方式Detailed ways
如图1所示,进化存储系统中包括存储库2、处理器3、交换机5和桥6。处理器3可以是单个处理器,SMP(Symmetric Multiple Processors,对称多处理器),MPP(Massively Parallel Processors,并行处理系统),群集或者其它网络计算装置。处理器3可以被多个注册装置所支持,如在进化存储系统中注册的计算机节点1所支持,计算机节点1可以是单个处理器,SMP,MPP,群集或者其它网络计算装置。存储库2包括多个物理存储设备4,如果存储设备支持SCSI接口,则每个物理存储设备4与一个SCSI(Small Computer Systems Interface)到FC(Fibre Channel,光纤通道)的桥6连接。由于SCSI通道数据传输速率与联接的存储设备个数有限,存储设备也可以采用FC接口,如果物理存储设备采用FC接口,桥6可以为光纤通道交换机。如果存储库2中物理存储设备的个数超过单个SCSI通道或者过光纤通道所能支持的范围,可以将SCSI通道或者光纤通道采用如图1所示的两级级联方式联接,光纤通道交换机5,也可以根据需要采用多级方式联接以扩充存储库2。将存储库2的物理存储设备指定为逻辑设备单元(LDU)。每一个物理存储设备4的LDU标识由该物理存储设备所在的SCSI通道(或FC通道)标识和该设备在SCSI通道(或FC通道)上的位置号共同确定,并在存储库2中唯一,例如位置号(1,1)可以标识在第一个通道上的第一个物理存储设备,该存储设备的LDU标识可能为LDU_20。存储设备4的LDU标识与位置号的映射关系表由系统维护,并可以动态更新。图1中的LDU由处理器3控制,并对其统一编址。整个存储系统的存储资源由处理器3进行管理。As shown in FIG. 1 , an evolutionary storage system includes a
在存储库2中,所有存储设备处于三种状态:运行,待机和禁用。In
在工作中,处理器3,通过采用SCSI协议的SCSI总线,或者通过采用光纤通道协议的光纤,与存储库2连通,并将从处理器3接受到的数据传送到存储库2处,再将数据从存储库2中传送出来送给处理器3。In work, the
处理器3对存储库2中的物理存储设备4的工作性能进行实时监控,根据结果决定存储设备在存储库中的运行状态。同时对存储库2的输入输出数据进行特征分析,以决定数据在物理存储设备上的位置分布。The
如图2所示,在工作中,计算机节点1传入的数据经过存储服务装置10传送到存储库2中,并通过该装置接收从存储库2中返回的数据,输出到计算机节点1,对外界提供可靠的存储服务。同时存储服务装置10对I/O数据进行采样,将采样数据传给数据I/O及分布特征分析装置9,由该装置对I/O数据的使用频率及该数据在存储库中的物理位置分布特征进行统计分析,将分析结果传送给决策分析装置8。存储设备管理装置7负责对存储库2中的存储设备进行管理,维护存储设备4的LDU标识与位置号的映射关系表,并完成决策分析装置8发送过来的操作命令。存储设备管理装置7还对存储库2中的物理存储设备4的运行状态信息进行实时采样分析,并将性能分析结果传送给决策分析装置8。As shown in Figure 2, during work, the data imported by
决策分析装置8综合从数据I/O及分布特征分析装置9传送来的I/O数据的使用频率和分布特征结果,与从存储设备管理装置7传送来的存储设备工作性能结果,在进化规则库11中选择适合当前运行状况的进化规则,得到该规则中制定的控制信息,将控制命令发送到存储设备管理装置7,由存储设备管理装置7对存储库中的数据重新进行分布或对存储设备进行再生处理。如果进化规则库11中有多条适用的进化规则,则让这几条规则进行竞争,即发送每条规则的测试数据到存储库中,根据返回的性能测试结果选择最佳的规则来优化系统。以自动优化存储系统工作性能,实现系统自身的进化功能。The decision analysis device 8 synthesizes the use frequency and distribution characteristic results of the I/O data transmitted from the data I/O and distribution characteristic analysis device 9, and the storage device work performance result transmitted from the storage
在工作过程中,数据I/O及分布特征分析装置9将系统I/O及分布特征分析结果实时传送到用户管理装置12,以便用户对系统的数据I/O特征进行监控。同样存储设备管理装置7将收集到的存储库2中的物理存储设备4的运行状态信息实时传送到用户管理装置12。用户可以根据收集到的所有系统信息对系统运行状况做出分析,可以根据需要直接发送系统控制命令到决策分析装置8,以便对系统进行性能优化。例如可以手工将某个使用率较高的应用的数据统一迁移到性能较高的物理存储设备上。During the working process, the data I/O and distribution characteristic analysis device 9 transmits the system I/O and distribution characteristic analysis results to the user management device 12 in real time, so that the user can monitor the data I/O characteristics of the system. Similarly, the storage
在工作过程中,由于系统会根据自身状况和外界的数据访问特征动态调整数据的分布策略,因此决策分析装置8会实时将存储库2中的数据分布信息传给存储服务装置10,以便存储服务装置能够对外界提供可靠的存储服务。During the working process, because the system will dynamically adjust the data distribution strategy according to its own conditions and external data access characteristics, the decision analysis device 8 will transmit the data distribution information in the
管理员在进化存储系统运行过程中,可以通过用户管理装置12对进化规则库11进行维护。根据存储系统的运行状态,如果发现需要添加新的进化规则,则可以通过用户管理装置12加入新的进化规则,添加到进化规则库11中。同样也可以修改或删除不适用的进化规则。The administrator can maintain the evolution rule base 11 through the user management device 12 during the operation of the evolution storage system. According to the running state of the storage system, if it is found that a new evolutionary rule needs to be added, the new evolutionary rule can be added to the evolutionary rule base 11 through the user management device 12 . Evolution rules that do not apply can also be modified or deleted as well.
图2中各装置均设置在处理器中,但用户管理装置12也可以设置在计算机节点上。Each device in FIG. 2 is set in the processor, but the user management device 12 can also be set on the computer node.
如图3所示,存储设备管理装置7包括存储设备管理控制子装置71、存储设备性能监控子装置72、LDU管理子装置73、存储设备维护子装置74。As shown in FIG. 3 , the storage
存储设备管理控制子装置71的作用是接受决策分析装置8发出的命令,发送相应的控制信息到对应的装置,完成存储设备的管理功能。并将存储库2中的物理存储设备4的状态信息分别反馈到用户管理装置12和决策分析装置8。The function of the storage device management and control sub-device 71 is to accept the command issued by the decision analysis device 8, send corresponding control information to the corresponding device, and complete the management function of the storage device. And the status information of the
存储设备性能监控子装置72对存储库2中的存储设备的性能(如数据吞吐率,平均响应时间,读写正确性)进行实时监控,并将结果传送给存储设备管理控制子装置71,由存储设备管理控制子装置71分别将结果传送给用户管理装置12和决策分析装置8。存储设备管理控制子装置71可根据运行情况决定是否向用户管理装置12发送告警信息。在工作过程中,存储设备管理控制子装置71可接收决策分析装置发出的命令,发送控制命令到性能监控子装置72,调整性能监控策略。The storage device performance monitoring sub-means 72 monitors the performance of the storage devices in the storage library 2 (such as data throughput, average response time, read and write accuracy), and sends the results to the storage device management control sub-means 71, by The storage device management control sub-unit 71 transmits the results to the user management unit 12 and the decision analysis unit 8 respectively. The storage device management control sub-device 71 can decide whether to send alarm information to the user management device 12 according to the operation situation. During the working process, the storage device management control sub-device 71 can receive the commands sent by the decision analysis device, send control commands to the performance monitoring sub-device 72, and adjust the performance monitoring strategy.
LDU管理子装置73接收存储设备管理控制子装置71传来的LDU动态配置信息,对存储设备4的LDU标识与位置号的映射关系表进行维护,并将相应信息通过决策分析装置8传到存储服务装置10。The LDU management sub-device 73 receives the LDU dynamic configuration information transmitted from the storage device management control sub-device 71, maintains the mapping relationship table between the LDU identifier and the location number of the
存储设备维护子装置74接收存储设备控制子装置71发出的存储设备维护命令,对存储库2中的存储设备进行维护,如格式化处理,文件碎片整理等等。The storage device maintenance sub-unit 74 receives the storage device maintenance command issued by the storage device control sub-unit 71, and performs maintenance on the storage device in the
图4表示进化存储系统中存储库的逻辑结构图。在存储库2中,所有存储设备处于三种状态:运行,待机和禁用。运行存储池21中为处于正常运行状态中的存储设备41,是系统中提供数据存储服务的存储空间,待机存储池22是没有存储数据并且等待使用的存储设备一即待机存储设备42的集合;禁用存储池23是已淘汰存储设备一即禁用的存储设备43的集合。Fig. 4 shows a logical structure diagram of a repository in an evolutionary storage system. In
在图4所示实施例中,系统对存储池中的每个存储设备进行检测,根据状态来决定存储设备不同状态的变迁,实现系统物理结构的进化。例如,当监控程序发现处于待机状态的存储设备42的物理性能参数低于设定值时或不能工作时,将该存储设备作禁用处理,并将该存储设备的状态改为禁用,转入禁用存储池23,等待人工或机械更换;待机存储池22中的处于待机态的存储设备42的物理性能大于或等于设定值时,如果系统需要该盘提供存储服务,则将该存储设备的状态改为运行,转入运行存储池。同样,监控程序可对处于运行状态的存储设备41进行监测,如物理性能不能满足要求则同样将有用数据转移,将存储设备转入禁用存储池23。禁用存储池中的存储设备43在经过更换或格式化后,若物理性能满足要求,则可向待机状态或运行状态迁移。通过存储系统中各个存储设备的类似新陈代谢的状态迁移,不断将不满足要求的物理存储设备进行更新或者替换,同时加入新的满足要求的物理设备,提高了整个系统的运行性能和稳定性,使整个系统始终保持先进,实现了系统物理结构进化。In the embodiment shown in FIG. 4 , the system detects each storage device in the storage pool, determines the transition of different states of the storage device according to the state, and realizes the evolution of the physical structure of the system. For example, when the monitoring program finds that the physical performance parameter of the storage device 42 in the standby state is lower than the set value or when it cannot work, the storage device will be disabled, and the state of the storage device will be changed into disabled and turned into disabled. The storage pool 23 is waiting for manual or mechanical replacement; when the physical performance of the storage device 42 in the standby state in the standby storage pool 22 is greater than or equal to the set value, if the system requires the disk to provide storage services, the state of the storage device will be Running instead, go to running storage pools. Similarly, the monitoring program can monitor the storage device 41 in the running state. If the physical performance cannot meet the requirements, the useful data will also be transferred, and the storage device will be transferred to the disabled storage pool 23 . After the storage device 43 in the disabled storage pool is replaced or formatted, if the physical performance meets the requirements, it can migrate to the standby state or the running state. Through the state transition similar to metabolism of each storage device in the storage system, the physical storage devices that do not meet the requirements are continuously updated or replaced, and new physical devices that meet the requirements are added at the same time, which improves the operation performance and stability of the entire system, and enables The whole system has always been advanced, realizing the evolution of the physical structure of the system.
运行池中的存储设备按照存储设备簇的方式存放。存储设备簇是系统中基本的逻辑存储单位,提供一个相对稳定和线性的存储空间,它是由一个或者多个同构的物理存储设备的集合。同一存储设备簇中的存储设备具有相近的存储容量和物理存储性能。不同存储设备簇可以有不同的存储参数,如数据块大小。存储设备簇中的存储设备在物理上不一定相邻。每个存储设备簇中存储设备的数量可以动态改变,最少可以是一个。在保证可靠的存储空间的基础上,存储设备簇中的存储设备可以被另一个性能相近的存储设备替换。当存储设备簇中的存储设备数量变化或发生存储设备替换时,系统自动重构数据。存储设备簇以数据块的形式提供数据存取。The storage devices in the running pool are stored in the form of storage device clusters. A storage device cluster is the basic logical storage unit in the system and provides a relatively stable and linear storage space. It is a collection of one or more homogeneous physical storage devices. Storage devices in the same storage device cluster have similar storage capacity and physical storage performance. Different storage device clusters can have different storage parameters, such as data block size. Storage devices in a storage device cluster are not necessarily physically adjacent. The number of storage devices in each storage device cluster can be changed dynamically, at least one. On the basis of ensuring a reliable storage space, a storage device in a storage device cluster can be replaced by another storage device with similar performance. When the number of storage devices in a storage device cluster changes or a storage device is replaced, the system automatically reconstructs data. A cluster of storage devices provides data access in the form of data blocks.
例如:在图4中,处于运行态的逻辑设备单元LDU_1与LDU_2、LDU_3、LDU_4、LDU_5一起组成RAID5,其中LDU_1的位置号为(2,3),当系统发现LDU_1的工作性能不满足系统要求,并不需要向通常的存储系统那样,将该位置的存储设备取出,换入新的存储设备。而只需要存储设备管理装置7中查询系统中所有存储设备的性能表,比如发现处于位置号(1,5)的存储设备满足系统的要求,则在保证数据一致性的前提下,将位于(2,3)位置的存储设备上的数据迁移或者重构到位置号位(1,5)的存储设备上,并将该设备标记为LDU_1,状态改为运行态,同时更新LDU与位置号的映射关系表。这样就保证了存储系统的可靠性。另外,将(2,3)位置的存储设备状态标记为禁用态,转入禁用存储池。如图5所示。For example: in Figure 4, the logical device unit LDU_1 in the running state forms RAID5 together with LDU_2, LDU_3, LDU_4, and LDU_5, and the position number of LDU_1 is (2, 3). When the system finds that the working performance of LDU_1 does not meet the system requirements , it is not necessary to take out the storage device at the location and replace it with a new storage device as in the usual storage system. And only need to query the performance tables of all storage devices in the system in the storage
图6描述的是一种可用于上述存储系统的进化方法的流程图,即用于进化规则库的一套进化策略。该方法根据状态来决定存储设备不同状态的变迁,实现系统物理结构的进化。如步骤102所示,进化存储系统实施对存储库中的每个存储设备轮流进行监测。在步骤104中,系统对存储设备的运行状态进行判断,如果状态为运行态,转入步骤106;状态为待机态,转入步骤108;状态为禁用态,转入步骤109。FIG. 6 depicts a flow chart of an evolutionary method that can be used in the above-mentioned storage system, that is, a set of evolutionary strategies for the evolutionary rule base. The method determines the transition of different states of the storage device according to the state, and realizes the evolution of the physical structure of the system. As shown in step 102, the evolutionary storage system monitors each storage device in the storage library in turn. In step 104, the system judges the running state of the storage device, if the state is running, go to step 106; if the state is standby, go to step 108; if the state is disabled, go to step 109.
在步骤106中,系统对处于运行状态的存储设备检测运行性能(如数据吞吐率,平均响应时间,读写正确性),如果发现存储设备运行时的工作性能低于指标(例如存储设备出现故障,I/O性能低于阈值或由于设备中的文件碎片过多造成性能低于运行性能指标),则执行步骤110;如果运行正常,即工作性能符合系统要求,则转到步骤102,对下一个存储设备进行监测。在步骤111中,存储系统实时告警,并对照待机存储池中所有存储设备的性能表,选择待机存储池中性能和空间满足要求的存储设备,提升至运行存储池中,然后将问题设备上的数据在新提升的存储设备上重构或迁移,以便从新设备上读取数据,保证数据访问的可靠性。在完成步骤111后,转入步骤110,将问题存储设备装入禁用存储池。In step 106, the system detects the operating performance (such as data throughput rate, average response time, and read and write correctness) of the storage device in the running state. , the I/O performance is lower than the threshold or the performance is lower than the running performance index due to too many file fragments in the device), then perform step 110; if it runs normally, that is, the working performance meets the system requirements, then go to step 102, and perform the following A storage device is monitored. In step 111, the storage system sends an alarm in real time, and compares the performance tables of all storage devices in the standby storage pool, selects a storage device whose performance and space meet the requirements in the standby storage pool, promotes it to the running storage pool, and then removes the storage device on the problematic device The data is reconstructed or migrated on the newly upgraded storage device, so that the data can be read from the new device to ensure the reliability of data access. After step 111 is completed, go to step 110 and load the problematic storage device into the disabled storage pool.
在步骤108中,系统对处于待机态的存储设备进行工作性能分析(如检测数据吞吐率,平均响应时间,读写正确性),如果存储设备的性能高于或者等于指定值,执行步骤112;如果性能低于指定值,则执行步骤114。在步骤112中,系统如发现需要加入新的运行存储设备来满足数据存储的要求,则将该存储设备在系统中的运行状态标记为运行态,转入运行存储池,以存储新增的数据。在步骤114中,系统将性能不满足要求的待机存储设备转入禁用存储池。In step 108, the system performs work performance analysis (such as detecting data throughput, average response time, read and write correctness) to the storage device in standby state, if the performance of the storage device is higher than or equal to a specified value, step 112 is performed; If the performance is lower than the specified value, step 114 is performed. In step 112, if the system finds that a new running storage device needs to be added to meet the requirements of data storage, the running state of the storage device in the system will be marked as running, and transferred to the running storage pool to store the newly added data . In step 114, the system transfers the standby storage device whose performance does not meet the requirements into the disabled storage pool.
在步骤109中,系统进一步对存储设备的物理性能进行分析,如检测控制器负载(注:当一个磁盘驱动器从主机收到一个读或写请求时,磁盘控制器必须花费一定的时间用于和总线之间传送数据以及对请求的命令进行译码解释,此过程称之为控制器负载)、寻道时间、数据读/写时间、存储设备的0柱面损坏、存储设备中失效的块的百分比等等,如果发现存储设备的物理性能不满足系统的要求,则执行步骤116,由系统自动或由管理员手工取出该设备,换入新的物理性能满足要求的存储设备,然后执行步骤118,将新设备转入待机存储池。如果存储设备的物理性能满足系统的要求,则对该存储设备做格式化处理,转入待机存储池,更新存储设备性能表,如步骤120所示。In step 109, the system further analyzes the physical performance of the storage device, such as detecting the controller load (note: when a disk drive receives a read or write request from the host, the disk controller must spend a certain amount of time for and Transferring data between buses and decoding and interpreting requested commands, this process is called controller load), seek time, data read/write time, zero cylinder damage of storage devices, failure of blocks in storage devices If it is found that the physical performance of the storage device does not meet the requirements of the system, then step 116 is performed, and the system automatically or manually takes out the device by the administrator, and replaces it with a new storage device whose physical performance meets the requirements, and then performs step 118 to transfer the new device to the standby storage pool. If the physical performance of the storage device meets the requirements of the system, the storage device is formatted, transferred to the standby storage pool, and the performance table of the storage device is updated, as shown in step 120 .
本发明也考虑了上述实施例的各种变化。例如,图1中的存储设备4的接口也可以是串行ATA、ST506/412、IDE、EIDE、SSA、1394、USB等接口。The present invention also contemplates various variations of the above-described embodiments. For example, the interface of the
尽管已经详细描述了本发明公开的实施例,但应理解在不脱离其构思和范围的情况下,本发明的实施例还有各种变化,替代和修改。Although the disclosed embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions and alterations could be made to the embodiments of the present invention without departing from the spirit and scope thereof.
Claims (3)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB031190219A CN1185581C (en) | 2003-04-30 | 2003-04-30 | Evolution storage system and its evolution method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB031190219A CN1185581C (en) | 2003-04-30 | 2003-04-30 | Evolution storage system and its evolution method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1445675A CN1445675A (en) | 2003-10-01 |
| CN1185581C true CN1185581C (en) | 2005-01-19 |
Family
ID=27815003
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB031190219A Expired - Fee Related CN1185581C (en) | 2003-04-30 | 2003-04-30 | Evolution storage system and its evolution method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN1185581C (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4227035B2 (en) * | 2004-02-03 | 2009-02-18 | 株式会社日立製作所 | Computer system, management device, storage device, and computer device |
| JP2005267008A (en) * | 2004-03-17 | 2005-09-29 | Hitachi Ltd | Storage management method and storage management system |
| US7493300B2 (en) * | 2004-09-30 | 2009-02-17 | International Business Machines Corporation | Model and system for reasoning with N-step lookahead in policy-based system management |
| US9542116B2 (en) | 2008-10-23 | 2017-01-10 | Hewlett-Packard Development Company, L.P. | Direct attached storage system and method for implementing multiple simultaneous storage schemes |
| CN101916211B (en) * | 2010-07-16 | 2012-06-27 | 中国人民解放军国防科学技术大学 | Feedback-based dynamic hierarchical data storage equipment reliability ensuring method |
-
2003
- 2003-04-30 CN CNB031190219A patent/CN1185581C/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN1445675A (en) | 2003-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10346360B1 (en) | Managing prefetching of data in storage systems | |
| US7181578B1 (en) | Method and apparatus for efficient scalable storage management | |
| US8131927B2 (en) | Fast accessible compressed thin provisioning volume | |
| US9454533B2 (en) | Reducing metadata in a write-anywhere storage system | |
| US8566549B1 (en) | Synchronizing performance requirements across multiple storage platforms | |
| CA2780220C (en) | Dynamic management of destage tasks in a storage controller | |
| US8214606B2 (en) | Storage system and control method thereof | |
| US20090210620A1 (en) | Method to handle demand based dynamic cache allocation between SSD and RAID cache | |
| US9323655B1 (en) | Location of data among storage tiers | |
| US10621057B2 (en) | Intelligent redundant array of independent disks with resilvering beyond bandwidth of a single drive | |
| US8966170B2 (en) | Elastic cache of redundant cache data | |
| US9063945B2 (en) | Apparatus and method to copy data | |
| US9965381B1 (en) | Indentifying data for placement in a storage system | |
| US11314431B2 (en) | Distributed data blocks using storage path cost values | |
| CN1545030A (en) | Method of Dynamic Mapping of Data Distribution Based on Disk Characteristics | |
| US10705853B2 (en) | Methods, systems, and computer-readable media for boot acceleration in a data storage system by consolidating client-specific boot data in a consolidated boot volume | |
| US10572464B2 (en) | Predictable allocation latency in fragmented log structured file systems | |
| Zhao et al. | Chameleon: An adaptive wear balancer for flash clusters | |
| US11592988B2 (en) | Utilizing a hybrid tier which mixes solid state device storage and hard disk drive storage | |
| CN1185581C (en) | Evolution storage system and its evolution method | |
| CN120216455B (en) | Smart storage integrated memory storage architecture | |
| US10838639B2 (en) | Intelligent redundant array of independent disks | |
| US11422702B2 (en) | Managing utilization of storage class memory (SCM) resources | |
| CN1225698C (en) | Regenerative evolution method for data on storage device | |
| US20070214313A1 (en) | Apparatus, system, and method for concurrent RAID array relocation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20050119 Termination date: 20200430 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |