CN103384550A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN103384550A CN103384550A CN2012800036568A CN201280003656A CN103384550A CN 103384550 A CN103384550 A CN 103384550A CN 2012800036568 A CN2012800036568 A CN 2012800036568A CN 201280003656 A CN201280003656 A CN 201280003656A CN 103384550 A CN103384550 A CN 103384550A
- Authority
- CN
- China
- Prior art keywords
- section
- data
- child resource
- resource pond
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2094—Redundant storage or storage space
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Storage Device Security (AREA)
Abstract
Description
技术领域technical field
本发明涉及电子信息技术领域,尤其涉及一种储存数据的方法及装置。The invention relates to the technical field of electronic information, in particular to a method and device for storing data.
背景技术Background technique
分布式存储技术是一种常用的数据存储技术,在分布式存储场景下,可以将系统中所有的硬盘作为一个存储资源池进行统一的管理,在系统对数据进行读写和存储时都是将整个存储资源池作为一个存储区域进行数据交互的。例如:Distributed storage technology is a commonly used data storage technology. In the distributed storage scenario, all hard disks in the system can be managed as a storage resource pool in a unified manner. When the system reads, writes and stores data, it uses the The entire storage resource pool is used as a storage area for data interaction. For example:
分布式存储系统可以将一个虚拟卷映射到存储资源池中的全部硬盘上,并会卷上的空间划分为若干个1M大小的数据块,每个数据块对应资源池中的全部硬盘中的一个块存放逻辑分区单元Partition。当Partition有多个副本时,一个Partition内的全部数据会被保存多份,比如:在双副本的场景下,每个数据块对应资源池中的全部硬盘中的二个Partition,并且这2个Partition的内容完全相同。在根据常用的分配算法,相同的二个Partition如:P1和P1’,往往会被分配在二个不同的硬盘上如:P1在硬盘1,P1’在硬盘2,使Partition可以被平均地分配在整个资源池中,由于这种分配Partition的方式能够使资源池中硬盘的读写速度最大化,从而被广泛地采用。The distributed storage system can map a virtual volume to all hard disks in the storage resource pool, and divide the space on the volume into several 1M data blocks, and each data block corresponds to one of all the hard disks in the resource pool. The block stores the logical partition unit Partition. When a Partition has multiple copies, all data in a Partition will be saved in multiple copies. For example, in a double-copy scenario, each data block corresponds to two Partitions in all hard disks in the resource pool, and the two The content of Partition is exactly the same. According to the commonly used allocation algorithm, the same two Partitions, such as: P1 and P1', are often allocated on two different hard disks, such as: P1 is on hard disk 1, and P1' is on hard disk 2, so that the Partition can be evenly allocated In the entire resource pool, because this method of allocating Partitions can maximize the read and write speed of hard disks in the resource pool, it is widely used.
当分布式存储系统中任意一个硬盘损坏时,可以进行数据重建,例如:硬盘1损坏,由于硬盘1上的P1、P4’、P5、P8’在硬盘2、3、4上都有对应的副本,则可以通过硬盘2、3、4上都对应的副本修复硬盘1的数据。When any hard disk in the distributed storage system is damaged, data reconstruction can be performed. For example, if hard disk 1 is damaged, P1, P4', P5, and P8' on hard disk 1 have corresponding copies on hard disks 2, 3, and 4. , then the data on hard disk 1 can be restored through the corresponding copies on hard disks 2, 3, and 4.
在现有的分布式存储系统中任意一个硬盘损坏,会进行数据重建,在进行数据重建的过程中,若存储资源池中其他硬盘再故障,则整个虚拟卷的数据就会损坏,例如:如图1所示,在硬盘1损坏并开始进行数据修复时,若硬盘2在损坏,则储存在硬盘2上的用于修复硬盘1的对应的副本P1’就会损坏,从而无法修复数据,导致整个虚拟卷的数据损坏,降低了数据的安全性。In the existing distributed storage system, if any hard disk is damaged, data reconstruction will be performed. During the data reconstruction process, if other hard disks in the storage resource pool fail again, the data of the entire virtual volume will be damaged. For example: As shown in Figure 1, when the hard disk 1 is damaged and the data recovery starts, if the hard disk 2 is damaged, the corresponding copy P1' for repairing the hard disk 1 stored on the hard disk 2 will be damaged, so that the data cannot be repaired, resulting in Data corruption of the entire virtual volume reduces data security.
发明内容Contents of the invention
本发明的实施例提供一种储存数据的方法及装置,能够将存储系统中的所有的存储设备划分为至少二个资源池,并将数据储存在所划分出的资源池中,由于各个资源池中的数据是独立的,只有同一个资源池中的2个存储设备损坏,数据才会完全损坏,从而降低了数据完全损坏的几率,提高了数据的安全性。Embodiments of the present invention provide a method and device for storing data, which can divide all storage devices in the storage system into at least two resource pools, and store data in the divided resource pools. Since each resource pool The data in the resource pool is independent, and the data will be completely damaged only if two storage devices in the same resource pool are damaged, thereby reducing the probability of complete data damage and improving data security.
为达到上述目的,本发明的实施例采用如下技术方案:In order to achieve the above object, embodiments of the present invention adopt the following technical solutions:
第一方面,本发明的实施例提供了一种储存数据的方法,用于存储系统,所述存储系统中包括用于存储数据的资源池,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述方法包括:In the first aspect, an embodiment of the present invention provides a method for storing data, which is used in a storage system. The storage system includes a resource pool for storing data, and the resource pool is logically divided into at least two sub-resource pools. , each sub-resource pool includes adjacent N storage devices, N≥2, and the method includes:
将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中,所述需要存储的数据期望生成的副本数量小于N,R≥2;dividing the data to be stored into R slices, and distributing the R slices to the divided sub-resource pools, the number of copies of the data to be stored is expected to be less than N, and R≥2;
对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。Generate a copy slice for each slice distributed to each sub-resource pool, and store the copy slice in the sub-resource pool to which the slice belongs, and the copy slice corresponds to the slice with the same data .
结合第一方面,在第一方面的第一种可能的实现方式中,所述将所述副本存储在对应所述切片所属的子资源池中包括:将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上,且每个所述副本切片存储的存储设备与对应所述切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。With reference to the first aspect, in a first possible implementation manner of the first aspect, storing the copy in the sub-resource pool corresponding to the slice includes: storing the copy slice in a sub-resource pool corresponding to the slice On the storage device in the sub-resource pool to which each copy slice is stored is different from the storage device corresponding to the slice storage, and copy slices with the same data are stored on different storage devices.
结合第一方面,以及第一方面的第一种可能的实现方式,在第一方面的第二种可能的实现方式中,还包括:将所述划分的子资源池再次划分为不同级别子资源池集合,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同;In combination with the first aspect and the first possible implementation of the first aspect, the second possible implementation of the first aspect further includes: dividing the divided sub-resource pool into sub-resources of different levels A pool set, each sub-resource pool set includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in different levels of sub-resource pool sets is different;
所述对分布到每个子资源池中的每个所述切片生成副本切片,包括:按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。The generating a copy slice for each of the slices distributed to each sub-resource pool includes: according to the number of copy slices expected to be generated for each of the slices corresponding to the level of the sub-resource pool set to which the sub-resource pool belongs, the distribution A corresponding number of duplicate slices are generated for each of the slices in the sub-resource pool.
结合第一方面,以及第一方面的第一种可能的实现方式,在第一方面的第三种可能的实现方式中,在将所述副本切片存储在对应所述切片所属的子资源池中后,还包括:With reference to the first aspect and the first possible implementation of the first aspect, in the third possible implementation of the first aspect, the copy slice is stored in the sub-resource pool to which the slice belongs After that, also include:
针对任一个所述子资源池,当减少所述子资源池中的存储设备的数量时,将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上,所述子资源池中的剩余的存储设备数量大于所述期望生成的副本数。For any of the sub-resource pools, when reducing the number of storage devices in the sub-resource pool, allocate the slices in the reduced storage devices to other storage devices in the sub-resource pool, so The number of remaining storage devices in the sub-resource pool is greater than the number of copies expected to be generated.
结合第一方面,以及第一方面的第一种可能的实现方式,在第一方面的第四种可能的实现方式中,在将所述副本切片存储在对应所述切片所属的子资源池中后,还包括:With reference to the first aspect and the first possible implementation of the first aspect, in the fourth possible implementation of the first aspect, the copy slice is stored in the sub-resource pool to which the slice belongs After that, also include:
针对任一个所述子资源池,当增加所述子资源池中的存储设备的数量时,将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。For any one of the sub-resource pools, when the number of storage devices in the sub-resource pool is increased, the copy slices of the slices in the sub-resource pool are reallocated to the storage devices in the sub-resource pool.
结合第一方面的第二种可能的实现方式,在第五种可能的实现方式中,所述将R个切片分布到预先划分的子资源池中包括:In combination with the second possible implementation of the first aspect, in a fifth possible implementation, the distributing the R slices to the pre-divided sub-resource pools includes:
根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。According to the importance of the data in the R slices, the R slices are distributed to sub-resource pools of different levels.
第二方面,本发明的实施例提供了一种储存数据的装置,用于存储系统,所述存储系统中包括用于存储数据的资源池,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述方法包括:In the second aspect, an embodiment of the present invention provides a device for storing data, which is used in a storage system, and the storage system includes a resource pool for storing data, and the resource pool is logically divided into at least two sub-resource pools , each sub-resource pool includes adjacent N storage devices, N≥2, and the method includes:
分布模块,用于将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中,所述需要存储的数据期望生成的副本数量小于N,R≥2;A distribution module, configured to divide the data to be stored into R slices, and distribute the R slices to the divided sub-resource pools, the number of copies of the data to be stored is expected to be less than N, and R≥2 ;
副本生成模块,用于对分布到每个子资源池中的每个所述切片生成副本切片;A replica generating module, configured to generate replica slices for each of the slices distributed in each sub-resource pool;
储存模块,用于将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。A storage module, configured to store the duplicate slice in a sub-resource pool corresponding to the slice, and the duplicate slice corresponds to the slice with the same data.
结合第二方面,在第二方面的第一种可能的实现方式中,还包括:所述储存模块,还用于将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上,且每个所述副本切片存储的存储设备与对应所述切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。With reference to the second aspect, in the first possible implementation manner of the second aspect, the storage module further includes: the storage module, further configured to store the copy slice in a storage device corresponding to the sub-resource pool to which the slice belongs , and each of the duplicate slices is stored in a storage device different from the storage device corresponding to the slice, and duplicate slices with the same data are stored in different storage devices.
结合第二方面,以及第二方面的第一种可能的实现方式,在第二方面的第二种可能的实现方式中,还包括:In combination with the second aspect and the first possible implementation of the second aspect, the second possible implementation of the second aspect further includes:
划分模块,用于将所述划分的子资源池再次划分为不同级别子资源池集合,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同;A dividing module, configured to divide the divided sub-resource pools into different-level sub-resource pool sets, each sub-resource pool set includes at least one sub-resource pool, and each of the slices in the different-level sub-resource pool sets The number of replica slices expected to be generated is different;
所述副本生成模块,还用于按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。The copy generation module is further configured to generate a corresponding number of copy slices distributed to each of the sub-resource pools according to the number of copy slices expected to be generated by each of the slices corresponding to the level of the set of sub-resource pools to which the sub-resource pool belongs. Number of replica slices.
结合第二方面,以及第二方面的第一种可能的实现方式,在第二方面的第三种可能的实现方式中,还包括:In combination with the second aspect and the first possible implementation of the second aspect, the third possible implementation of the second aspect further includes:
负调节模块,用于针对任一个所述子资源池,减少所述子资源池中的存储设备的数量;A negative adjustment module, configured to reduce the number of storage devices in the sub-resource pool for any one of the sub-resource pools;
数据分配模块,用于将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上,所述子资源池中的剩余的存储设备数量大于所述期望生成的副本数。A data allocation module, configured to allocate the slices in the reduced storage device to other storage devices in the sub-resource pool, where the number of remaining storage devices in the sub-resource pool is greater than the desired copy number.
结合第二方面,以及第二方面的第一种可能的实现方式,在第二方面的第四种可能的实现方式中,还包括:In combination with the second aspect and the first possible implementation of the second aspect, the fourth possible implementation of the second aspect further includes:
正调节模块,用于针对任一个所述子资源池,增加所述子资源池中的存储设备的数量;A positive adjustment module, configured to increase the number of storage devices in the sub-resource pool for any one of the sub-resource pools;
所述数据分配模块,还用于当增加所述子资源池中的存储设备的数量时,将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。The data allocation module is further configured to reallocate the copy slices of the slices in the sub-resource pool to the storage devices in the sub-resource pool when the number of storage devices in the sub-resource pool is increased.
结合第二方面的第二种可能的实现方式,在第五种可能的实现方式中,还包括:In combination with the second possible implementation of the second aspect, the fifth possible implementation also includes:
所述分布模块,还用于根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。The distribution module is further configured to distribute the R slices to sub-resource pools of different levels according to the importance of data in the R slices.
第三方面,本发明的实施例提供了一种储存数据的存储系统,所述存储系统中包括用于存储数据的资源池,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述存储系统包括:处理器、通信接口、总线,其特征在于还包括存储设备,所述处理器、所述通信接口和所有的存储设备,通过所述总线完成相互间的通信,其中:In the third aspect, the embodiment of the present invention provides a storage system for storing data. The storage system includes a resource pool for storing data. The resource pool is logically divided into at least two sub-resource pools, and each sub-resource The pool includes adjacent N storage devices, N≥2, the storage system includes: a processor, a communication interface, a bus, and is characterized in that it also includes a storage device, the processor, the communication interface and all storage devices The devices communicate with each other through the bus, wherein:
所述处理器,用于将需要存储的数据划分为R个切片,通过所述通信接口将R个所述切片分布到划分的所述子资源池中,所述需要存储的数据期望生成的副本数量小于N,R≥2;The processor is configured to divide the data to be stored into R slices, distribute the R slices to the divided sub-resource pools through the communication interface, and the data to be stored is expected to generate a copy The number is less than N, R≥2;
所述处理器,还用于对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。The processor is further configured to generate a copy slice for each of the slices distributed in each sub-resource pool, and store the copy slice in the sub-resource pool corresponding to the slice, and the copy slice and The slices having the same data correspond to each other.
结合第三方面,在第三方面的第一种可能的实现方式中,包括:In combination with the third aspect, the first possible implementation of the third aspect includes:
所述处理器,还用于将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上,且每个所述副本切片存储的存储设备与对应所述切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。The processor is further configured to store the copy slice on a storage device corresponding to the sub-resource pool to which the slice belongs, and each storage device stored in the copy slice is the same as the storage device corresponding to the slice stored Different, replica slices with the same data are stored on different storage devices.
结合第三方面,以及第三方面的第一种可能的实现方式,在第三方面的第二种可能的实现方式中,所述处理器,还用于将所述划分的子资源池再次划分为不同级别子资源池集合,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同;With reference to the third aspect and the first possible implementation manner of the third aspect, in a second possible implementation manner of the third aspect, the processor is further configured to divide the divided sub-resource pools again It is a set of sub-resource pools at different levels, each sub-resource pool set includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in the sub-resource pool sets of different levels is different;
所述处理器,还用于按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。The processor is further configured to generate a corresponding number of duplicate slices distributed to each of the sub-resource pools according to the number of duplicate slices expected to be generated by each of the slices corresponding to the level of the set of sub-resource pools to which the sub-resource pool belongs. A copy slice of .
结合第三方面,以及第三方面的第一种可能的实现方式,在第三方面的第三种可能的实现方式中,所述处理器,还用于在将所述副本切片存储在对应所述切片所属的子资源池中后,针对任一个所述子资源池,当减少所述子资源池中的存储设备的数量时,通过所述通信接口将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上,所述子资源池中的剩余的存储设备数量大于所述期望生成的副本数。With reference to the third aspect, and the first possible implementation manner of the third aspect, in a third possible implementation manner of the third aspect, the processor is further configured to store the copy slice in the corresponding After the sub-resource pool to which the slice belongs, for any one of the sub-resource pools, when reducing the number of storage devices in the sub-resource pool, transfer the slices in the reduced storage devices through the communication interface allocated to other storage devices in the sub-resource pool, and the number of remaining storage devices in the sub-resource pool is greater than the expected number of copies to be generated.
结合第三方面,以及第三方面的第一种可能的实现方式,在第三方面的第四种可能的实现方式中,所述处理器,还用于在将所述副本切片存储在对应所述切片所属的子资源池中后,针对任一个所述子资源池,当增加所述子资源池中的存储设备的数量时,通过所述通信接口将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。With reference to the third aspect and the first possible implementation manner of the third aspect, in a fourth possible implementation manner of the third aspect, the processor is further configured to store the copy slice in the corresponding After the sub-resource pool to which the slice belongs, for any one of the sub-resource pools, when the number of storage devices in the sub-resource pool is increased, the The replica slices are reallocated to the storage devices in the sub-resource pool.
结合第三方面的第二种可能的实现方式,在第五种可能的实现方式中,所述处理器,还用于根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。In combination with the second possible implementation of the third aspect, in a fifth possible implementation, the processor is further configured to distribute the R slices to different levels of in the sub-resource pool.
第四方面,本发明的实施例提供了一种储存数据的计算机程序产品,用于存储系统,所述存储系统中包括用于存储数据的资源池,其特征在于,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述计算机程序产品包括存储了程序代码的计算机可读存储介质,所述程序代码包括的指令用于:In a fourth aspect, an embodiment of the present invention provides a computer program product for storing data, which is used in a storage system, and the storage system includes a resource pool for storing data, wherein the resource pool is logically divided It is at least 2 sub-resource pools, each sub-resource pool includes adjacent N storage devices, N≥2, the computer program product includes a computer-readable storage medium storing program codes, and the instructions included in the program codes are used At:
将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中,所述需要存储的数据期望生成的副本数量小于N,R≥2;dividing the data to be stored into R slices, and distributing the R slices to the divided sub-resource pools, the number of copies of the data to be stored is expected to be less than N, and R≥2;
对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。Generate a copy slice for each slice distributed to each sub-resource pool, and store the copy slice in the sub-resource pool to which the slice belongs, and the copy slice corresponds to the slice with the same data .
本发明实施例提供的修复数据的方法、装置、存储系统以及计算机程序产品,能够将存储系统中的所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个子资源池,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个子资源池中的2个存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The method, device, storage system, and computer program product for repairing data provided by the embodiments of the present invention can divide all storage devices in the storage system into at least two sub-resource pools, and store data in the divided sub-resource pools In , since the data in each sub-resource pool is independent, the data will be completely damaged only if two storage devices in the same sub-resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. Since the embodiment of the present invention can divide multiple sub-resource pools, and store data in the multiple sub-resource pools divided In a sub-resource pool, only when two storage devices in the same sub-resource pool are damaged, the data will be completely damaged, because the probability of damage to two storage devices in the same sub-resource pool is less than that in the prior art when there is only one resource In the case of a pool, there is a probability of damaging two storage devices, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
附图说明Description of drawings
为了更清楚地说明本发明实施例中的技术方案,下面将对实施例中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其它的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.
图1为本发明实施例提供的一种储存数据的方法的流程图;FIG. 1 is a flow chart of a method for storing data provided by an embodiment of the present invention;
图1a为本发明实施例提供的一种储存数据的方法的一种具体实例的示意图;Fig. 1a is a schematic diagram of a specific example of a method for storing data provided by an embodiment of the present invention;
图1b为本发明实施例提供的一种储存数据的方法的另一种具体实例的示意图;Fig. 1b is a schematic diagram of another specific example of a method for storing data provided by an embodiment of the present invention;
图2为本发明实施例提供的另一种储存数据的方法的流程图;FIG. 2 is a flow chart of another method for storing data provided by an embodiment of the present invention;
图3a为本发明实施例提供的又一种储存数据的方法的一种流程图;Fig. 3a is a flowchart of another method for storing data provided by an embodiment of the present invention;
图3a1为本发明实施例提供的又一种储存数据的方法的一种具体实例的示意图;Fig. 3a1 is a schematic diagram of a specific example of another method for storing data provided by an embodiment of the present invention;
图3b为本发明实施例提供的又一种储存数据的方法的另一种流程图;Fig. 3b is another flow chart of another method for storing data provided by an embodiment of the present invention;
图3b1为本发明实施例提供的又一种储存数据的方法的另一种具体实例的示意图;FIG. 3b1 is a schematic diagram of another specific example of another method for storing data provided by an embodiment of the present invention;
图4a为本发明实施例提供的再一种储存数据的方法的一种流程图;Fig. 4a is a flowchart of another method for storing data provided by an embodiment of the present invention;
图4b为本发明实施例提供的再一种储存数据的方法的另一种流程图;Fig. 4b is another flow chart of another method for storing data provided by an embodiment of the present invention;
图5为本发明实施例提供的一种储存数据的装置的结构示意图;FIG. 5 is a schematic structural diagram of a device for storing data provided by an embodiment of the present invention;
图6为本发明实施例提供的另一种储存数据的装置的结构示意图;FIG. 6 is a schematic structural diagram of another device for storing data provided by an embodiment of the present invention;
图7为本发明实施例提供的一种储存数据的存储系统的网络架构示意图。FIG. 7 is a schematic diagram of a network architecture of a storage system for storing data provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其它实施例,都属于本发明保护的范围。The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
一方面,本发明实施例提供了一种储存数据的方法,如图1所示,包括:On the one hand, an embodiment of the present invention provides a method for storing data, as shown in FIG. 1 , including:
需要说明的是,本发明实施例可以用于存储系统,所述存储系统中包括用于存储数据的资源池,其特征在于,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2。并且,本发明实施例可以由具备数据传输、处理、存储功能的设备执行,例如:在分布式存储系统中用于管理硬盘列阵的管理服务器、移动工作站等设备。It should be noted that the embodiment of the present invention can be used in a storage system, and the storage system includes a resource pool for storing data, wherein the resource pool is logically divided into at least two sub-resource pools, and each sub-resource The pool includes N adjacent storage devices, where N≥2. Moreover, the embodiments of the present invention may be implemented by devices with data transmission, processing, and storage functions, for example, devices such as management servers and mobile workstations used to manage hard disk arrays in a distributed storage system.
101,将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中。101. Divide data to be stored into R slices, and distribute the R slices to the divided sub-resource pools.
其中,需要存储的数据期望生成的副本数量小于N,即需要存储的数据期望生成的副本数量小于子资源池中的存储设备的数量,且R≥2。例如:可以将数据切片分布打散在分布式存储架构中的技术手段,比如:分配到子资源池1中的切片为P1、P2、P3,期望生成的副本数量为2,则管理服务器可以根据分配到子资源池1中的切片为P1、P2、P3,生成第一副本的副本切片P1’、P2’、P3’,和第二副本的副本切片P1’’、P2’’、P3’’。并通过分布打散的技术手段将P1、P2、P3、P1’、P2’、P3’、P1’’、P2’’、P3’’储存在子资源池1中的存储设备上。Wherein, the expected number of copies of the data to be stored is less than N, that is, the expected number of copies of the data to be stored is smaller than the number of storage devices in the sub-resource pool, and R≥2. For example: a technical method that can distribute data slices in a distributed storage architecture. For example, the slices allocated to sub-resource pool 1 are P1, P2, and P3, and the number of copies expected to be generated is 2. Then the management server can allocate The slices in the sub-resource pool 1 are P1, P2, and P3, and the duplicate slices P1', P2', P3' of the first copy and the duplicate slices P1'', P2'', P3'' of the second copy are generated. And store P1, P2, P3, P1', P2', P3', P1'', P2'', P3'' on the storage device in the sub-resource pool 1 through the technical means of distribution and dispersal.
在本实施例中,管理服务器可以通过常用的技术手段将需要存储的数据划分为多个数据切片,例如:管理服务器可以通过本领域技术人员所熟知的RAID技术中的数据切片方法将需要存储的数据划分为至少2个数据切片,并将所划分出的数据切片分布在子资源池上。并且,在本实施例中,在子资源池上分布数据切片的具体实施方式可以有多种,例如:管理服务器可以通过RAID技术中的分布打散的方式将数据切片分布在所划分出的子资源池上,也可以将数据切片平局分布在所划分出的子资源池上。In this embodiment, the management server can divide the data to be stored into multiple data slices through common technical means, for example: the management server can divide the data to be stored into multiple The data is divided into at least two data slices, and the divided data slices are distributed on the sub-resource pools. Moreover, in this embodiment, there may be multiple specific implementations for distributing data slices on the sub-resource pools. For example, the management server may distribute data slices to the divided sub-resources in a way of distributing in RAID technology. On the resource pool, the data slices can also be evenly distributed on the divided sub-resource pools.
进一步的,在本实施例中,管理服务器可以将系统中的存储设备在逻辑层面上划分为多个子资源池,每一个子资源池由多个存储设备组成,例如:Further, in this embodiment, the management server can logically divide the storage devices in the system into multiple sub-resource pools, and each sub-resource pool consists of multiple storage devices, for example:
分布式存储系统中总共有24个硬盘,管理服务器可以通过MDC(MetadataController,元数据控制器)将这24个硬盘划分为3个子资源池,每一个子资源池包括了8个硬盘,从而可以形成如图1a所示的多子资源池的架构。There are a total of 24 hard disks in the distributed storage system. The management server can divide these 24 hard disks into 3 sub-resource pools through MDC (MetadataController). Each sub-resource pool includes 8 hard disks, which can form The architecture of multiple sub-resource pools is shown in Figure 1a.
在实际应用中,在所储存的数据进行寻址的过程中常用的LBA(LogicalBlock Addressing,逻辑块寻址)可以映射到不同的子资源池中,各个子资源池以及块存放逻辑分区单元Partition可以通过MDC的集群管理节点进行分配,比如:在每个子资源池中有n个Partition,每个子资源池中的每个硬盘上可以有多个Partition,则MDC中可以记录每个子资源池中的PartitionID(块存放逻辑分区单元ID)和子资源池ID,并通过“[子资源池ID.PartitionID]”这种字符信息的形式标识整个硬盘列阵中的每一个Partition,从而使每一个Partition都有唯一的标识信息;系统中的其他设备可以根据每一个Partition的标识信息采用常用的技术手段访问子资源池中的Partition。In practical applications, the commonly used LBA (LogicalBlock Addressing, Logical Block Addressing) in the process of addressing stored data can be mapped to different sub-resource pools, and each sub-resource pool and block storage logical partition unit Partition can be Allocation is done through the cluster management node of MDC. For example, if there are n Partitions in each sub-resource pool, and each hard disk in each sub-resource pool can have multiple Partitions, then the PartitionID in each sub-resource pool can be recorded in MDC (The block stores the logical partition unit ID) and the sub-resource pool ID, and identifies each Partition in the entire hard disk array in the form of "[sub-resource pool ID.PartitionID]", so that each Partition has a unique identification information; other devices in the system can access the Partition in the sub-resource pool by using common technical means according to the identification information of each Partition.
102,对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中。102. Generate a duplicate slice for each slice distributed to each sub-resource pool, and store the duplicate slice in the sub-resource pool to which the corresponding slice belongs.
在本实施例中,需要存储的数据期望生成至少二个相同的副本,且所述需要存储的数据期望生成的副本数小于N。并且,副本切片与需要存储的数据的相同的切片相互对应,储存在一个子资源池中的副本切片与其他储存在该子资源池中的副本切片相同。即,一个子资源池中的副本切片与其他子资源池中的副本切片不同。例如:如图1b所示,第一副本中的副本切片可以被管理服务器分为P1-P8,第二副本中的副本切片可以被分为P1’-P8’,并且第一副本与第二副本的内容相同,即P1与P1’相同,P2与P2’相同,直至P8与P8’相同。In this embodiment, at least two identical copies of the data to be stored are expected to be generated, and the number of copies of the data to be stored is expected to be less than N. Moreover, the duplicate slices correspond to the same slices of the data to be stored, and the duplicate slices stored in one sub-resource pool are the same as the duplicate slices stored in other sub-resource pools. That is, the replica slices in one subresource pool are different from the replica slices in other subresource pools. For example: as shown in Figure 1b, the replica slices in the first replica can be divided into P1-P8 by the management server, the replica slices in the second replica can be divided into P1'-P8', and the first replica and the second replica The content is the same, that is, P1 is the same as P1', P2 is the same as P2', until P8 is the same as P8'.
在子资源池1中,硬盘1承载了P1、P2’、P5、P6’,硬盘2承载了P1’、P2、P5’、P6,即第一副本在子资源池1中的内容与第二副本在子资源池1中的内容相同,同理,第一副本在子资源池2中的内容与第二副本在子资源池2中的内容相同。In sub-resource pool 1, hard disk 1 carries P1, P2', P5, and P6', and hard disk 2 carries P1', P2, P5', and P6. The content of the copy in sub-resource pool 1 is the same, and similarly, the content of the first copy in sub-resource pool 2 is the same as that of the second copy in sub-resource pool 2 .
进一步的,比如:P1和P1’都在子资源池1中,即与P1相同的Partition只存在于子资源池1中,同理,相同的Partition都在同一个子资源池中,使得一个子资源池中的数据在内容上独立于其他子资源池。当硬盘损坏时,比如:硬盘1损坏,则只需根据同在子资源池1中的硬盘2中的P1’、P2、P5’、P6,即可修复硬盘1中的P1、P2’、P5、P6’,使得当一个子资源池中的硬盘损坏时,管理服务器只需根据该子资源池中其他硬盘中的数据即可修复受损硬盘中的数据,在从而使得子资源池中的数据在内容上独立于其他子资源池。Further, for example: both P1 and P1' are in sub-resource pool 1, that is, the same Partition as P1 only exists in sub-resource pool 1. Similarly, the same Partition is in the same sub-resource pool, so that a sub-resource The data in a pool is content independent of other child resource pools. When the hard disk is damaged, for example: hard disk 1 is damaged, P1, P2', and P5 in hard disk 1 can be repaired only according to P1', P2, P5', and P6 in hard disk 2 that are also in sub-resource pool 1 , P6', so that when a hard disk in a sub-resource pool is damaged, the management server can repair the data in the damaged hard disk only according to the data in other hard disks in the sub-resource pool, so that the data in the sub-resource pool Independent in content from other sub-resource pools.
需要说明的是,在本发明实施例中,由于一个子资源池中的副本切片与其他子资源池中的副本切片不同,使得子资源池中的数据在内容上独立于其他子资源池,并且根据需要储存的数据所生成的副本是储存在多个子资源池上的,则整个副本数据完全损坏的条件是:在一个子资源池中有2个存储设备同时损坏,例如:在如图1b所示的双副本的场景下,整个副本数据完全损坏的条件是:子资源池1中的硬盘1和硬盘2同时损坏,或子资源池2中的硬盘3和硬盘4同时损坏。若出现子资源池1中只损坏一个硬盘,同时子资源池2中也只损坏一个硬盘,则副本数据依然可以被修复,即只有出现一个子资源池中的2个硬盘同时损坏的情况时,副本数据才会损坏。由于硬盘的损坏几率基本上是相同的,设硬盘的损坏几率为A,则若采用现有技术的方案,比如:现有技术的方案是图1b中的硬盘1、2、3、4都在一个资源池中,则副本数据损坏的几率(即2个硬盘同时损坏的几率)为A2,或可以理解为分布式存储系统在数据重建的时间窗口内副本数据损坏的几率为A;而本发明实施例的方案,图1b中的硬盘1、2在子资源池1中,硬盘3、4在子资源池4中,并且由于子资源池1中的数据在内容上独立于子资源池2,则副本数据损坏的几率(即必须是同一个子资源池中2个硬盘同时损坏的几率)为0.5A2,或可以理解为分布式存储系统在数据重建的时间窗口内副本数据损坏的几率为0.5A;由此可知,在本发明实施例的方案中,所划分出的子资源池越多则副本数据损坏的几率越低,比如:划分出100个子资源池,则在数据重建的时间窗口内副本数据损坏的几率为0.01A,若每一个子资源池正好损坏一个硬盘,则本发明实施例所提供的方案能够应对最多100个硬盘损坏这种极端情况,并且由于各个子资源池中的数据是相互独立的,还能够修复数据。It should be noted that, in this embodiment of the present invention, since the copy slices in one sub-resource pool are different from those in other sub-resource pools, the data in the sub-resource pool is independent of other sub-resource pools in terms of content, and The copies generated according to the data to be stored are stored in multiple sub-resource pools, and the condition for the entire copy data to be completely damaged is that two storage devices in one sub-resource pool are damaged at the same time, for example: as shown in Figure 1b In the double-copy scenario, the conditions for the entire copy data to be completely damaged are: hard disk 1 and hard disk 2 in sub-resource pool 1 are damaged at the same time, or hard disk 3 and hard disk 4 in sub-resource pool 2 are damaged at the same time. If only one hard disk in sub-resource pool 1 is damaged, and only one hard disk in sub-resource pool 2 is damaged at the same time, the replica data can still be repaired, that is, only when two hard disks in a sub-resource pool are damaged at the same time, The copy data will be corrupted. Since the damage probability of the hard disk is basically the same, if the damage probability of the hard disk is A, then if the prior art scheme is adopted, for example: the prior art scheme is that hard disks 1, 2, 3, and 4 in Fig. In a resource pool, the probability of copy data damage (that is, the probability of two hard disks being damaged at the same time) is A 2 , or it can be understood that the probability of copy data damage in the distributed storage system within the time window of data reconstruction is A; while this In the scheme of the embodiment of the invention, hard disks 1 and 2 in Fig. 1b are in sub-resource pool 1, hard disks 3 and 4 are in sub-resource pool 4, and since the data in sub-resource pool 1 is independent of sub-resource pool 2 in content , then the probability of replica data damage (that is, the probability that two hard disks in the same sub-resource pool must be damaged at the same time) is 0.5A 2 , or it can be understood that the probability of replica data damage in the distributed storage system within the data reconstruction time window is 0.5A; it can be seen that, in the solution of the embodiment of the present invention, the more sub-resource pools are divided, the lower the probability of copy data damage. For example, if 100 sub-resource pools are divided, the time window for data reconstruction The probability of internal copy data damage is 0.01A. If exactly one hard disk is damaged in each sub-resource pool, the solution provided by the embodiment of the present invention can deal with the extreme situation that a maximum of 100 hard disks are damaged. The data is independent of each other, and it is also possible to repair the data.
进一步可以举例,在实际应用中,存储设备如硬盘的故障率一般为4%,再考虑到一些常用的存储设备的性能参数对硬盘故障率的影响,根据本发明实施例的方案,可以获得如表一所示的数据:Further, for example, in practical applications, the failure rate of storage devices such as hard disks is generally 4%, and considering the influence of performance parameters of some commonly used storage devices on the failure rate of hard disks, according to the solutions of the embodiments of the present invention, the following can be obtained: Data shown in Table 1:
表一Table I
由此可见,本发明实施例相对于现有技术,子资源池中包括相邻的N个存储设备,而相邻的存储设备同时损坏的几率很小,因此损坏的存储设备不在同一个子资源池中的几率非常大,而当一个子资源池中的数据发生损坏,就可以通过这个子资源池中的其他存储设备上的数据来恢复从而增加了分布式存储系统的容错性,降低了数据损坏的几率,从而提高数据的安全性。It can be seen that, compared with the prior art, the embodiment of the present invention includes N adjacent storage devices in the sub-resource pool, and the probability of adjacent storage devices being damaged at the same time is very small, so the damaged storage devices are not in the same sub-resource pool The chances are very high, and when the data in a sub-resource pool is damaged, it can be recovered through the data on other storage devices in the sub-resource pool, which increases the fault tolerance of the distributed storage system and reduces data damage probability, thereby improving data security.
本发明实施例提供的修复数据的方法,能够将所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,例如:如图1b所示,子资源池1中存储的数据切片为P1’、P2、P5’、P6、P1、P2’、P5、P6’,没有任何一个数据切片与子资源池2中所存储的数据切片相同,则在子资源池2中的数据损坏与否都不会影响到子资源池1中的数据,且不需要提取子资源池1中的数据参与修复子资源池2中的数据,因此各个子资源池中的数据是独立的。由于各个子资源池中的数据是独立,一个子资源池中的数据损坏与否、数据修复等过程都不会影响到其他的子资源池中的数据,则只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中,存储有相互可以修复的数据的存储设备可能会同时损坏,造成数据完全损坏的,本发明实施例由于能够划分出多个子资源池,每个子资源池中包括N个相邻的存储设备,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的相互可以修复的数据所在的存储设备都损坏时,数据才会完全损坏,由于同一个子资源池中的相邻的存储设备都损坏的几率很小从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The method for repairing data provided by the embodiment of the present invention can divide all storage devices into at least two sub-resource pools, and store data in the divided sub-resource pools, because the data in each sub-resource pool is independent , for example: as shown in Figure 1b, the data slices stored in sub-resource pool 1 are P1', P2, P5', P6, P1, P2', P5, and P6', and none of the data slices is related to sub-resource pool 2 The stored data slices are the same, so whether the data in sub-resource pool 2 is damaged or not will not affect the data in sub-resource pool 1, and there is no need to extract the data in sub-resource pool 1 to participate in repairing sub-resource pool 2 data, so the data in each sub-resource pool is independent. Since the data in each sub-resource pool is independent, whether the data in one sub-resource pool is damaged or not, data repair and other processes will not affect the data in other sub-resource pools, there are only two sub-resource pools in the same sub-resource pool The data will be completely damaged only if the storage device is damaged. In the prior art, storage devices storing mutually repairable data may be damaged at the same time, resulting in complete data damage. Since the embodiment of the present invention can divide multiple sub-resource pools, each sub-resource pool includes N adjacent storage device, and store the data in multiple sub-resource pools. Only when the storage devices where the mutually repairable data in the same sub-resource pool are located are all damaged, the data will be completely damaged. Because the same sub-resource pool The probability that the adjacent storage devices are all damaged is very small, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
可选的,本发明实施例还提供了一种储存数据的方法,如图2所示,还可以包括:Optionally, the embodiment of the present invention also provides a method for storing data, as shown in Figure 2, which may also include:
201,将需要存储的数据划分为R个切片,将R个所述切片分布到包含指定类型的存储设备所述子资源池中。201. Divide data to be stored into R slices, and distribute the R slices to the sub-resource pools containing storage devices of a specified type.
其中,一个子资源池可以包含至少2个相邻的存储设备,存储设备用于存储数据。Wherein, a sub-resource pool may include at least two adjacent storage devices, and the storage devices are used to store data.
并且,所获取的子资源池包含指定类型的存储设备,存储设备的类型包括但不限于:硬盘、固态硬盘SSD。In addition, the obtained sub-resource pool includes a specified type of storage device, and the type of the storage device includes but is not limited to: hard disk, solid state disk SSD.
202,对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中。202. Generate a duplicate slice for each slice distributed to each sub-resource pool, and store the duplicate slice in the sub-resource pool to which the corresponding slice belongs.
在本实施例中,管理服务器可以确定所要使用的存储设备的类型,并将需要储存的数据的切片储存在包含该类型存储设备的子资源池,再对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中,例如:In this embodiment, the management server can determine the type of storage device to be used, and store the slices of the data to be stored in the sub-resource pool containing the storage device of this type, and then distribute to each sub-resource pool The slice generates a copy slice, and stores the copy slice in the sub-resource pool corresponding to the slice, for example:
可以在管理服务器中预存如表二所示的数据的类型与存储设备的类型之间的映射关系,并根据需要储存的数据的类型确定所要使用的存储设备的类型,再获取包含该类型存储设备的子资源池,最后将需要储存的数据储存在包含该类型存储设备的子资源池中。The mapping relationship between the type of data and the type of storage device shown in Table 2 can be pre-stored in the management server, and the type of storage device to be used can be determined according to the type of data to be stored, and then the storage device containing this type can be obtained. The sub-resource pool of the storage device, and finally store the data to be stored in the sub-resource pool containing this type of storage device.
表二Table II
再例如:Another example:
可以在管理服务器中预存如表三所示的数据的大小与存储设备的类型之间的映射关系,并根据需要储存的数据的大小确定所要使用的存储设备的类型,再获取包含该类型存储设备的子资源池,最后将需要储存的数据储存在包含该类型存储设备的子资源池中。The mapping relationship between the size of data and the type of storage device as shown in Table 3 can be pre-stored in the management server, and the type of storage device to be used can be determined according to the size of the data to be stored, and then the storage device containing this type can be obtained The sub-resource pool of the storage device, and finally store the data to be stored in the sub-resource pool containing this type of storage device.
表三Table three
即在本实施例中,管理服务器确定所要使用的存储设备的类型具体实施手段可以有多种,在实际应用中,本领域技术人员也可以在本发明实施例的基础上结合现有技术手段确定所要使用的存储设备的类型。由于不同类型的存储设备的安全性也不同,例如:目前很多固态硬盘的损坏的几率小于机械硬盘。从而管理服务器可以根据数据的重要程度选择相应类型的存储设备,从而进一步提高了重要数据的安全性。That is to say, in this embodiment, the management server can determine the type of the storage device to be used in various specific implementation means. In practical applications, those skilled in the art can also determine The type of storage device to use. Because the security of different types of storage devices is also different, for example, many solid state drives are less likely to be damaged than mechanical hard drives. Therefore, the management server can select a corresponding type of storage device according to the importance of the data, thereby further improving the security of important data.
进一步可选的,本发明实施例又提供了一种储存数据的方法,其中如图3a所示,包括:Further optionally, an embodiment of the present invention provides a method for storing data, as shown in FIG. 3a, including:
301,将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中。301. Divide data to be stored into R slices, and distribute the R slices to the divided sub-resource pools.
302,对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上。302. Generate a duplicate slice for each slice distributed to each sub-resource pool, and store the duplicate slice on a storage device corresponding to the sub-resource pool to which the slice belongs.
其中,副本切片与需要存储的数据的相同的切片相互对应。每个所述副本切片存储的存储设备与对应所述切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。例如:Wherein, the copy slice corresponds to the same slice of the data to be stored. The storage device for storing each of the duplicate slices is different from the storage device for storing the corresponding slices, and the duplicate slices with the same data are stored on different storage devices. For example:
如图1b所示,第一副本的副本切片为P1-P8,第二副本的副本切片为P1’-P8’,并且P1与P1’相同,P2与P2’相同,直至P8与P8’相同。在子资源池1中,硬盘1承载了P1、P2’、P5、P6’,硬盘2承载了P1’、P2、P5’、P6,即第一副本在子资源池1中的内容与第二副本在子资源池1中的内容相同,同理,第一副本在子资源池2中的内容与第二副本在子资源池2中的内容相同。As shown in Figure 1b, the copy slices of the first copy are P1-P8, the copy slices of the second copy are P1'-P8', and P1 is the same as P1', P2 is the same as P2', until P8 is the same as P8'. In sub-resource pool 1, hard disk 1 carries P1, P2', P5, and P6', and hard disk 2 carries P1', P2, P5', and P6. The content of the copy in sub-resource pool 1 is the same, and similarly, the content of the first copy in sub-resource pool 2 is the same as that of the second copy in sub-resource pool 2 .
进一步的,P1和P1’都在子资源池1中,即与P1相同的Partition只存在于子资源池1中,同理,相同的Partition都在同一个子资源池中,使得一个子资源池中的数据在内容上独立于其他子资源池。并且,每个副本切片存储的硬盘与对应切片存储的硬盘不同,比如:子资源池1中的一个副本切片与子资源池2中的中的任一个副本的任一个切片都不相同。且数据相同的副本切片存储在不同的存储设备上,比如:P1和P1’在不同的硬盘上。Further, both P1 and P1' are in sub-resource pool 1, that is, the same Partition as P1 only exists in sub-resource pool 1. Similarly, the same Partition is in the same sub-resource pool, so that a sub-resource pool The data of is independent in content from other sub-resource pools. Moreover, the hard disk stored in each copy slice is different from the hard disk stored in the corresponding slice. For example, a copy slice in sub-resource pool 1 is different from any slice in any copy in sub-resource pool 2. And the copy slices with the same data are stored on different storage devices, for example: P1 and P1' are on different hard disks.
303a,针对任一个所述子资源池,减少所述子资源池中的存储设备的数量。303a. For any one of the sub-resource pools, reduce the number of storage devices in the sub-resource pool.
在本实施例中,管理服务器可以通过常用的技术手段实现子资源池中的存储设备的热插/拔,即可以在子资源池正在运行的过程中增加或减少子资源池中的存储设备的数量。例如:当子资源池中的某一块硬盘故障并需要通过常用的技术手段进行数据重建时,管理服务器可以将故障硬盘上的数据在本子资源池中进行重建恢复,如图3a1所示。当子资源池x中的硬盘1故障后,硬盘1上的Partition可以按照实现通过事先设定的规律全部迁移到了子资源池中的其他3块硬盘中。从而实现在子资源池正在运行的过程中减少子资源池中的存储设备的数量。In this embodiment, the management server can implement hot plugging/removing of storage devices in the sub-resource pool by common technical means, that is, the number of storage devices in the sub-resource pool can be increased or decreased while the sub-resource pool is running. quantity. For example: when a hard disk in a sub-resource pool fails and needs to be reconstructed by common technical means, the management server can rebuild and restore the data on the failed hard disk in this sub-resource pool, as shown in Figure 3a1. When hard disk 1 in sub-resource pool x fails, the Partitions on hard disk 1 can all be migrated to the other three hard disks in the sub-resource pool according to the rules set in advance. In this way, the number of storage devices in the sub-resource pool can be reduced while the sub-resource pool is running.
304a,将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上。304a. Allocate the slices in the reduced storage device to other storage devices in the sub-resource pool.
并列的,如图3b所示,还可以包括:Parallel, as shown in Figure 3b, may also include:
303b,针对任一个所述子资源池,增加所述子资源池中的存储设备的数量。303b. For any one of the sub-resource pools, increase the number of storage devices in the sub-resource pool.
例如:如图3b1所示。当管理服务器对子资源池x中增加了硬盘4后,硬盘1-3上的Partition可以按照实现通过事先设定的规律平均分配在硬盘1-4这4快硬盘中,从而将硬盘1-3上的部分内容迁移到了硬盘4中。从而实现在子资源池正在运行的过程中增加子资源池中的存储设备的数量。For example: as shown in Figure 3b1. When the management server adds hard disk 4 to the sub-resource pool x, the Partition on hard disk 1-3 can be evenly distributed among the 4 fast hard disks of hard disk 1-4 according to the rule set in advance, so that hard disk 1-3 Part of the content on the network has been migrated to hard disk 4. In this way, the number of storage devices in the sub-resource pool can be increased while the sub-resource pool is running.
304b,将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。304b. Reallocate the copy slice of the slice in the sub-resource pool to the storage device in the sub-resource pool.
在本实施例中,管理服务器可以在增加了子资源池中的存储设备后,将子资源池中的切片进行重新分配,比如:如图3b1所示,将硬盘1-3上的所有数据切片,在硬盘1-4上重新进行分布打散。In this embodiment, the management server can reallocate slices in the sub-resource pool after adding storage devices in the sub-resource pool, for example: as shown in Figure 3b1, all data slices on hard disks 1-3 , re-distribute and break up on hard disks 1-4.
本发明实施例还可以实现子资源池中的存储设备的热插/拔,使得各个子资源池在出现存储设备损坏、存储设备停机维护等非正常运行情况时,可以保证储存在子资源池中的数据不丢失,使得分布式存储系统在运行时可以在保证数据稳定的情况下随时排除不能用的存储设备,从而进一步提高了数据的安全性。The embodiments of the present invention can also realize hot plugging/removing of storage devices in the sub-resource pools, so that each sub-resource pool can be guaranteed to be stored in the sub-resource pools when abnormal operation conditions such as storage device damage and storage device shutdown maintenance occur. The data is not lost, so that the distributed storage system can exclude unusable storage devices at any time while ensuring data stability during operation, thereby further improving data security.
本发明实施例提供的修复数据的方法,能够将所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,以一个子资源池中包括两个存储设备为例,只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就可能会造成数据完全损坏的情况,本发明实施例由于能够划分出多个子资源池,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个子资源池中的2个相邻的存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储有可以相互进行数据恢复的存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The method for repairing data provided by the embodiment of the present invention can divide all storage devices into at least two sub-resource pools, and store data in the divided sub-resource pools, because the data in each sub-resource pool is independent , taking two storage devices included in a sub-resource pool as an example, the data will be completely damaged only if the two storage devices in the same sub-resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices may cause complete data damage. The embodiment of the present invention can divide multiple sub-resource pools and store data in the divided sub-resource pools. In multiple sub-resource pools, only when two storage devices in the same sub-resource pool are damaged, the data will be completely damaged, because the probability of damage to two adjacent storage devices in the same sub-resource pool is smaller than that in the prior art In the case of only one resource pool, the probability of damaging two storage devices that can restore data to each other makes the embodiment of the present invention reduce the probability of complete data damage compared with the prior art, thereby improving data security. sex.
再进一步可选的,本发明实施例还提供了一种储存数据的方法,还可以包括:将所述划分的子资源池再次划分为不同级别子资源池集合,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同;所述对分布到每个子资源池中的每个所述切片生成副本切片,包括:按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。具体实现中,可以如图4a所示,包括:Still further optionally, the embodiment of the present invention also provides a method for storing data, which may further include: subdividing the divided sub-resource pools into sub-resource pool sets of different levels, and each sub-resource pool set includes at least A sub-resource pool, the number of duplicate slices expected to be generated for each slice in different levels of sub-resource pool sets is different; the generation of duplicate slices for each slice distributed to each sub-resource pool includes: The number of duplicate slices expected to be generated by each slice corresponding to the level of the sub-resource pool set to which the sub-resource pool belongs, will generate a corresponding number of duplicate slices for each of the slices distributed to the sub-resource pools. In specific implementation, it can be shown in Figure 4a, including:
401a,将所述划分的子资源池再次划分为不同级别子资源池集合。401a. Divide the divided sub-resource pools into sub-resource pool sets of different levels.
其中,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同。例如:Wherein, each set of sub-resource pools includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in the set of sub-resource pools of different levels is different. For example:
确定Q个一级子资源池,或Q’个二级子资源池。其中,一级子资源池储存的切片期望生成的副本数量为M,二级子资源池储存的切片期望生成的副本数量为O,且M>O≥2,Q≥2,Q’≥2。Determine Q first-level sub-resource pools, or Q' second-level sub-resource pools. Among them, the number of copies expected to be generated from slices stored in the first-level sub-resource pool is M, and the number of copies expected to be generated from slices stored in the second-level sub-resource pool is O, and M>O≥2, Q≥2, and Q'≥2.
在本实施例中,管理服务器可以通过常用的技术手段设定可以在子资源池储存的切片期望生成的副本数量,例如:副本1、2、3的内容相同,子资源池x中切片期望生成的副本数量为2,子资源池y中切片期望生成的副本数量为3。则在子资源池x中,在储存了来自副本1中的副本切片P1、P2、P3,和来自副本2中的副本切片P1’、P2’、P3’后,就不能再储存来自副本3中的副本切片P1’’、P2’’、P3’’,若要同时储存来自副本1、2、3的全部或部分内容,则需要使用子资源池y。In this embodiment, the management server can use common technical means to set the expected number of copies of the slices stored in the sub-resource pool. The number of copies of is 2, and the number of copies expected to be generated by slices in sub-resource pool y is 3. Then in the sub-resource pool x, after storing the copy slices P1, P2, P3 from copy 1 and the copy slices P1', P2', P3' from copy 2, it cannot store the copy slices from copy 3 any more. Copy slices P1'', P2'', and P3'' of , if you want to store all or part of the content from copies 1, 2, and 3 at the same time, you need to use sub-resource pool y.
并且,可以在用于表示子资源池属性的标识信息中加入用于限定副本数量的字符,比如:010表示子资源池允许所储存的数据期望的副本的数量为2,011表示子资源池允许所储存的数据期望的副本的数量为3。In addition, characters used to limit the number of copies can be added to the identification information used to represent the attributes of the sub-resource pool, for example: 010 indicates that the sub-resource pool allows the expected number of copies of stored data to be 2, and 011 indicates that the sub-resource pool allows The desired number of copies of the stored data is three.
402a,按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。402a. According to the number of duplicate slices expected to be generated for each slice corresponding to the level of the sub-resource pool set to which the sub-resource pool belongs, generate a corresponding number of duplicate slices for each of the slices distributed to the sub-resource pools.
在本实施例中,管理服务器确定所使用的子资源池的等级方式可以由多种,例如:管理服务器可以根据需要储存的数据的类型,确定子资源池的等级,比如:如表四所示,管理服务器可以预存数据的类型与子资源池等级之间的映射关系,并根据需要储存的数据的类型和表四所示的映射关系确定储存需要储存的数据所需的子资源池的等级。In this embodiment, the management server can determine the level of the sub-resource pool used in various ways, for example: the management server can determine the level of the sub-resource pool according to the type of data to be stored, for example: as shown in Table 4 , the management server can pre-store the mapping relationship between the type of data and the level of the sub-resource pool, and determine the level of the sub-resource pool required to store the data to be stored according to the type of data to be stored and the mapping relationship shown in Table 4.
表四Table four
由于在实际应用中,不同数据的重要程度也不同,本发明实施例可以对不同重要程度的数据采取不同等级的存储方式,对于相对而言重要的数据可以通过具备较多副本的存储方式以保证数据的安全性,对于相对而言不重要的数据可以通过具备较少副本的存储方式以增加系统中存储设备的利用率,从而提高分布式存储系统的运行效率,使得管理服务器可以将更多的存储设备用于储存较为重要的数据,从而进一步提高了重要数据的安全性。Since in practical applications, different data have different degrees of importance, the embodiment of the present invention can adopt different levels of storage methods for data with different degrees of importance, and for relatively important data, a storage method with more copies can be used to ensure For data security, relatively unimportant data can be stored with fewer copies to increase the utilization of storage devices in the system, thereby improving the operating efficiency of the distributed storage system, so that the management server can use more The storage device is used to store relatively important data, thereby further improving the security of the important data.
并列的,如图4b所示,还包括:Parallel, as shown in Figure 4b, also includes:
401b,将所述划分的子资源池再次划分为不同级别子资源池集合。401b. Divide the divided sub-resource pools into sub-resource pool sets of different levels.
其中,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同。Wherein, each set of sub-resource pools includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in the set of sub-resource pools of different levels is different.
402b,将需要存储的数据划分为R个切片,根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。402b. Divide the data to be stored into R slices, and distribute the R slices to sub-resource pools of different levels according to the importance of the data in the R slices.
其中,R≥2。例如:Among them, R≥2. For example:
在使用虚拟机的过程中需要在子资源池中储存虚拟机的系统卷的数据和用户卷的数据,由于系统卷的数据的安全性更加重要,则管理服务器可以根据系统卷的数据所对应的切片生成3个副本并储存在一级子资源池中,并根据用户卷的数据所对应的切片生成2个副本并储存在二级子资源池中,从而实现根据同一份需要储存的数据的不同部分生成不同数量的副本,并储存在相应级别的子资源池中。In the process of using the virtual machine, the data of the system volume and user volume of the virtual machine need to be stored in the sub-resource pool. Since the data security of the system volume is more important, the management server can Three copies of the slice are generated and stored in the first-level sub-resource pool, and two copies are generated and stored in the second-level sub-resource pool according to the slice corresponding to the data of the user volume, so as to achieve different data storage according to the same copy. Partially generate different numbers of copies and store them in sub-resource pools of corresponding levels.
由于在实际应用中,同一份数据的不同部分的重要程度也不同,本发明实施例可以针对数据的不同部分的重要程度的数据采取不同等级的存储方式,对于数据中相对而言重要的部分可以通过具备较多副本的存储方式以保证安全性,对于相对与不重要的部分可以通过具备较少副本的存储方式以增加系统中存储设备的利用率,从而提高分布式存储系统的运行效率,使得管理服务器可以将更多的存储设备用于储存数据中较为重要的部分,从而进一步提高了数据中重要部分的安全性。Since in practical applications, the importance of different parts of the same data is also different, the embodiment of the present invention can adopt different levels of storage methods for the data of different parts of the importance of the data, and the relatively important parts of the data can be The storage method with more copies is used to ensure security. For relatively unimportant parts, the storage method with fewer copies can be used to increase the utilization rate of storage devices in the system, thereby improving the operating efficiency of the distributed storage system. The management server can use more storage devices to store more important parts of the data, thereby further improving the security of the important parts of the data.
本发明实施例提供的修复数据的方法,能够将所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个子资源池,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个子资源池中的2个存储设备损坏的几率很小,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The method for repairing data provided by the embodiment of the present invention can divide all storage devices into at least two sub-resource pools, and store data in the divided sub-resource pools, because the data in each sub-resource pool is independent , the data will be completely damaged only if two storage devices in the same sub-resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. Since the embodiment of the present invention can divide multiple sub-resource pools, and store data in the multiple sub-resource pools divided In a sub-resource pool, only when two storage devices in the same sub-resource pool are damaged, the data will be completely damaged. Since the probability of damage to the two storage devices in the same sub-resource pool is very small, the embodiment of the present invention is relatively Because the existing technology can reduce the probability of complete data damage, thereby improving the security of data.
另一方面,本发明实施例提供了一种储存数据的装置,用于存储系统,所述存储系统中包括用于存储数据的资源池,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,如图5所示,包括:On the other hand, an embodiment of the present invention provides an apparatus for storing data, which is used in a storage system. The storage system includes a resource pool for storing data, and the resource pool is logically divided into at least two sub-resource pools. Each sub-resource pool includes adjacent N storage devices, N≥2, as shown in Figure 5, including:
分布模块51,用于将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中。The
其中,需要存储的数据期望生成的副本数量小于N,R≥2。Wherein, the number of copies of the data to be stored is expected to be less than N, and R≥2.
副本生成模块52,用于对分布到每个子资源池中的每个所述切片生成副本切片。The
储存模块53,用于将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。The
本发明实施例提供的修复数据的装置,能够将所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个子资源池,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个子资源池中的2个存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The device for repairing data provided by the embodiment of the present invention can divide all storage devices into at least two sub-resource pools, and store data in the divided sub-resource pools, because the data in each sub-resource pool is independent , the data will be completely damaged only if two storage devices in the same sub-resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. Since the embodiment of the present invention can divide multiple sub-resource pools, and store data in the multiple sub-resource pools divided In a sub-resource pool, only when two storage devices in the same sub-resource pool are damaged, the data will be completely damaged, because the probability of damage to two storage devices in the same sub-resource pool is less than that in the prior art when there is only one resource In the case of a pool, there is a probability of damaging two storage devices, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
可选的,如图6所示,本发明实施例提供的储存数据的装置的,可以包括:Optionally, as shown in Figure 6, the device for storing data provided by the embodiment of the present invention may include:
分布模块62,用于将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中。The
其中,需要存储的数据期望生成的副本数量小于N,R≥2。Wherein, the number of copies of the data to be stored is expected to be less than N, and R≥2.
副本生成模块63,用于对分布到每个子资源池中的每个所述切片生成副本切片。A
储存模块64,用于将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。The
进一步的,所述储存模块64,还用于将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上。Further, the
且每个所述副本切片存储的存储设备与对应所述切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。In addition, the storage device for storing each of the duplicate slices is different from the storage device for storing the corresponding slices, and the duplicate slices with the same data are stored on different storage devices.
进一步可选的,本发明实施例提供的储存数据的装置还可以包括:Further optionally, the device for storing data provided in the embodiment of the present invention may also include:
划分模块61,用于将所述划分的子资源池再次划分为不同级别子资源池集合。A
其中,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个所述切片的期望生成的副本切片数量不同。Wherein, each set of sub-resource pools includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in the set of sub-resource pools of different levels is different.
所述副本生成模块63,还用于按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。The
由于在实际应用中,不同数据的重要程度也不同,本发明实施例所提供的装置可以对不同重要程度的数据采取不同等级的存储方式,对于相对而言重要的数据可以通过具备较多副本的存储方式以保证数据的安全性,对于相对而言不重要的数据可以通过具备较少副本的存储方式以增加系统中存储设备的利用率,从而提高分布式存储系统的运行效率,使得管理服务器可以将更多的存储设备用于储存较为重要的数据,从而进一步提高了重要数据的安全性。Since in practical applications, different data have different degrees of importance, the device provided by the embodiment of the present invention can adopt different levels of storage methods for data of different degrees of importance, and for relatively important data, the Storage method to ensure data security. Relatively unimportant data can be stored with fewer copies to increase the utilization of storage devices in the system, thereby improving the operating efficiency of the distributed storage system, so that the management server can More storage devices are used to store more important data, thereby further improving the security of important data.
再进一步可选的,本发明实施例提供的储存数据的装置还可以包括:Still further optionally, the device for storing data provided by the embodiment of the present invention may also include:
负调节模块65,用于针对任一个所述子资源池,减少所述子资源池中的存储设备的数量。The
正调节模块66,用于针对任一个所述子资源池,增加所述子资源池中的存储设备的数量。The
数据分配模块67,用于将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上。The
其中,子资源池中的剩余的存储设备数量大于所述期望生成的副本数。Wherein, the number of remaining storage devices in the sub-resource pool is greater than the number of copies expected to be generated.
所述数据分配模块67,还用于当增加所述子资源池中的存储设备的数量时,将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。The
更进一步可选的,所述分布模块62,还用于根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。Further optionally, the
本发明实施例还可以实现子资源池中的存储设备的热插/拔,使得各个子资源池中的存储设备出现损坏、停机维护等非正常运行情况时,可以保证储存在子资源池中的数据不丢失,使得分布式存储系统在运行时可以在保证数据稳定的情况下随时排除不能用的存储设备,从而进一步提高了数据的安全性。本发明实施例提供的修复数据的装置,能够将所有的存储设备划分为至少二个子资源池,并将数据储存在所划分出的子资源池中,由于各个子资源池中的数据是独立的,只有同一个子资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个子资源池,并将数据储存在所划分出的多个子资源池中,只有当同一个子资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个子资源池中的2个存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The embodiment of the present invention can also realize the hot plug/unplug of the storage devices in the sub-resource pools, so that when the storage devices in each sub-resource pool are damaged, shut down for maintenance and other abnormal operation conditions, the data stored in the sub-resource pools can be guaranteed Data is not lost, so that the distributed storage system can eliminate unusable storage devices at any time while ensuring data stability during operation, thereby further improving data security. The device for repairing data provided by the embodiment of the present invention can divide all storage devices into at least two sub-resource pools, and store data in the divided sub-resource pools, because the data in each sub-resource pool is independent , the data will be completely damaged only if two storage devices in the same sub-resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. Since the embodiment of the present invention can divide multiple sub-resource pools, and store data in the multiple sub-resource pools divided In a sub-resource pool, only when two storage devices in the same sub-resource pool are damaged, the data will be completely damaged, because the probability of damage to two storage devices in the same sub-resource pool is less than that in the prior art when there is only one resource In the case of a pool, there is a probability of damaging two storage devices, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
再一方面,本发明实施例提供了一种储存数据的存储系统,如图7所示,所述存储系统中包括用于存储数据的存储区域74,所述存储区域74被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述存储系统包括:处理器71、通信接口72、总线73,的所述处理器71、所述通信接口72和所述存储区域74中的所有的存储设备,通过所述总线73完成相互间的通信,其中:In another aspect, an embodiment of the present invention provides a storage system for storing data. As shown in FIG. 7 , the storage system includes a storage area 74 for storing data, and the storage area 74 is logically divided into at least 2 Sub-resource pools, each sub-resource pool includes adjacent N storage devices, N≥2, the storage system includes: a processor 71, a communication interface 72, a bus 73, the processor 71, the communication interface 72 and all storage devices in the storage area 74 complete mutual communication through the bus 73, wherein:
所述处理器71,用于将需要存储的数据划分为R个切片,通过所述通信接口72将R个所述切片分布到划分的所述子资源池中。The processor 71 is configured to divide the data to be stored into R slices, and distribute the R slices to the divided sub-resource pools through the communication interface 72 .
其中,需要存储的数据期望生成的副本数量小于N,R≥2。Wherein, the number of copies of the data to be stored is expected to be less than N, and R≥2.
所述处理器71,还用于对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中。The processor 71 is further configured to generate a duplicate slice for each slice distributed to each sub-resource pool, and store the duplicate slice in the sub-resource pool to which the slice belongs.
其中,副本切片与需要存储的数据的相同的切片相互对应。Wherein, the copy slice corresponds to the same slice of the data to be stored.
进一步的,所述处理器71,还用于将所述副本切片存储到对应所述切片所属的子资源池中的存储设备上。Further, the processor 71 is further configured to store the duplicate slice on a storage device corresponding to the sub-resource pool to which the slice belongs.
其中,每个所述副本切片存储的存储设备与对应切片存储的存储设备不同,数据相同的副本切片存储在不同的存储设备上。Wherein, the storage device for storing each duplicate slice is different from the storage device for corresponding slice storage, and duplicate slices with the same data are stored on different storage devices.
可选的,所述处理器71,还用于将所述划分的子资源池再次划分为不同级别子资源池集合。Optionally, the processor 71 is further configured to further divide the divided sub-resource pools into sub-resource pool sets of different levels.
其中,每个子资源池集合中包括至少一个子资源池,不同级别的子资源池集合中每个切片的期望生成的副本切片数量不同。Wherein, each set of sub-resource pools includes at least one sub-resource pool, and the number of duplicate slices expected to be generated for each slice in the set of sub-resource pools of different levels is different.
所述处理器71,还用于按照子资源池所属子资源池集合的级别所对应的每个所述切片期望生成的副本切片数量,将分布到子资源池中的每个所述切片生成对应数量的副本切片。The processor 71 is further configured to, according to the number of duplicate slices expected to be generated by each slice corresponding to the level of the set of sub-resource pools to which the sub-resource pool belongs, generate corresponding slices distributed to each of the sub-resource pools Number of replica slices.
进一步可选的,所述处理器71,还用于在将所述副本切片存储在对应所述切片所属的子资源池中后,针对任一个所述子资源池,当减少所述子资源池中的存储设备的数量时,通过所述通信接口72将所减少的存储设备中的所述切片分配到所述子资源池中的其他存储设备上。Further optionally, the processor 71 is further configured to, after storing the copy slice in the sub-resource pool to which the slice belongs, for any one of the sub-resource pools, when reducing the sub-resource pool When the number of storage devices in the sub-resource pool is reduced, the slices in the reduced storage devices are allocated to other storage devices in the sub-resource pool through the communication interface 72 .
其中,子资源池中的剩余的存储设备数量大于所述期望生成的副本数。Wherein, the number of remaining storage devices in the sub-resource pool is greater than the number of copies expected to be generated.
并列的可选的,所述处理器71,还用于在将所述副本切片存储在对应所述切片所属的子资源池中后,针对任一个所述子资源池,当增加所述子资源池中的存储设备的数量时,通过所述通信接口72将该子资源池中的所述切片的副本切片重新分配到该子资源池中的存储设备上。Optionally in parallel, the processor 71 is further configured to, after storing the copy slice in the sub-resource pool to which the slice belongs, for any one of the sub-resource pools, when adding the sub-resource When the number of storage devices in the pool is equal, the copy slices of the slices in the sub-resource pool are reallocated to the storage devices in the sub-resource pool through the communication interface 72 .
更进一步可选的,所述处理器71,还用于根据R个切片中数据的重要程度,将R个切片分布到不同级别的子资源池中。Further optionally, the processor 71 is further configured to distribute the R slices to sub-resource pools of different levels according to the importance of data in the R slices.
本发明实施例提供的修复数据的存储系统,能够将存储系统中的所有的存储设备划分为至少二个资源池,并将数据储存在所划分出的资源池中,由于各个资源池中的数据是独立的,只有同一个资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个资源池,并将数据储存在所划分出的多个资源池中,只有当同一个资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个资源池中的2个存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The repair data storage system provided by the embodiment of the present invention can divide all storage devices in the storage system into at least two resource pools, and store data in the divided resource pools. Since the data in each resource pool It is independent, and the data will be completely damaged only if two storage devices in the same resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. The embodiment of the present invention can divide multiple resource pools and store data in the divided multiple resource pools. In a resource pool, only when two storage devices in the same resource pool are damaged, the data will be completely damaged, because the probability of two storage devices in the same resource pool being damaged is less than that in the prior art when there is only one resource In the case of a pool, there is a probability of damaging two storage devices, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
再一方面,本发明实施例提供了一种修复数据的计算机程序产品,用于存储系统,所述存储系统中包括用于存储数据的资源池,所述资源池被逻辑划分为至少2个子资源池,每个子资源池中包括相邻的N个存储设备,N≥2,所述计算机程序产品包括存储了程序代码的计算机可读存储介质,所述程序代码包括的指令用于:In yet another aspect, an embodiment of the present invention provides a computer program product for repairing data, which is used in a storage system, and the storage system includes a resource pool for storing data, and the resource pool is logically divided into at least two sub-resources A pool, each sub-resource pool includes adjacent N storage devices, N≥2, the computer program product includes a computer-readable storage medium storing program code, and the program code includes instructions for:
将需要存储的数据划分为R个切片,将R个所述切片分布到划分的所述子资源池中,所述需要存储的数据期望生成的副本数量小于N,R≥2。The data to be stored is divided into R slices, and the R slices are distributed to the divided sub-resource pools, the number of copies of the data to be stored is expected to be less than N, and R≥2.
对分布到每个子资源池中的每个所述切片生成副本切片,并将所述副本切片存储在对应所述切片所属的子资源池中,所述副本切片与数据相同的所述切片相互对应。Generate a copy slice for each slice distributed to each sub-resource pool, and store the copy slice in the sub-resource pool to which the slice belongs, and the copy slice corresponds to the slice with the same data .
本发明实施例提供的修复数据的计算机程序产品,能够将存储系统中的所有的存储设备划分为至少二个资源池,并将数据储存在所划分出的资源池中,由于各个资源池中的数据是独立的,只有同一个资源池中的2个存储设备损坏,数据才会完全损坏。现有技术中在只有一个资源池的情况下,损坏2个存储设备就会造成数据完全损坏的情况,本发明实施例由于能够划分出多个资源池,并将数据储存在所划分出的多个资源池中,只有当同一个资源池中的2个存储设备损坏时,数据才会完全损坏,由于同一个资源池中的2个存储设备损坏的几率,小于现有技术中在只有一个资源池的情况下损坏2个存储设备的几率,从而使得本发明实施例相对于现有技术能够降低了数据完全损坏的几率,从而提高了数据的安全性。The computer program product for repairing data provided by the embodiments of the present invention can divide all storage devices in the storage system into at least two resource pools, and store data in the divided resource pools. The data is independent, and the data will be completely damaged only if two storage devices in the same resource pool are damaged. In the prior art, when there is only one resource pool, damage to two storage devices will result in complete data damage. The embodiment of the present invention can divide multiple resource pools and store data in the divided multiple resource pools. In a resource pool, only when two storage devices in the same resource pool are damaged, the data will be completely damaged, because the probability of two storage devices in the same resource pool being damaged is less than that in the prior art when there is only one resource In the case of a pool, there is a probability of damaging two storage devices, so that the embodiment of the present invention can reduce the probability of complete data damage compared with the prior art, thereby improving data security.
本说明书中的各个实施例均采用递进的方式描述,各个实施例之间相同相似的部分互相参见即可,每个实施例重点说明的都是与其他实施例的不同之处。尤其,对于设备实施例而言,由于其基本相似于方法实施例,所以描述得比较简单,相关之处参见方法实施例的部分说明即可。Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于计算机可读取存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。其中,所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(Random Access Memory,RAM)等。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. During execution, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM) or a random access memory (Random Access Memory, RAM), etc.
以上所述,仅为本发明的具体实施方式,但本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到的变化或替换,都应涵盖在本发明的保护范围之内。因此,本发明的保护范围应该以权利要求的保护范围为准。The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. All should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.
Claims (19)
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2012/087922 WO2014101144A1 (en) | 2012-12-28 | 2012-12-28 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103384550A true CN103384550A (en) | 2013-11-06 |
| CN103384550B CN103384550B (en) | 2016-05-25 |
Family
ID=49492132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201280003656.8A Active CN103384550B (en) | 2012-12-28 | 2012-12-28 | The method of storage data and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103384550B (en) |
| WO (1) | WO2014101144A1 (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106416405A (en) * | 2014-06-27 | 2017-02-15 | 夏普株式会社 | Resource pool access for device to device communications |
| CN106792692A (en) * | 2016-12-27 | 2017-05-31 | 兴唐通信科技有限公司 | A kind of physics dicing method based on SDN technologies |
| WO2017113280A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Distributed storage system and metadata managing method |
| CN107925587A (en) * | 2015-08-21 | 2018-04-17 | 华为技术有限公司 | Method and apparatus for network section |
| WO2018176998A1 (en) * | 2017-03-29 | 2018-10-04 | 华为技术有限公司 | Data storage method and device |
| CN110275669A (en) * | 2018-03-15 | 2019-09-24 | 杭州海康威视数字技术股份有限公司 | Date storage method, device and electronic equipment |
| CN110532271A (en) * | 2019-08-30 | 2019-12-03 | 北京浪潮数据技术有限公司 | Distributed storage method, control system and the device realized based on DHT algorithm |
| CN112181913A (en) * | 2020-10-10 | 2021-01-05 | 上海威固信息技术股份有限公司 | Computer storage system and storage method based on block chain |
| US11032858B2 (en) | 2017-03-28 | 2021-06-08 | Huawei Technologies Co., Ltd. | Network access method, terminal, access network, and core network |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998038568A1 (en) * | 1997-02-27 | 1998-09-03 | International Business Machines Corporation | Transformational raid for hierarchical storage management system |
| CN101459597A (en) * | 2007-12-14 | 2009-06-17 | 华为技术有限公司 | Method and system for using logic resource |
| US20100040058A1 (en) * | 2008-08-15 | 2010-02-18 | Vmware, Inc. | Systems and Methods of Configuring a Resource Pool as a Network End Point |
| CN102073560A (en) * | 2011-01-17 | 2011-05-25 | 北京深思洛克软件技术股份有限公司 | Data backup method and device |
| CN102123167A (en) * | 2010-12-31 | 2011-07-13 | 成都市华为赛门铁克科技有限公司 | Distributed file system, and data storage processing method and data storage processing device thereof |
| CN102843403A (en) * | 2011-06-23 | 2012-12-26 | 盛大计算机(上海)有限公司 | File processing method based on distributed file system, system, and client |
-
2012
- 2012-12-28 WO PCT/CN2012/087922 patent/WO2014101144A1/en not_active Ceased
- 2012-12-28 CN CN201280003656.8A patent/CN103384550B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO1998038568A1 (en) * | 1997-02-27 | 1998-09-03 | International Business Machines Corporation | Transformational raid for hierarchical storage management system |
| CN101459597A (en) * | 2007-12-14 | 2009-06-17 | 华为技术有限公司 | Method and system for using logic resource |
| US20100040058A1 (en) * | 2008-08-15 | 2010-02-18 | Vmware, Inc. | Systems and Methods of Configuring a Resource Pool as a Network End Point |
| CN102123167A (en) * | 2010-12-31 | 2011-07-13 | 成都市华为赛门铁克科技有限公司 | Distributed file system, and data storage processing method and data storage processing device thereof |
| CN102073560A (en) * | 2011-01-17 | 2011-05-25 | 北京深思洛克软件技术股份有限公司 | Data backup method and device |
| CN102843403A (en) * | 2011-06-23 | 2012-12-26 | 盛大计算机(上海)有限公司 | File processing method based on distributed file system, system, and client |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106416405A (en) * | 2014-06-27 | 2017-02-15 | 夏普株式会社 | Resource pool access for device to device communications |
| CN107925587A (en) * | 2015-08-21 | 2018-04-17 | 华为技术有限公司 | Method and apparatus for network section |
| US10644955B2 (en) | 2015-08-21 | 2020-05-05 | Huawei Technologies Co., Ltd. | Method and apparatus for network slicing |
| CN107925587B (en) * | 2015-08-21 | 2020-04-03 | 华为技术有限公司 | Method and apparatus for network slicing |
| WO2017113280A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Distributed storage system and metadata managing method |
| CN106792692A (en) * | 2016-12-27 | 2017-05-31 | 兴唐通信科技有限公司 | A kind of physics dicing method based on SDN technologies |
| CN106792692B (en) * | 2016-12-27 | 2019-11-05 | 兴唐通信科技有限公司 | A kind of physics dicing method based on SDN technology |
| US11032858B2 (en) | 2017-03-28 | 2021-06-08 | Huawei Technologies Co., Ltd. | Network access method, terminal, access network, and core network |
| US20200028911A1 (en) * | 2017-03-29 | 2020-01-23 | Huawei Technologies Co., Ltd. | Data Storage Method and Apparatus |
| CN108667867A (en) * | 2017-03-29 | 2018-10-16 | 华为技术有限公司 | Date storage method and device |
| WO2018176998A1 (en) * | 2017-03-29 | 2018-10-04 | 华为技术有限公司 | Data storage method and device |
| US11575748B2 (en) | 2017-03-29 | 2023-02-07 | Huawei Technologies Co., Ltd. | Data storage method and apparatus for combining different data distribution policies |
| US10972542B2 (en) | 2017-03-29 | 2021-04-06 | Huawei Technologies Co., Ltd. | Data storage method and apparatus |
| CN110275669A (en) * | 2018-03-15 | 2019-09-24 | 杭州海康威视数字技术股份有限公司 | Date storage method, device and electronic equipment |
| CN110275669B (en) * | 2018-03-15 | 2022-11-25 | 杭州海康威视数字技术股份有限公司 | Data storage method and device and electronic equipment |
| CN110532271A (en) * | 2019-08-30 | 2019-12-03 | 北京浪潮数据技术有限公司 | Distributed storage method, control system and the device realized based on DHT algorithm |
| CN112181913B (en) * | 2020-10-10 | 2021-05-07 | 上海威固信息技术股份有限公司 | Block chain-based computer storage method and system |
| CN112181913A (en) * | 2020-10-10 | 2021-01-05 | 上海威固信息技术股份有限公司 | Computer storage system and storage method based on block chain |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2014101144A1 (en) | 2014-07-03 |
| CN103384550B (en) | 2016-05-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103384550B (en) | The method of storage data and device | |
| CN110413201B (en) | Method, apparatus and computer program product for managing a storage system | |
| US10977124B2 (en) | Distributed storage system, data storage method, and software program | |
| CN107844268B (en) | Data distribution method, data storage method, related device and system | |
| CN102405460B (en) | Virtual storage system and operation method thereof | |
| KR101813431B1 (en) | Method and system for distributing data in a distributed storage system | |
| US20170075741A1 (en) | Prioritizing Data Reconstruction in Distributed Storage Systems | |
| US11144396B1 (en) | Raid reliability with a provisional spare disk | |
| US10817376B2 (en) | RAID with heterogeneous combinations of segments | |
| CN110383251B (en) | Storage system, computer-readable recording medium, and method for controlling system | |
| US10353787B2 (en) | Data stripping, allocation and reconstruction | |
| CN109144406B (en) | Metadata storage method, system and storage medium in distributed storage system | |
| CN103064765A (en) | Method and device for data recovery and cluster storage system | |
| US11327668B1 (en) | Predictable member assignment for expanding flexible raid system | |
| JP7140688B2 (en) | Data storage system and method of accessing key-value pair objects | |
| US20160342360A1 (en) | Raid random distribution scheme | |
| US11314608B1 (en) | Creating and distributing spare capacity of a disk array | |
| CN109725823A (en) | Method and apparatus for managing mixing storage disk array | |
| US11860746B2 (en) | Resilient data storage system with efficient space management | |
| CN108228085A (en) | For managing the method and apparatus of storage system | |
| US20210011640A1 (en) | Scaling raid-based storage by redistributing splits | |
| US11327666B2 (en) | RAID member distribution for granular disk array growth | |
| CN108733307B (en) | Storage management method, apparatus and computer readable medium | |
| US11481275B2 (en) | Managing reconstruction of a malfunctioning disk slice | |
| US20210124517A1 (en) | Method, device and computer program product for storing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |