CN108089814B - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN108089814B CN108089814B CN201611038740.XA CN201611038740A CN108089814B CN 108089814 B CN108089814 B CN 108089814B CN 201611038740 A CN201611038740 A CN 201611038740A CN 108089814 B CN108089814 B CN 108089814B
- Authority
- CN
- China
- Prior art keywords
- memory
- data
- identification information
- space utilization
- utilization rate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0611—Improving I/O performance in relation to response time
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0685—Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明实施例公开了一种数据存储方法，所述方法包括：获取待处理数据的流量值；基于所述流量值和预设选择策略，确定目标存储器并存储所述待处理数据至所述目标存储器中；其中，所述目标存储器包括第一存储器集合或者第二存储器集合中的一个存储器；当预设周期到来时，获取所述第一存储器集合的空间使用率和所述第二存储器集合的空间使用率；根据所述第一存储器集合的空间使用率和所述第二存储器集合的空间使用率，确定是否迁移所述第一存储器集合中的数据至所述第二存储器集合中。本发明实施例同时还公开了一种数据存储装置。
An embodiment of the present invention discloses a data storage method, the method includes: acquiring a flow value of data to be processed; based on the flow value and a preset selection strategy, determining a target memory and storing the to-be-processed data to the target In the memory; wherein, the target memory includes one memory in the first memory set or the second memory set; when a preset period comes, obtain the space usage rate of the first memory set and the second memory set. space usage rate; according to the space usage rate of the first memory set and the space usage rate of the second memory set, determine whether to migrate the data in the first memory set to the second memory set. The embodiment of the present invention also discloses a data storage device.
Description
Technical Field
The invention relates to a cloud storage technology in the field of cloud computing, in particular to a data storage method and device.
Background
With the rapid development of cloud computing, the importance of the cloud storage technology to the work and life of people is increasingly prominent. For a cloud storage system, data writing speed is an important factor affecting its performance. In the data writing process, the data storage medium limits the improvement of the data writing speed of the cloud storage system. Many cloud storage providers provide a storage scheme that simultaneously uses a storage medium with a fast write speed, a small storage capacity, and a high price, such as a Solid State Drive (SSD), and a storage medium with a large storage capacity, a slow write speed, and a low price, such as a Hard Disk Drive (HDD), in a storage server of a cloud storage system. In the existing cloud storage system, the writing speed of data is improved mainly by technologies such as compressing and transmitting the data to be stored or blocking and transmitting the data to be stored, or the load of the storage system is balanced by monitoring the load condition of each storage node.
However, in the prior art, only the improvement of the data writing speed is considered during data writing, and the problem of unbalanced load of the storage space of the storage medium is not considered; or only the load balance between the storage spaces of the storage medium is considered, the data writing speed is not considered, and the load balance between the data writing speed and the storage spaces of the storage medium cannot be considered at the same time.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide a data storage method and apparatus, which solve the problem in the prior art that the data writing speed and the load of the storage space of the storage medium cannot be balanced at the same time, and implement optimization of the data writing speed by the characteristic that the writing speed is different when the data is stored in different storage media, and ensure balanced use of the storage space.
In order to achieve the above purpose, the technical solution of the embodiment of the present invention is realized as follows:
a method of data storage, the method comprising:
acquiring a flow value of data to be processed;
determining a target memory and storing the data to be processed into the target memory based on the flow value and a preset selection strategy; wherein the target memory comprises one of a first set of memories or a second set of memories;
when a preset period comes, acquiring the space utilization rate of the first memory set and the space utilization rate of the second memory set;
and determining whether to migrate the data in the first memory set into the second memory set according to the space utilization rate of the first memory set and the space utilization rate of the second memory set.
Optionally, the determining a target memory and storing the upload data in the target memory based on the flow value and a preset selection policy includes:
selecting any one memory from the first memory set and the second memory set to obtain a first memory;
acquiring the selection probability of a first memory;
obtaining a reference value based on a preset selection rule, and comparing the magnitude relation between the reference value and the selection probability of the first memory;
and if the reference value is less than or equal to the selection probability of the first memory, selecting the first memory as the target memory, and storing the uploaded data to the target memory.
Optionally, the method further includes:
and if the reference value is greater than the selection probability of the first memory, executing the selection of any one memory from the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
Optionally, the obtaining the selection probability of the first memory includes:
acquiring the writing speed and the space utilization rate of the first memory;
obtaining a total number of memories in the first set of memories and the second set of memories;
and calculating by adopting a preset selection strategy to obtain the selection probability of the first memory based on the flow value, the writing speed of the first memory, the total number and the space utilization rate of the first memory.
Optionally, the determining whether to migrate data in the first memory set to the second memory set according to the space usage of the first memory set and the space usage of the second memory set includes:
judging the size relation between the space utilization rate of each memory in the first memory set and a first threshold value;
if a memory with the space utilization rate larger than or equal to the first threshold exists in the first memory set, counting the identification information of the memory with the space utilization rate larger than or equal to the first threshold in the first memory set to obtain a first identification information set;
judging the size relation between the space utilization rate of the memories in the second memory set and a second threshold value;
if a memory with the space utilization rate smaller than the second threshold exists in the second memory set, counting the identification information of the memory with the space utilization rate smaller than the second threshold in the second memory set to obtain a second identification information set;
and based on the first identification information set and the second identification information set, migrating the data in the memory corresponding to the first identification information set to the memory corresponding to the second identification information set according to a data migration rule.
Optionally, the migrating, based on the first identification information set and the second identification information set, data in a memory corresponding to the first identification information set to a memory corresponding to the second identification information set according to a data migration rule includes:
acquiring a communication interface idle memory in a memory corresponding to the first identification information set, and determining a data packet to be migrated of the communication interface idle memory corresponding to the first identification information set according to a data migration rule;
and acquiring a communication interface idle memory corresponding to the second identification information set, and transferring a data packet to be transferred in the communication interface idle memory corresponding to the first identification information set to the communication interface idle memory corresponding to the second identification information set.
Optionally, the first set of memories is a set of SSD memories, and the second set of memories is a set of HDD memories.
A data storage device, the device comprising: the device comprises a first acquisition unit, a first processing unit, a second acquisition unit and a second processing unit; wherein:
the first acquisition unit is used for acquiring a flow value of data to be processed;
the first processing unit is used for determining a target memory and storing the uploaded data into the target memory based on the flow value and a preset selection strategy; wherein the target memory comprises one of a first set of memories or a second set of memories;
the second obtaining unit is configured to obtain a space utilization rate of the first memory set and a space utilization rate of the second memory set when a preset period comes;
the second processing unit is configured to determine whether to migrate data in the first memory set to the second memory set according to the space utilization of the first memory set and the space utilization of the second memory set.
Optionally, the first processing unit includes: the system comprises a first processing module, a first obtaining module, a second processing module and a third processing module; wherein:
the first processing module is used for selecting any one memory from the first memory set and the second memory set to obtain a first memory;
the first obtaining module is used for obtaining the selection probability of the first memory;
the second processing module is used for obtaining a reference value based on a preset selection rule and comparing the magnitude relation between the reference value and the selection probability of the first memory;
and the third processing module is configured to select the first memory as the target memory and store the upload data in the target memory if the reference value is less than or equal to the selection probability of the first memory.
Optionally, the first processing unit further includes: a fourth processing module; wherein:
the fourth processing module is configured to, if the reference value is greater than the selection probability of the first memory, execute the selection of any one of the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
Optionally, the first obtaining module is specifically configured to:
acquiring the writing speed and the space utilization rate of the first memory;
obtaining a total number of memories in the first set of memories and the second set of memories;
and calculating by adopting a preset selection strategy to obtain the selection probability of the first memory based on the flow value, the writing speed of the first memory, the total number and the space utilization rate of the first memory.
Optionally, the second processing unit includes: the device comprises a judgment module, a fifth processing module and a migration module; wherein:
the judging module is used for judging the size relationship between the space utilization rate of each memory in the first memory set and a first threshold value;
the fifth processing module is configured to, if a memory with a space utilization rate greater than or equal to the first threshold exists in the first memory set, count identification information of a memory with a space utilization rate greater than or equal to the first threshold in the first memory set to obtain a first identification information set;
the judging module is further configured to judge a size relationship between a space utilization rate of the memories in the second memory set and a second threshold;
the fifth processing module is further configured to, if a memory with a space utilization rate smaller than the second threshold exists in the second memory set, count identification information of the memory with a space utilization rate smaller than the second threshold in the second memory set to obtain the second identification information set;
and the migration module is used for migrating the data in the storage corresponding to the first identification information set to the storage corresponding to the second identification information set according to a data migration rule based on the first identification information set and the second identification information set.
Optionally, the migration module is specifically configured to:
acquiring a communication interface idle memory in a memory corresponding to the first identification information set, and determining a data packet to be migrated of the communication interface idle memory corresponding to the first identification information set according to a data migration rule;
and acquiring a communication interface idle memory corresponding to the second identification information set, and transferring a data packet to be transferred in the communication interface idle memory corresponding to the first identification information set to the communication interface idle memory corresponding to the second identification information set.
According to the data storage method and device provided by the embodiment of the invention, the flow value of the data to be processed is obtained, the target memory is determined and the data to be processed is stored in the target memory based on the flow value and the preset selection strategy, then when the preset period comes, the space utilization rate of the first memory set and the space utilization rate of the second memory set are obtained, and finally whether the data value in the first memory set is transferred to the second memory set is determined according to the space utilization rate of the first memory set and the space utilization rate of the second memory set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
Drawings
Fig. 1 is a schematic flowchart of a data storage method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating another data storage method according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating another data storage method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a data storage system according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;
FIG. 6 is a schematic structural diagram of another data storage device according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of another data storage device according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a data storage device according to another embodiment of the present invention;
fig. 9 is a system diagram of a data migration flow according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
An embodiment of the present invention provides a data storage method, which is shown in fig. 1 and includes the following steps:
Specifically, the step 101 of obtaining the flow value of the data to be processed may be implemented by a data storage device. The data to be processed can be data which needs to be stored and is uploaded to a cloud storage system by a user; the flow value of the data to be processed may be a flow size value of data to be stored, which is uploaded to the cloud storage system by a user. The data storage may be a cloud storage system.
And 102, determining a target memory and storing the data to be processed into the target memory based on the flow value and a preset selection strategy.
Wherein the target memory comprises one of the first set of memories or the second set of memories.
Specifically, the step 102 may be implemented by a data storage device, based on the flow value and the preset selection policy, determining the target memory and storing the data to be processed in the target memory. The preset selection policy may be a method for determining a target memory, and the preset selection policy may have the following characteristics: the larger the space utilization rate of the memory is, the smaller the probability that the memory is selected is; the larger the flow value is, the faster the writing speed of the memory is, and the greater the probability that the memory is selected is; the smaller the flow value, the slower the write speed of the memory, and the greater the probability that the memory will be selected. Wherein the first set of memory may be a set of SSD memory and the second set of memory may be a set of HDD memory.
And 103, acquiring the space utilization rate of the first memory set and the space utilization rate of the second memory set when a preset period comes.
Specifically, when the preset period comes, the step 103 may be implemented by the data storage device to acquire the space usage rate of the first memory set and the space usage rate of the second memory set. The data storage device can acquire the space utilization rates of the first storage set and the second storage set every preset period and record the space utilization rates of the first storage set and the second storage set. For example, the preset period may be 1 minute or 5 minutes, and may also be set according to the user requirement.
And 104, determining whether to migrate the data in the first storage set to the second storage set according to the space utilization rate of the first storage set and the space utilization rate of the second storage set.
Specifically, step 104 determines whether to migrate data in the first memory set to the second memory set according to the space usage of the first memory set and the space usage of the second memory set, which may be implemented by the data storage device. When the size relationship of the space utilization of the first memory conforms to a first predetermined value, the data in the first memory may be migrated to the second memory, wherein the size relationship of the space utilization of the second memory also needs to conform to a second predetermined value.
The data storage method provided by the embodiment of the invention comprises the steps of determining a target storage and storing data to be processed to the target storage by acquiring a flow value of the data to be processed and based on the flow value and a preset selection strategy, then acquiring the space utilization rate of a first storage set and the space utilization rate of a second storage set when a preset period comes, and finally determining whether to migrate the data value in the first storage set to the second storage set according to the space utilization rate of the first storage set and the space utilization rate of the second storage set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
An embodiment of the present invention provides a data storage method, which is shown in fig. 2 and includes the following steps:
Specifically, the first memory set is an SSD memory set, and the second memory set is an HDD memory set. The cloud storage system may randomly select one memory from the SSD memory set and the HDD memory set as the first memory, which may be an HDD memory, for example.
Specifically, the selection probability of the first storage may be obtained by computing by the cloud storage system. For example, the cloud storage system has a probability of 0.3 of selecting to acquire HDD memory.
Specifically, the preset selection rule may be a selection algorithm, which may be used to obtain a reference value that can be compared with the selection probability of the first memory.
Wherein, the data storage device in step 204 obtains a reference value based on a preset selection rule, and after comparing the magnitude relationship between the reference value and the selection probability of the first memory, the data storage device may choose to execute step 205 or step 206, if the reference value is less than or equal to the selection probability of the first memory, the data storage device may choose to execute step 205, and if the reference value is greater than the selection probability of the first memory, the data storage device may choose to execute step 206;
in step 205, if the reference value is less than or equal to the selection probability of the first memory, the data storage device selects the first memory as the target memory and stores the upload data in the target memory.
Specifically, for example, when the reference value obtained according to the preset selection rule is 0.2 and is less than 0.3 of the selection probability of the HDD memory obtained by the cloud storage system, the cloud storage system may determine that the target memory is the first memory HDD memory, and store the data uploaded by the user in the target memory HDD memory.
And step 206, if the reference value is greater than the selection probability of the first memory, the data storage device selects any one memory from the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
Specifically, for example, when the reference value obtained according to the preset selection rule is 0.4, which is greater than the probability of selection of the HDD memory of 0.3, the HDD memory cannot be selected as the target memory. The cloud storage system may reselect a memory from the SSD memory set or the HDD memory set as the first memory, for example, the memory may be an SSD memory, calculate a selection probability of the reselected SSD memory to be 0.6, and may obtain a reference value, for example, 0.5, again according to a preset selection rule, and then compare the selection probability of the reselected SSD memory 0.6 with the reference value 0.5, and may determine that the target memory is the reselected SSD memory; if a reference value 0.8 is obtained again according to the preset rule, the newly selected SSD memory is not the target memory, one memory needs to be selected from the SSD memory set and the HDD memory set again to serve as the first memory, the newly obtained first memory is analyzed, the steps are repeated until the target memory is determined, and finally the data uploaded by the user is stored in the target memory.
And step 207, when the preset period comes, the data storage device acquires the space utilization rate of the first memory set and the space utilization rate of the second memory set.
Specifically, when the preset period comes, the cloud storage system obtains the space utilization rate of the SSD memory set and the space utilization rate of the HDD memory set.
In particular, the first threshold may be an empirical value obtained by the user during use of the first set of memories. For example, the first threshold may be 0.8, and the cloud storage system may determine a size relationship between the space usage of each memory in the SSD memory set and the first threshold of 0.8.
Specifically, if a memory with a space utilization rate greater than or equal to 0.8 exists in the SSD memory set, the identification information of the SSD memory with a space utilization rate greater than or equal to the first threshold value 0.8 in the SSD memory set is counted to obtain a first identification information set, and the SSD memory to which the data needs to be migrated is determined.
In particular, the second threshold may be used to select a memory in the second set of memories that may receive data of the memory migration in the first set of memories. For example, the second threshold may be 0.2.
Specifically, if there is an HDD memory with a space utilization rate smaller than the second threshold 0.2 in the HDD memory set of the second memory set, the identification information of the HDD memory is counted to obtain the second identification information set.
Specifically, the data migration rule may be configured to determine how much data needs to be migrated in the memory corresponding to the first identification information set, and migrate the data that needs to be migrated to the memory corresponding to the second identification information set that meets the requirement under the condition that the memory corresponding to the first identification information set meets the requirement. For example, the data migration rule may be that after the SSD memory migrates the data, the space usage rate of the SSD memory is smaller than the space usage rate of the entire cloud storage system and is also smaller than the space usage rate threshold of the SSD memory by 0.6.
The data storage method provided by the embodiment of the invention comprises the steps of determining a target storage and storing data to be processed to the target storage by acquiring a flow value of the data to be processed and based on the flow value and a preset selection strategy, then acquiring the space utilization rate of a first storage set and the space utilization rate of a second storage set when a preset period comes, and finally determining whether to migrate the data value in the first storage set to the second storage set according to the space utilization rate of the first storage set and the space utilization rate of the second storage set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
An embodiment of the present invention provides a data storage method, which is shown in fig. 3 and includes the following steps:
Specifically, the first storage set may be an SSD storage set composed of 10 blocks of SSD storage with a capacity of 500G, the second storage set may be an HDD storage set composed of 90 blocks of HDD storage with a capacity of 2T, and the cloud storage system randomly selects one storage from the 10 blocks of SSD storage set and the 90 blocks of HDD storage set, for example, the SSD storage may be used as the first storage.
And 305, calculating by the data storage device through a preset selection strategy based on the flow value, the writing speed and the total number of the first memory and the space utilization rate of the first memory to obtain the selection probability of the first memory.
Specifically, the preset selection policy may be implemented by using formula (1):
P1(x)＝sig mod(λ*|x+μ|+γ)+v (2)
wherein T is the flow value of the data to be processed, N is the total number of the memories in the first memory set and the second memory set, SIIs the write speed of the first memory, RIThe space utilization rate I of the first memory is the first memory; p1(x) Can be realized by the formulas (2) and (4), P2(x) The function may be implemented by equations (3) and (4); λ, μ, γ, and ν in the formula (2) may be adjustment coefficients, which are related to space usage rates of a memory and a memory constituting the cloud storage system, and a convergence requirement of the calculation method of the formula (4), and may be adjusted as needed, and have no fixed value, and for example, λ ═ 3 pi, μ ═ 0.75, γ ═ 1, and ν ═ 0.15 may be adopted in this embodiment; α and β in the formula (3) may be adjustment coefficients, which are related to safety capacity limitation of the memory, and may be adjusted as needed, for example, α ═ 2 pi and β ═ pi may be adopted in the present embodiment.
And step 306, when the preset period comes, the data storage device acquires the space utilization rate of the first memory set and the space utilization rate of the second memory set.
And 307, the data storage device obtains a reference value based on a preset selection rule, and compares the magnitude relation between the reference value and the selection probability of the first memory.
Wherein, in step 307, the data storage device obtains a reference value based on a preset selection rule, and after comparing the magnitude relationship between the reference value and the selection probability of the first memory, the data storage device may selectively perform step 308 or step 309, if the reference value is less than or equal to the selection probability of the first memory, the data storage device may selectively perform step 308, and if the reference value is greater than the selection probability of the first memory, the data storage device may selectively perform step 309;
and 308, if the reference value is less than or equal to the selection probability of the first memory, selecting the first memory as a target memory by the data storage device, and storing the uploaded data to the target memory.
In step 311, the data storage device determines a size relationship between the space utilization rate of each memory in the first memory set and the first threshold.
In step 312, if there is a memory with a space utilization rate greater than or equal to the first threshold in the first memory set, the data storage device counts the identification information of the memory with a space utilization rate greater than or equal to the first threshold in the first memory set to obtain a first identification information set.
Specifically, the cloud storage system acquires an SSD storage with an idle input/output (I/O) interface from a first identification information set of the SSD storage with a space utilization rate greater than 0.8, and determines a data packet to be migrated of the SSD storage with the idle I/O interface according to a data migration rule. Wherein, when the I/O rate is less than 30% of the write rate of the memory, the I/O interface of the memory can be considered idle.
Specifically, after the cloud storage system acquires the HDD storage with the idle I/O interface from the second identification information set of the HDD storage with the space utilization rate less than 0.2, the cloud storage system transfers the determined data packet to be transferred in the SSD storage with the idle I/O interface corresponding to the first identification information set to the HDD storage with the idle I/O interface. In the data migration, only the first data stored for a certain time is stored in the SSD memory of the migrated data.
Specifically, the data storage method provided in the embodiment of the present invention may be applied to the data storage system shown in fig. 4, where the data storage system may include: flow monitoring module, write strategy control module, solid state drive, hard disk drive and data migration module, wherein: the flow monitoring module is used for acquiring a data flow value uploaded to the cloud storage system by a user; the write strategy control module is used for determining a target memory according to the flow value and the write strategy, wherein the target memory can be a solid state drive or a hard disk drive; the data migration module is used for determining a solid state disk drive needing to migrate data and a hard disk drive receiving the data, and is also used for migrating the data needing to be migrated in the solid state disk drive to the hard disk drive capable of receiving the data.
The data storage method provided by the embodiment of the invention comprises the steps of determining a target storage and storing data to be processed to the target storage by acquiring a flow value of the data to be processed and based on the flow value and a preset selection strategy, then acquiring the space utilization rate of a first storage set and the space utilization rate of a second storage set when a preset period comes, and finally determining whether to migrate the data value in the first storage set to the second storage set according to the space utilization rate of the first storage set and the space utilization rate of the second storage set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
The data storage device 4 provided in the embodiment of the present invention can be applied to the data storage method provided in the embodiment corresponding to fig. 1 to 3, and as shown in fig. 5, the data storage device includes: a first acquisition unit 41, a first processing unit 42, a second acquisition unit 43, and a second processing unit 44, wherein:
the first obtaining unit 41 is configured to obtain a flow value of data to be processed.
And the first processing unit 42 is used for determining a target memory and storing the uploaded data into the target memory based on the flow value and a preset selection strategy.
Wherein the target memory comprises one of the first set of memories or the second set of memories.
A second obtaining unit 43, configured to obtain a space usage rate of the first memory set and a space usage rate of the second memory set when a preset period comes.
And the second processing unit 44 is configured to determine whether to migrate data in the first memory set to the second memory set according to the space utilization of the first memory set and the space utilization of the second memory set.
Further, the first set of memory is a set of SSD memory and the second set of memory is a set of HDD memory.
The data storage device provided by the embodiment of the invention determines a target memory and stores data to be processed to the target memory by acquiring a flow value of the data to be processed and based on the flow value and a preset selection strategy, then acquires the space utilization rate of a first memory set and the space utilization rate of a second memory set when a preset period comes, and finally determines whether to migrate the data value in the first memory set into the second memory set according to the space utilization rate of the first memory set and the space utilization rate of the second memory set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
Specifically, referring to fig. 6, the first processing unit includes: a first processing module 421, a first obtaining module 422, a second processing module 423, and a third processing module 424, wherein:
the first processing module 421 is configured to select any one of the first memory set and the second memory set to obtain the first memory.
A first obtaining module 422, configured to obtain the selection probability of the first memory.
The second processing module 423 is configured to obtain a reference value based on a preset selection rule, and compare the magnitude relationship between the reference value and the selection probability of the first memory.
The third processing module 424 is configured to select the first memory as the target memory and store the upload data in the target memory if the reference value is less than or equal to the selection probability of the first memory.
More specifically, the first obtaining module 422 is specifically configured to perform the following steps:
the writing speed and the space utilization rate of the first memory are acquired.
The total number of memories in the first set of memories and the second set of memories is obtained.
And calculating by adopting a preset selection strategy to obtain the selection probability of the first memory based on the flow value, the writing speed and the total number of the first memory and the space utilization rate of the first memory.
Specifically, referring to fig. 7, the first processing unit 42 further includes: a fourth processing module 425, wherein:
a fourth processing module 425, configured to, if the reference value is greater than the selection probability of the first memory, perform selecting any one of the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
Specifically, as shown in fig. 8, the second processing unit 44 includes: a determination module 441, a fifth processing module 442, and a migration module 443, wherein:
the determining module 441 is configured to determine a size relationship between the space utilization of each memory in the first memory set and the first threshold.
The fifth processing module 442 is configured to, if a memory with a space usage rate greater than or equal to a first threshold exists in the first memory set, count identification information of the memory with the space usage rate greater than or equal to the first threshold in the first memory set to obtain a first identification information set.
The determining module 441 is further configured to determine a size relationship between a space usage rate of the memory in the second memory set and a second threshold.
The fifth processing module 442 is further configured to, if a memory with a space usage rate smaller than a second threshold exists in the second memory set, count the identification information of the memory with a space usage rate smaller than the second threshold in the second memory set to obtain a second identification information set.
And a migration module 443, configured to migrate, based on the first identification information set and the second identification information set, data in a memory corresponding to the first identification information set to a memory corresponding to the second identification information set according to the data migration rule.
More specifically, the migration module 443 is specifically configured to perform the following steps:
and acquiring a communication interface idle memory in the memory corresponding to the first identification information set, and determining a data packet to be migrated of the communication interface idle memory corresponding to the first identification information set according to a data migration rule.
And acquiring a communication interface idle memory corresponding to the second identification information set, and transferring the data packet to be transferred in the communication interface idle memory corresponding to the first identification information set to the communication interface idle memory corresponding to the second identification information set.
Specifically, the data storage apparatus provided in the embodiment of the present invention may be applied to the data migration flow system shown in fig. 9, where the data migration flow system may include: solid state drive, hard disk drive and data migration module, wherein: the data migration module may scan the solid state disk drive and the space utilization of the hard disk drive, and when the space utilization of the solid state disk drive is greater than or equal to a first threshold, migrate data in the solid state disk drive to the hard disk drive whose space utilization is less than a second threshold.
It should be noted that in this embodiment, the interaction process between each unit and each module may refer to the interaction process in the data storage method provided in the embodiments corresponding to fig. 1 to 3, and details are not described here.
The data storage device provided by the embodiment of the invention determines a target memory and stores data to be processed to the target memory by acquiring a flow value of the data to be processed and based on the flow value and a preset selection strategy, then acquires the space utilization rate of a first memory set and the space utilization rate of a second memory set when a preset period comes, and finally determines whether to migrate the data value in the first memory set into the second memory set according to the space utilization rate of the first memory set and the space utilization rate of the second memory set; therefore, the target memory is determined through the flow value and the preset selection strategy, the writing speed of the data to be processed written into the target memory is optimized, when the data volume stored in the SSD memory exceeds the threshold value, the data in the SSD memory is transferred to the HDD memory capable of expanding the capacity, so that the storage space of the SSD memory is used in a balanced manner, the problem that the data writing speed and the storage space load of the storage medium cannot be balanced at the same time in the prior art is solved, the optimization of the writing speed of the data is realized through the characteristic that the writing speed is different when the data are stored in different storage media, and the balanced use of the storage space is ensured.
In practical applications, the first obtaining Unit 41, the first Processing Unit 42, the second obtaining Unit 43, the second Processing Unit 44, the first Processing module 421, the first obtaining module 422, the second Processing module 423, the third Processing module 424, the fourth Processing module 425, the determining module 441, the fifth Processing module 442, and the migrating module 443 may be implemented by a Central Processing Unit (CPU), a microprocessor Unit (MPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or the like, which are located in the wireless data transmitting device.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (11)
1. A method of data storage, the method comprising:
acquiring a flow value of data to be processed;
determining a target memory and storing the data to be processed into the target memory based on the flow value and a preset selection strategy, wherein the target memory comprises one of a first memory set or a second memory set;
when a preset period comes, acquiring the space utilization rate of the first memory set and the space utilization rate of the second memory set;
determining whether to migrate data in the first memory set to the second memory set according to the space utilization of the first memory set and the space utilization of the second memory set;
determining a target memory and storing the data to be processed into the target memory based on the flow value and a preset selection strategy, wherein the step of determining the target memory and storing the data to be processed into the target memory comprises the following steps:
selecting any one memory from the first memory set and the second memory set to obtain a first memory; obtaining a selection probability of the first memory based on the flow value; obtaining a reference value based on a preset selection rule, and comparing the magnitude relation between the reference value and the selection probability of the first memory; and if the reference value is less than or equal to the selection probability of the first memory, selecting the first memory as the target memory, and storing the data to be processed to the target memory.
2. The method of claim 1, further comprising:
and if the reference value is greater than the selection probability of the first memory, executing the selection of any one memory from the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
3. The method of claim 1, wherein obtaining the probability of selection of the first memory based on the flow value comprises:
acquiring the writing speed and the space utilization rate of the first memory;
obtaining a total number of memories in the first set of memories and the second set of memories;
and calculating by adopting a preset selection strategy to obtain the selection probability of the first memory based on the flow value, the writing speed of the first memory, the total number and the space utilization rate of the first memory.
4. The method of claim 1, wherein determining whether to migrate data in the first set of memory to the second set of memory based on the space usage of the first set of memory and the space usage of the second set of memory comprises:
judging the size relation between the space utilization rate of each memory in the first memory set and a first threshold value;
if a memory with the space utilization rate larger than or equal to the first threshold exists in the first memory set, counting the identification information of the memory with the space utilization rate larger than or equal to the first threshold in the first memory set to obtain a first identification information set;
judging the size relation between the space utilization rate of the memories in the second memory set and a second threshold value;
if a memory with the space utilization rate smaller than the second threshold exists in the second memory set, counting the identification information of the memory with the space utilization rate smaller than the second threshold in the second memory set to obtain a second identification information set;
and based on the first identification information set and the second identification information set, migrating the data in the memory corresponding to the first identification information set to the memory corresponding to the second identification information set according to a data migration rule.
5. The method according to claim 4, wherein the migrating data in the memory corresponding to the first identification information set to the memory corresponding to the second identification information set according to a data migration rule based on the first identification information set and the second identification information set includes:
acquiring a communication interface idle memory in a memory corresponding to the first identification information set, and determining a data packet to be migrated of the communication interface idle memory corresponding to the first identification information set according to a data migration rule;
and acquiring a communication interface idle memory corresponding to the second identification information set, and transferring a data packet to be transferred in the communication interface idle memory corresponding to the first identification information set to the communication interface idle memory corresponding to the second identification information set.
6. The method of any of claims 1 to 5, wherein the first set of memory is a set of SSD memory and the second set of memory is a set of HDD memory.
7. A data storage device, characterized in that the device comprises: the device comprises a first acquisition unit, a first processing unit, a second acquisition unit and a second processing unit; wherein:
the first acquisition unit is used for acquiring a flow value of data to be processed;
the first processing unit is used for determining a target memory and storing upload data into the target memory based on the flow value and a preset selection strategy, wherein the target memory comprises one of a first memory set or a second memory set;
the second obtaining unit is configured to obtain a space utilization rate of the first memory set and a space utilization rate of the second memory set when a preset period comes;
the second processing unit is used for determining whether to migrate the data in the first memory set into the second memory set according to the space utilization rate of the first memory set and the space utilization rate of the second memory set;
determining a target memory and storing the data to be processed into the target memory based on the flow value and a preset selection strategy, wherein the step of determining the target memory and storing the data to be processed into the target memory comprises the following steps:
selecting any one memory from the first memory set and the second memory set to obtain a first memory; obtaining a selection probability of the first memory based on the flow value; obtaining a reference value based on a preset selection rule, and comparing the magnitude relation between the reference value and the selection probability of the first memory; and if the reference value is less than or equal to the selection probability of the first memory, selecting the first memory as the target memory, and storing the data to be processed to the target memory.
8. The data storage device of claim 7, wherein the first processing unit further comprises: a fourth processing module; wherein:
the fourth processing module is configured to, if the reference value is greater than the selection probability of the first memory, execute the selection of any one of the first memory set and the second memory set to obtain the first memory until the target memory is determined again.
9. The data storage device of claim 7, wherein the first obtaining module is specifically configured to:
acquiring the writing speed and the space utilization rate of the first memory;
obtaining a total number of memories in the first set of memories and the second set of memories;
and calculating by adopting a preset selection strategy to obtain the selection probability of the first memory based on the flow value, the writing speed of the first memory, the total number and the space utilization rate of the first memory.
10. The data storage device of claim 7, wherein the second processing unit comprises: the device comprises a judgment module, a fifth processing module and a migration module; wherein:
the judging module is used for judging the size relationship between the space utilization rate of each memory in the first memory set and a first threshold value;
the fifth processing module is configured to, if a memory with a space utilization rate greater than or equal to the first threshold exists in the first memory set, count identification information of a memory with a space utilization rate greater than or equal to the first threshold in the first memory set to obtain a first identification information set;
the judging module is further configured to judge a size relationship between a space utilization rate of the memories in the second memory set and a second threshold;
the fifth processing module is further configured to, if a memory with a space utilization rate smaller than the second threshold exists in the second memory set, count identification information of the memory with a space utilization rate smaller than the second threshold in the second memory set to obtain a second identification information set;
and the migration module is used for migrating the data in the storage corresponding to the first identification information set to the storage corresponding to the second identification information set according to a data migration rule based on the first identification information set and the second identification information set.
11. The data storage device of claim 10, wherein the migration module is specifically configured to:
acquiring a communication interface idle memory in a memory corresponding to the first identification information set, and determining a data packet to be migrated of the communication interface idle memory corresponding to the first identification information set according to a data migration rule;
and acquiring a communication interface idle memory corresponding to the second identification information set, and transferring a data packet to be transferred in the communication interface idle memory corresponding to the first identification information set to the communication interface idle memory corresponding to the second identification information set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611038740.XA CN108089814B (en) | 2016-11-23 | 2016-11-23 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611038740.XA CN108089814B (en) | 2016-11-23 | 2016-11-23 | Data storage method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108089814A CN108089814A (en) | 2018-05-29 |
| CN108089814B true CN108089814B (en) | 2021-04-06 |
Family
ID=62170886
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611038740.XA Active CN108089814B (en) | 2016-11-23 | 2016-11-23 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108089814B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110196752A (en) * | 2018-06-07 | 2019-09-03 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| WO2020037672A1 (en) * | 2018-08-24 | 2020-02-27 | 深圳市大疆创新科技有限公司 | Method and system for synchronizing data, and movable platform and readable storage medium |
| US11256437B2 (en) | 2018-11-19 | 2022-02-22 | Micron Technology, Inc. | Data migration for memory operation |
| US11182090B2 (en) | 2018-11-19 | 2021-11-23 | Micron Technology, Inc. | Systems, devices, and methods for data migration |
| US11163473B2 (en) | 2018-11-19 | 2021-11-02 | Micron Technology, Inc. | Systems, devices, techniques, and methods for data migration |
| US10782911B2 (en) * | 2018-11-19 | 2020-09-22 | Micron Technology, Inc. | Data migration dynamic random access memory |
| CN110018788B (en) * | 2019-03-01 | 2024-10-15 | 平安科技（深圳）有限公司 | Hierarchical storage method, hierarchical storage device, electronic equipment and computer readable storage medium |
| CN110045924B (en) * | 2019-03-01 | 2022-02-11 | 平安科技（深圳）有限公司 | Hierarchical storage method and device, electronic equipment and computer readable storage medium |
| CN111124304B (en) * | 2019-12-19 | 2022-04-22 | 北京浪潮数据技术有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN112363678B (en) * | 2021-01-13 | 2021-04-30 | 北京东方通软件有限公司 | Data migration method and system based on message middleware |
| CN113741819B (en) * | 2021-09-15 | 2024-09-27 | 第四范式（北京）技术有限公司 | Method and device for hierarchical storage of data |
| CN114489504B (en) * | 2022-01-21 | 2023-08-08 | 苏州浪潮智能科技有限公司 | SSD data management method and related components |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101646994A (en) * | 2006-12-06 | 2010-02-10 | 弗森多系统公司（dba弗森-艾奥） | Apparatus, system, and method for managing commands of solid-state storage using bank interleave |
| CN102968281A (en) * | 2012-11-26 | 2013-03-13 | 华为技术有限公司 | Data migration method and device |
| CN105487824A (en) * | 2015-12-07 | 2016-04-13 | 联想(北京)有限公司 | Information processing method, storage device and electronic device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6086007B2 (en) * | 2013-03-26 | 2017-03-01 | 富士通株式会社 | Storage control device control program, storage control device control method, storage control device, and storage system |
-
2016
- 2016-11-23 CN CN201611038740.XA patent/CN108089814B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101646994A (en) * | 2006-12-06 | 2010-02-10 | 弗森多系统公司（dba弗森-艾奥） | Apparatus, system, and method for managing commands of solid-state storage using bank interleave |
| CN102968281A (en) * | 2012-11-26 | 2013-03-13 | 华为技术有限公司 | Data migration method and device |
| CN105487824A (en) * | 2015-12-07 | 2016-04-13 | 联想(北京)有限公司 | Information processing method, storage device and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108089814A (en) | 2018-05-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108089814B (en) | Data storage method and device | |
| US11467767B2 (en) | Storage device throttling amount of communicated data depending on suspension frequency of operation | |
| US20130006948A1 (en) | Compression-aware data storage tiering | |
| CN107534615A (en) | Apparatus and method for self-adapting data compression | |
| CN102968281A (en) | Data migration method and device | |
| KR20200067962A (en) | Method and apparatus for writing data into solid state disk | |
| JP2012151840A (en) | Data compression device, operation method thereof, and data processing device including the same | |
| CN111064808A (en) | Load balancing method and device based on distributed storage system | |
| CN111245732B (en) | Flow control method, device and equipment | |
| CN103645860B (en) | Memory space management method and memory management device | |
| CN108874324A (en) | A kind of access request processing method, device, equipment and readable storage medium storing program for executing | |
| CN103927263A (en) | Garbage recycling method and garbage recycling device | |
| CN109976680A (en) | A kind of data migration method, device, equipment and readable storage medium storing program for executing | |
| CN115129253B (en) | A snapshot processing method, device, equipment and medium | |
| CN104933010B (en) | A method and device for deduplication | |
| CN107844593B (en) | Video data distribution method and device in distributed computing platform | |
| CN107783990B (en) | Data compression method and terminal | |
| WO2016141803A1 (en) | Image recognition accelerator, terminal device and image recognition method | |
| CN103530240B (en) | Data-block cache method and apparatus | |
| US20220050614A1 (en) | System and method for approximating replication completion time | |
| CN113055433A (en) | File transmission method, device, equipment and machine-readable storage medium | |
| CN104750547B (en) | The input and output I/O request processing method and processing device of virtual machine | |
| CN119003244A (en) | File synchronization method, device, computer equipment and storage medium | |
| CN105095197B (en) | A kind of method and device for handling data | |
| US9077325B2 (en) | Digital slew rate limiter |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |