CN110333824B - Capacity expansion method and device for storage system - Google Patents
Capacity expansion method and device for storage system Download PDFInfo
- Publication number
- CN110333824B CN110333824B CN201910494118.7A CN201910494118A CN110333824B CN 110333824 B CN110333824 B CN 110333824B CN 201910494118 A CN201910494118 A CN 201910494118A CN 110333824 B CN110333824 B CN 110333824B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- expanded
- list
- current node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a capacity expansion method and a device of a storage system, wherein the method comprises the following steps: obtaining a current node list, wherein the current node list is a list of nodes for data reading and writing at present; acquiring a node to be expanded, wherein the node to be expanded is a newly added temporary node for data movement; calculating the current node list and the data distribution of the nodes to be expanded to obtain a prediction result of data storage of each node; obtaining difference data in the current node list according to the prediction result; moving the difference data in the current node list to the node to be expanded; synchronizing the data of the node to be expanded with the data of the current node list to obtain a data synchronization result; and according to the data synchronization result, adding the nodes to be expanded in the storage system so as to expand the storage system. The method has the advantages of high data equalization speed, high data reliability and no influence on the performance of external services.
Description
Technical Field
The present invention relates to the field of capacity expansion of storage systems, and in particular, to a capacity expansion method and apparatus for a storage system.
Background
When the traditional storage system expands the capacity, a new storage service node is added into the storage system, data balance is carried out according to a certain algorithm in the storage system, and data are moved to the newly added node. Because the newly added storage node is added into the storage system, the data storage service is provided for the external customer service in the data relocation process.
Therefore, in the process of data balancing, data is moved to a newly added storage node inside the storage system, the newly added storage node serves as a data receiving end, a large amount of hard disk storage, network bandwidth, computing resources and the like of the storage node are occupied, especially, data written in the hard disk is the largest bottleneck, the service capacity of the node to customers is reduced, the duration of the capacity expansion process is long, and the node becomes a service bottleneck node in the cluster.
In order to keep the service performance not reduced in the capacity expansion process, the speed of moving data to a capacity expansion node is generally limited to reduce the influence on the service, but the capacity expansion time is increased, the probability of hardware failure of a storage system in the data equalization process is increased, and the data reliability risk is increased.
Disclosure of Invention
The invention provides a method and a device for expanding a storage system, which aim to solve the problem that service performance or expansion speed is influenced when the storage system is expanded and obtain the technical effects that external services normally come and go without reducing the expansion speed when the storage system is expanded.
In one aspect, the present invention provides a capacity expansion method for a storage system, where the method includes:
obtaining a current node list, wherein the current node list is a list of nodes for data reading and writing at present;
acquiring a node to be expanded, wherein the node to be expanded is a newly added temporary node for data movement;
calculating the current node list and the data distribution of the nodes to be expanded so as to obtain a prediction result of data storage of each node;
obtaining difference data in the current node list according to the prediction result;
moving the difference data in the current node list to the node to be expanded;
synchronizing the data of the node to be expanded and the data of the current node list to obtain a data synchronization result;
and according to the data synchronization result, adding the nodes to be expanded in the storage system so as to expand the storage system.
In another aspect, a capacity expansion apparatus of a storage system is provided, the apparatus including: the system comprises a current node list obtaining module, a node to be expanded obtaining module, a distribution result calculating module, a difference data moving module, a data synchronization module and a system expansion module;
the current node list obtaining module is used for obtaining a current node list, and the current node list is a node list for currently reading and writing data;
the node to be expanded obtaining module is used for obtaining a node to be expanded, wherein the node to be expanded is a newly added temporary node for data movement;
the distribution result calculation module is used for calculating the current node list and the data distribution of the nodes to be expanded so as to obtain a prediction result of the data distribution of each node;
the difference data moving module is used for moving the difference data in the current node list to the node to be expanded according to the prediction result;
the data synchronization module is used for synchronizing the data of the node to be expanded and the data of the current node list to obtain a data synchronization result;
and the system capacity expansion module is used for adding the nodes to be subjected to capacity expansion into the storage system according to the data synchronization result so as to expand the capacity of the storage system.
Another aspect provides a computer-readable storage medium for storing a program, where the program implements the capacity expansion method of the storage system when executed.
In another aspect, a terminal device is provided, where the terminal device includes the above capacity expansion device for a storage system.
The method comprises the steps of temporarily adding nodes to be expanded when the storage system is expanded, redistributing stored data, moving part of the data to the nodes to be expanded, performing incremental synchronization between the nodes to be expanded and existing storage nodes in the storage system, and formally adding the nodes to be expanded into the storage system when the synchronized difference data is smaller than a preset threshold value. The method has the advantages of high data equalization speed, high data reliability and no influence on the performance of external services.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic application scenario diagram of a capacity expansion method of a storage system according to an embodiment of the present invention;
fig. 2 is a flowchart of a capacity expansion method of a storage system according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for obtaining a prediction result of data storage of each node in a capacity expansion method of a storage system according to an embodiment of the present invention;
fig. 4 is a flowchart of a method executed by a self-attention layer model in a capacity expansion method of a storage system according to an embodiment of the present invention;
fig. 5 is a flowchart of a method for moving difference data to a node to be expanded in the expansion method of the storage system according to the embodiment of the present invention;
fig. 6 is a flowchart of a method for obtaining a data synchronization result through incremental synchronization in a capacity expansion method of a storage system according to an embodiment of the present invention;
fig. 7 is a flowchart illustrating a method for expanding a storage system according to an embodiment of the present invention when the method is applied to expanding the storage system;
fig. 8 is a schematic structural diagram of a capacity expansion device of a storage system according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a distributed result calculation module in a capacity expansion device of a storage system according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of a difference data moving module in a capacity expansion device of a storage system according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of a data synchronization module in a capacity expansion device of a storage system according to an embodiment of the present invention;
fig. 12 is a schematic hardware structure diagram of an apparatus for implementing the method provided by the embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
In the description of the present invention, it is to be understood that the terms "first", "second" and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Moreover, the terms "first," "second," and the like, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.
Referring to fig. 1, a schematic view of an application scenario of a capacity expansion method of a storage system according to an embodiment of the present invention is shown, where the application scenario includes a user terminal 110, a server 120, and a node 130 to be capacity expanded, and the user terminal includes, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like. The server obtains the data information written in the server from the user terminal, and the user terminal reads out the required data information through the server. When the server carries out capacity expansion operation, temporarily increasing nodes to be expanded, transferring the data part stored in the server to the nodes to be expanded, simultaneously, the nodes to be expanded are not opened to the user terminal, and the user terminal only accesses the existing nodes in the server to read and write data. And after the data movement is finished, formally adding the nodes to be expanded into the server so as to expand the capacity of the server.
Referring to fig. 2, a capacity expansion method of a storage system is shown, which is applicable to a server side, and the method includes:
s210, obtaining a current node list, wherein the current node list is a list of nodes for data reading and writing at present.
S220, obtaining nodes to be expanded, wherein the nodes to be expanded are newly added temporary nodes for data movement.
Specifically, the node to be expanded does not participate in a service of reading or writing data from or into the outside, the node to be expanded only performs data movement operation inside the storage system, and after the data movement is completed, the node to be expanded is formally added to the storage system.
In the subsequent data movement process of the node to be expanded, because the node to be expanded does not really add to the storage system, if the node to be expanded has an abnormality such as a hardware fault, the data reliability of the original storage system is not reduced.
And S230, calculating the current node list and the data distribution of the nodes to be subjected to capacity expansion to obtain a prediction result of the data storage of each node.
Further, the calculating the current node list and the data distribution of the node to be expanded to obtain the prediction result of the data storage of each node includes:
and S310, adding the nodes to be expanded into the current node list to obtain a new node list.
S320, performing data distribution calculation on the newly added node list according to a preset algorithm to obtain a prediction result of data storage of each node in the newly added node list.
Specifically, after the node to be expanded is temporarily added to the storage system, the original current node list and the node to be expanded may be combined into a new node list, and according to the number of members in the new node list, data distribution calculation is performed through a preset data distribution algorithm to obtain a prediction result of data stored in each node in the new node list, that is, after the node to be expanded is temporarily added to the storage system, the data stored in the storage system is redistributed. For example, the current storage system has three storage nodes a, B, and C, each storage node stores 4 data, and when the storage system is expanded, a node D to be expanded is temporarily added, so that 12 data in the original storage system are redistributed to the four nodes a, B, C, and D, and the obtained prediction result of data storage is that each node stores 3 data.
By data distribution prediction, data required to be stored in each node can be obtained, and the storage space of each node is conveniently re-allocated.
S240, obtaining difference value data in the current node list according to the prediction result;
specifically, the obtaining difference data in the current node list according to the prediction result includes:
and comparing the prediction result with the stored data in the current node list to obtain difference data in the current node list, wherein the difference data is the difference data between the stored data of each node in the current node list and the prediction result of data storage. For example, the current storage system has three storage nodes a, B, and C, each storage node stores 4 data, adds a node D to be expanded, and redistributes the data by a data distribution algorithm, where the number of data stored in each storage node is 3, and at this time, there is a difference data between 4 data originally stored in the three storage nodes a, B, and C and 3 data stored in the prediction result. The difference data generated in the current node list of the storage system is the data to be moved to the node to be expanded.
And S250, moving the difference data in the current node list to the node to be expanded.
Further, the moving the difference data in the current node list to the node to be expanded includes:
and S410, generating a task list of the nodes to be expanded according to the prediction result and the difference data, wherein the task list is a task list for distributing data to the nodes to be expanded for the current node list.
And S420, according to the task list, concurrently moving the difference data in the current node list to the node to be expanded.
Specifically, the task list is a task list used for listing which data in the current node list needs to be transferred to the node to be expanded. For example, the current storage system has three storage nodes, a, B, and C, each of which stores 4 data, the a storage node stores four data, x1, x2, x3, and x4, the B storage node stores four data, x5, x6, x7, and x8, and the C storage node stores four data, x9, x10, x11, and x 12. After the data is redistributed by a data distribution algorithm, the difference data x4 in the storage node A, the difference data x8 in the storage node B and the difference data x12 in the storage node C are moved to the node D to be expanded, so that the task list comprises three tasks of moving x4 to the node D to be expanded, moving x8 to the node D to be expanded and moving x12 to the node D to be expanded.
According to the tasks displayed in the task list, the difference data can be moved to the nodes to be expanded. When data is moved, because the nodes to be expanded do not have other services which need to be carried out except for the mobile data, the task of copying the differential data can be executed to the greatest extent, the data copying time can be shortened, and data balance can be completed more quickly.
S260, synchronizing the data of the node to be subjected to capacity expansion and the data of the current node list to obtain a data synchronization result;
further, the synchronizing the data of the node to be expanded with the data of the current node list to obtain a data synchronization result includes:
and S510, acquiring difference data between the node to be expanded and the current node list.
S520, incremental synchronization is carried out on the difference data between the node to be expanded and the current node list to obtain a data synchronization result.
Specifically, when the node to be expanded moves data, the storage node in the current node list of the storage system may perform a business transaction to and from the outside, and therefore there may be an operation of reading or writing data, so that data in the storage node in the current node list changes. For example, in the morning 6:00, starting to move data to a node to be expanded, and ending the data movement to 9 am, wherein a service can exist to the outside in the storage system, at this time, an incremental synchronization method can be adopted to select a time point when the data movement starts, namely, data coming and going in the storage system 6 am later than 00, and synchronize with the node to be expanded, so that difference data between the node to be expanded and a current node list is reduced.
After the node to be expanded completes the movement of the data, the node to be expanded and each node in the current node list need to be subjected to incremental synchronization, so that the data in each node is balanced.
And performing incremental synchronization to ensure that the difference data between the final node to be expanded and each node in the current node list is smaller, so that the data synchronization can be completed in a very short time, and the risk of data reliability is reduced.
And S270, adding the nodes to be expanded in the storage system according to the data synchronization result so as to expand the storage system.
Further, according to the data synchronization result, adding the node to be expanded in the storage system to expand the storage system includes:
s610, comparing the data synchronization result with a preset difference threshold value to obtain a comparison result;
and S620, if the data synchronization result is smaller than a preset difference threshold, adding the nodes to be expanded in the storage system to expand the storage system.
And S630, if the data synchronization result is larger than a preset difference threshold value, repeating incremental synchronization on the difference data between the node to be expanded and the current node list.
Specifically, when performing data increment synchronization, the size between the data synchronization result and the preset difference threshold after each increment synchronization is completed may be compared to obtain the data synchronization result of the current increment synchronization. And if the data synchronization result is greater than a preset difference threshold, repeating the incremental synchronization data process until the incremental difference data is less than the preset difference threshold, wherein the preset difference threshold is 5% of the capacity of the normal disk. When the nodes to be expanded are formally added into the storage system, the data synchronization can be completed in a very short time because the difference data is smaller than the preset difference threshold value, thereby reducing the data reliability risk.
Referring to fig. 7, fig. 7 shows a general flow when the method of the present embodiment is used to perform capacity expansion of a storage system, where the flow is as follows:
s710, acquiring a member list of a current storage system;
s720, adding the nodes to be expanded into a list;
s730, a new member list is obtained;
s740, calculating data distribution according to a data distribution algorithm in the storage system;
s750, generating a task list with data balance for the newly added node;
s760, concurrently executing a difference data copying task;
s770, incremental data synchronization is carried out;
and S780, judging whether the difference data is smaller than a preset threshold value, and if the difference data is larger than the preset threshold value, repeating incremental data synchronization.
And S790, if the difference data is smaller than a preset threshold value, formally adding the node to be expanded into the storage system.
Specifically, a current node list in the storage system is obtained, the current node list is a member list of data service nodes in the storage system, when the storage system needs capacity expansion, a node to be expanded is added, and the node to be expanded and the current node list are merged into a new node list. The nodes to be expanded can be a plurality of newly added nodes. And according to a data distribution algorithm in the storage system, calculating data distribution by referring to the newly added node list, and predicting the data quantity required to be stored by each node after the node to be expanded is added into the storage system. And according to the predicted data quantity required to be stored by each node, obtaining the difference data of each node in the current node list of the storage system, which needs to be moved to the node to be expanded, and generating a task list aiming at the newly added node and used for balancing the data to be expanded. The differential data copying task is started to be executed concurrently, and because the node to be expanded is not formally added into the storage system, the redundant data copying task in the data balancing process can be parallel to the maximum extent, and even if all resources of the node, such as hard disk storage, network bandwidth, computing resources and the like, can be occupied by the internal data migration task, the performance of external services cannot be influenced at the moment. The data movement is carried out on the nodes to be expanded fully, so that the duration of expansion can be shortened. And after the data is moved to the node to be expanded, performing incremental data synchronization on the node to be expanded and the nodes in the current node list. If the difference data between the node to be expanded and the current node list is smaller than a preset difference threshold value after data increment synchronization, formally adding the node into the storage system;
otherwise, performing incremental data synchronization again until the difference data is smaller than the preset difference threshold value, and formally adding the storage node into the storage system.
When the node to be expanded carries out data movement, the data movement can be carried out in a data copying mode, so that the expansion operation can be cancelled at any time before the node to be expanded formally joins the storage system, and the original storage system cannot be influenced.
In the capacity expansion method of the storage system provided by the embodiment of the invention, the node to be subjected to capacity expansion is newly added, part of data is moved to the node to be subjected to capacity expansion, and the node to be subjected to capacity expansion is not added into the storage system temporarily, so that the node to be subjected to capacity expansion does not bear external service, the performance of the storage system for processing external service is not influenced even if all resources of the node are occupied by the internal data relocation task, the redundant data replication task can be executed to the greatest extent concurrently, the data replication duration is reduced, and the data equalization process is completed more quickly.
An embodiment of the present invention further provides a capacity expansion device of a storage system, please refer to fig. 8, where the capacity expansion device includes: a current node list obtaining module 810, a node to be expanded obtaining module 820, a distribution result calculating module 830, a difference data moving module 840, a data synchronizing module 850 and a system expansion module 860;
the current node list obtaining module 810 is configured to obtain a current node list, where the current node list is a node list for currently performing data reading and writing;
the node to be expanded obtaining module 820 is configured to obtain a node to be expanded, where the node to be expanded is a newly added temporary node for performing data movement;
the distribution result calculating module 830 is configured to calculate data distributions of the current node list and the node to be expanded to obtain a prediction result of the data distribution of each node;
the difference data moving module 840 is configured to move the difference data in the current node list to the node to be expanded according to the prediction result;
the data synchronization module 850 is configured to synchronize the data of the node to be expanded with the data of the current node list to obtain a data synchronization result;
the system capacity expansion module 860 is configured to add the node to be capacity expanded to the storage system according to the data synchronization result, so as to perform capacity expansion on the storage system.
Further, referring to fig. 9, the distribution result calculating module 830 includes a newly added node list obtaining unit 910 and a data distribution calculating unit 920;
the new added node list obtaining unit 910 is configured to add the node to be expanded to the current node list to obtain a new added node list;
the data distribution calculating unit 920 is configured to perform data distribution calculation on the newly added node list according to a preset algorithm to obtain a prediction result of data storage of each node in the newly added node list.
Further, referring to fig. 10, the difference data moving module 840 includes a difference data obtaining unit 1010, a task list generating unit 1020, and a data moving unit 1030;
the difference data obtaining unit 1010 is configured to compare the prediction result with stored data in the current node list to obtain difference data in the current node list;
the task list generating unit 1020 is configured to generate a task list of the node to be expanded according to the prediction result and the difference data, where the task list is a task list that allocates data to the node to be expanded for the current node list;
the data moving unit 1030 is configured to concurrently move the difference data in the current node list to the node to be expanded according to the task list.
Further, referring to fig. 11, the data synchronization module 850 includes a difference data obtaining unit 1110, an increment synchronization unit 1120, and a comparison unit 1130;
the difference data obtaining unit 1110 is configured to obtain difference data between the node to be expanded and the current node list;
the increment synchronization unit 1120 is configured to perform increment synchronization on difference data between the node to be expanded and the current node list to obtain a data synchronization result;
the comparing unit 1130 is configured to compare the data synchronization result with a preset difference threshold to obtain a comparison result.
The device provided in the above embodiments can execute the method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For details of the technology that is not described in detail in the above embodiments, reference may be made to a capacity expansion method of a storage system provided in any embodiment of the present invention.
The present embodiment also provides a computer-readable storage medium, where computer-executable instructions are stored in the storage medium, and the computer-executable instructions are loaded by a processor and execute the capacity expansion method of the storage system according to the present embodiment.
The present embodiment further provides an apparatus, where the apparatus includes a processor and a memory, where the memory stores a computer program, and the computer program is adapted to be loaded by the processor and execute the capacity expansion method of the storage system according to the present embodiment.
The device may be a computer terminal, a mobile terminal or a server, and the device may also participate in forming the apparatus or system provided by the embodiments of the present invention. As shown in fig. 12, the mobile terminal 12 (or computer terminal 12 or server 12) may include one or more (shown here as 1202a, 1202b, \8230;, 1202 n) processors 1202 (the processors 1202 may include, but are not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1204 for storing data, and a transmission device 1206 for communication functions. Besides, the method can also comprise the following steps: a display, an input/output interface (I/O interface), a network interface, a power source, and/or a camera. It will be understood by those skilled in the art that the structure shown in fig. 12 is only an illustration and is not intended to limit the structure of the electronic device. For example, mobile device 12 may also include more or fewer components than shown in FIG. 12, or have a different configuration than shown in FIG. 12.
It should be noted that the one or more processors 1202 and/or other data processing circuitry described above may be referred to generally herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part in software, hardware, firmware, or any combination thereof. Further, the data processing circuitry may be a single, stand-alone processing module, or incorporated in whole or in part into any of the other elements in the mobile device 12 (or computer terminal). As referred to in the embodiments of the application, the data processing circuit acts as a processor control (e.g. selection of a variable resistance termination path connected to the interface).
The memory 1204 may be used for storing software programs and modules of application software, such as program instructions/data storage devices corresponding to the method described in the embodiment of the present invention, and the processor 1202 executes various functional applications and data processing by running the software programs and modules stored in the memory 1204, so as to implement the above-mentioned method for generating the self-attention network-based time-series behavior capture box. The memory 1204 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, memory 1204 may further include memory located remotely from processor 1202, which may be connected to mobile device 12 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmitting device 1206 is used for receiving or sending data via a network. Specific examples of such networks may include wireless networks provided by the communication provider of the mobile terminal 12. In one example, the transmission device 1206 includes a Network adapter (NIC) that can be connected to other Network devices via a base station to communicate with the internet. In one example, the transmitting device 1206 can be a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
The display may be, for example, a touch screen type Liquid Crystal Display (LCD) that may enable a user to interact with a user interface of the mobile device 12 (or computer terminal).
The present specification provides method steps as described in the examples or flowcharts, but may include more or fewer steps based on routine or non-inventive practice. The steps and sequences recited in the embodiments are merely one manner of performing the steps in a plurality of orders, and do not represent a unique order of performance. In the actual system or interrupted product execution, it may be performed sequentially or in parallel (e.g., in the context of parallel processors or multi-threaded processing) according to the embodiments or methods shown in the figures.
The configurations shown in the present embodiment are only partial configurations related to the present application, and do not constitute a limitation on the devices to which the present application is applied, and a specific device may include more or less components than those shown, or combine some components, or have different arrangements of components. It should be understood that the methods, apparatuses, and the like disclosed in the embodiments may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one division of logical functions, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or unit modules.
Based on such understanding, the technical solution of the present invention, which is substantially or partly contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (6)
1. A capacity expansion method of a storage system is characterized by comprising the following steps:
obtaining a current node list of the storage system, wherein the current node list is a list of nodes for reading and writing data currently;
acquiring a node to be expanded, wherein the node to be expanded is a newly added temporary node for data movement, and the node to be expanded does not provide data storage service for a user terminal;
calculating the data distribution of the current node list and the nodes to be expanded based on the re-allocation of the stored data in the current node list so as to obtain a prediction result of the data storage of each node in the current node list;
comparing the prediction result with the stored data of each node in the current node list to obtain difference data of each node in the current node list;
generating a task list of the node to be expanded according to the prediction result and the difference data, wherein the task list is a task list for distributing data to the node to be expanded for the current node list;
according to the task list, concurrently moving the difference data in the current node list to the nodes to be expanded;
acquiring difference data between the node to be expanded and the current node list, wherein the difference data is data which is newly stored to each node in the current node list after data migration starts;
performing incremental synchronization on the difference data between the node to be expanded and the current node list to obtain a data synchronization result;
comparing the data synchronization result with a preset difference threshold value to obtain a comparison result;
if the data synchronization result is smaller than a preset difference threshold value, adding the nodes to be expanded in the storage system to expand the storage system;
if the data synchronization result is larger than a preset difference threshold value, repeating incremental synchronization on the difference data between the node to be expanded and the current node list until the incremental difference data is smaller than the preset difference threshold value.
2. The capacity expansion method of the storage system according to claim 1, wherein the calculating the current node list and the data distribution of the nodes to be expanded to obtain a prediction result of the data storage of each node includes:
adding the nodes to be expanded to the current node list to obtain a new node list; and performing data distribution calculation on the newly added node list according to a preset algorithm to obtain a prediction result of data storage of each node in the newly added node list.
3. A capacity expansion apparatus for a storage system, the apparatus comprising: the system comprises a current node list obtaining module, a node to be expanded obtaining module, a distribution result calculating module, a difference data moving module, a data synchronization module and a system expansion module;
the current node list obtaining module is used for obtaining a current node list of the storage system, wherein the current node list is a node list for currently reading and writing data;
the node to be expanded acquiring module is used for acquiring a node to be expanded, wherein the node to be expanded is a newly added temporary node for data movement, and the node to be expanded does not provide data storage service for the user terminal;
the distribution result calculation module is used for calculating the data distribution of the current node list and the nodes to be expanded based on the redistribution of the stored data in the current node list so as to obtain a prediction result of the data distribution of each node in the current node list;
the differential data moving module is used for comparing the prediction result with the stored data of each node in the current node list to obtain the differential data of each node in the current node list and moving the differential data of each node in the current node list to the node to be expanded;
the data synchronization module is used for acquiring difference data between the node to be expanded and the current node list, wherein the difference data is data which is newly stored to each node in the current node list after data migration starts; performing incremental synchronization on the difference data between the node to be expanded and the current node list to obtain a data synchronization result;
the system capacity expansion module is used for adding the node to be expanded into the storage system according to the data synchronization result so as to expand the capacity of the storage system, so that the node to be expanded provides data storage service for the user terminal, and the method comprises the following steps:
comparing the data synchronization result with a preset difference threshold value to obtain a comparison result;
if the data synchronization result is smaller than a preset difference threshold value, adding the nodes to be expanded in the storage system to expand the storage system; the difference data moving module comprises a difference data obtaining unit, a task list generating unit and a data moving unit;
the difference data obtaining unit is used for comparing the prediction result with stored data in the current node list to obtain difference data in the current node list;
the task list generating unit is used for generating a task list of the nodes to be expanded according to the prediction result and the difference data, wherein the task list is a task list for distributing data to the nodes to be expanded for the current node list;
and the data moving unit is used for concurrently moving the difference data in the current node list to the node to be expanded according to the task list.
4. A flash device of a storage system according to claim 3, wherein the data synchronization module includes a difference data obtaining unit, an increment synchronization unit, and a comparison unit;
the difference data acquisition unit is used for acquiring difference data between the node to be expanded and the current node list;
the incremental synchronization unit is used for performing incremental synchronization on difference data between the node to be expanded and the current node list to obtain a data synchronization result;
the comparison unit is used for comparing the data synchronization result with a preset difference threshold value to obtain a comparison result.
5. An electronic device, characterized in that the device comprises a processor and a memory, wherein the memory stores a computer program adapted to be loaded by the processor and to perform a method of capacity expansion of a storage system according to any one of claims 1 to 2.
6. A computer-readable storage medium having stored thereon computer-executable instructions that are loaded by a processor and execute a method of expanding a storage system according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910494118.7A CN110333824B (en) | 2019-06-05 | 2019-06-05 | Capacity expansion method and device for storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910494118.7A CN110333824B (en) | 2019-06-05 | 2019-06-05 | Capacity expansion method and device for storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110333824A CN110333824A (en) | 2019-10-15 |
| CN110333824B true CN110333824B (en) | 2022-10-25 |
Family
ID=68140769
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910494118.7A Active CN110333824B (en) | 2019-06-05 | 2019-06-05 | Capacity expansion method and device for storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110333824B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111208952B (en) * | 2020-02-13 | 2022-07-12 | 河南文正电子数据处理有限公司 | Storage system capacity expansion method, readable storage medium and computing device |
| CN111723148B (en) * | 2020-04-10 | 2024-01-26 | 浙江大华技术股份有限公司 | Data storage method and device, storage medium and electronic device |
| CN118963646A (en) * | 2024-07-10 | 2024-11-15 | 濮阳市华龙区产品质量检验检测中心 | A method for optimizing storage of medical archive files |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377073A (en) * | 2012-04-26 | 2013-10-30 | 中国银联股份有限公司 | Data information processing device and method |
| WO2018076973A1 (en) * | 2016-10-28 | 2018-05-03 | 华为技术有限公司 | Load adjustment method, apparatus, and system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9116941B2 (en) * | 2013-03-15 | 2015-08-25 | International Business Machines Corporation | Reducing digest storage consumption by tracking similarity elements in a data deduplication system |
| CN106843745A (en) * | 2015-12-03 | 2017-06-13 | 南京中兴新软件有限责任公司 | Capacity expansion method and device |
| CN108132949B (en) * | 2016-12-01 | 2021-02-12 | 腾讯科技（深圳）有限公司 | Data migration method and device in database cluster |
| CN106843755B (en) * | 2017-01-04 | 2019-10-11 | 北京百度网讯科技有限公司 | Data balance method and device for server cluster |
| CN109426439A (en) * | 2017-09-04 | 2019-03-05 | 阿里巴巴集团控股有限公司 | The method and device of dilatation is carried out to distributed memory system |
| CN109379158A (en) * | 2018-12-10 | 2019-02-22 | 浪潮电子信息产业股份有限公司 | Time synchronization method, device and equipment for newly added nodes in storage cluster |
| CN109683826B (en) * | 2018-12-26 | 2023-08-29 | 北京百度网讯科技有限公司 | Capacity expansion method and device for distributed storage system |
-
2019
- 2019-06-05 CN CN201910494118.7A patent/CN110333824B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377073A (en) * | 2012-04-26 | 2013-10-30 | 中国银联股份有限公司 | Data information processing device and method |
| WO2018076973A1 (en) * | 2016-10-28 | 2018-05-03 | 华为技术有限公司 | Load adjustment method, apparatus, and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110333824A (en) | 2019-10-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12019652B2 (en) | Method and device for synchronizing node data | |
| JP7214295B2 (en) | Distributed system data synchronization method, apparatus, computer program and electronic equipment | |
| US11146502B2 (en) | Method and apparatus for allocating resource | |
| CN109949111B (en) | Electronic bill identification distribution method, electronic bill generation method, device and system | |
| CN109951331B (en) | Method, apparatus and computing cluster for sending information | |
| US9667750B2 (en) | Client-initiated leader election in distributed client-server systems | |
| CN110333824B (en) | Capacity expansion method and device for storage system | |
| CN111488121B (en) | Mapping system and method based on dynamic application access | |
| US11720593B2 (en) | Managing identifiers for multinodal master systems of unknown or changing size | |
| US9652331B2 (en) | Techniques for creating a bootable image in a cloud-based computing environment | |
| CN107016029A (en) | A kind of processing method of business datum, apparatus and system | |
| CN113377500B (en) | Resource scheduling method, device, equipment and medium | |
| CN104111957A (en) | Method and system for synchronizing distributed transaction | |
| CN111427918A (en) | Transaction detail data comparison method and device | |
| CN119904228A (en) | Transaction processing method, device, computer equipment and storage medium | |
| US9866444B2 (en) | Dynamic conversion of hardware resources of a server system | |
| CN109639704B (en) | A kind of master-slave mode server system application method, system, server and storage medium | |
| CN115277735A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114205416A (en) | Resource caching method and device, electronic equipment and computer readable medium | |
| US11630738B2 (en) | Automatic objective-based compression level change for individual clusters | |
| CN114265708A (en) | Data synchronization method and device, equipment and medium | |
| US11455318B2 (en) | Data replication using probabilistic replication filters | |
| CN119166047B (en) | Disk load dynamic adjustment method, system, device and computer-readable storage medium | |
| US20240231937A1 (en) | Shared memory fabric workload performance system | |
| US12197461B2 (en) | Distributed function data transformation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |