WO2019174608A1 - Data processing method and related device therefor - Google Patents

Abstract

Provided in the embodiments of the present application are a data processing method and a related device therefor, which are used for splitting up first metadata according to a snapshot of the first metadata after performing a snapshot operation on the first metadata and processing data sent by a client in the splitting process. The method according to the embodiments of the present application comprises: a first partition server module receives a splitting instruction, the splitting instruction being used to instruct the first partition server module to split up first metadata; the first partition server module performs a snapshot operation on the first metadata; the first partition server module splits up the first metadata into metadata of at least two data ranges according to the snapshot of the first metadata and a pre-configured splitting policy, the metadata of the at least two data ranges comprising second metadata and third metadata, the second metadata being stored in a second partition, and the third metadata being stored in a third partition; and the first partition server module processes data sent by a client according to a splitting result.

Description

Data processing method and related equipment Technical field

The present application relates to the field of computers, and in particular, to a data processing method and related devices.

Background technique

The existing distributed system architecture is divided into storage and computing convergence and storage and computing separation. As the number of client accesses increases, the data traffic that the server needs to process grows rapidly. If the amount of data to be processed exceeds the server's The processing limit will cause the server to crash. Therefore, no matter which architecture is adopted, the load balancing problem between nodes needs to be solved. Load balancing refers to the reasonable allocation of data to multiple servers or operating units to avoid certain Some servers handle excessive amounts of data when the server is idle.

The best way to achieve load balancing is to partition the server's database. The existing storage computing separation architecture performs data storage and calculation on different devices. The process of database partitioning is as follows: First, the control server selects the partition to be split, and notifies the partition server that manages the partition to execute the split command, and the partition server closes the partition service, that is, stops processing the data sent by the client, and marks the partition. As the offline state, the partition server starts to split the partitions it manages. The specific partition splitting method is to split the partition metadata and split the partition metadata into metadata of at least two partitions. After the splitting is completed, the at least two partitions are online and the routing table information is updated. After the routing table is updated, the server managing the two partitions can normally process the data sent by the client.

It can be seen that when any one or more partition metadata managed by the partition server is split, if the data sent by the client is received, the metadata of the partition server is in the split state, and the client sends the data. The data will cause the metadata of the partition to change. Therefore, the partition server must stop processing the client data. After the splitting is completed, the partition server needs to re-determine the data range managed by itself and update the routing table information before the I/O can be processed normally. The request, visible partition server to partition the data it manages, is unable to process the data sent by the client.

Summary of the invention

The embodiment of the present application provides a data processing method and a related device thereof, after performing a snapshot operation on the first metadata, splitting the first metadata according to the snapshot of the first metadata and the client in the splitting process The data sent by the terminal is processed.

A first aspect of the embodiments of the present application provides a data processing method, including:

The first partition server module receives a split command sent by the control server module, where the split command is used to instruct the first partition server module to split the first metadata, and the control server module requests the first element in the shared storage space in advance. After the data is split into multiple metadata, the storage space of the third metadata in the plurality of metadata is stored in the first partition before being split, and the first partition is managed by the first partition server.

Wherein, the metadata is used to indicate the address of the data, and the server may find the corresponding data according to the metadata;

The first partition server module performs a snapshot operation on the first metadata to generate a snapshot of the first metadata, and the snapshot of the first metadata is also stored in the first partition;

The first partition server module splits the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in the second partition, and the second partition is managed by the first partition server module, so the second metadata can be regarded as the first element The metadata left by the first partition server module remaining after the data is split. The third metadata is stored in a third partition, the third partition is managed by the second partition server module, and the second partition server module is determined by the control server module.

In this embodiment, the partition is also split when the metadata is split, so the second partition and the third partition are formed by splitting the first partition;

The first partition server module processes the data sent by the client according to the split result.

The embodiment of the present application has the following advantages: after receiving the split instruction, the first partition server module performs a snapshot operation on the first metadata managed by the first partition server module, and according to the snapshot of the first metadata and the preset splitting strategy, The metadata is split into metadata of at least two pieces of data ranges, the metadata of the at least two pieces of data ranges includes second metadata and third metadata, the second metadata is stored in the second partition, and the second partition is configured by the second partition The first partition server manages, the third metadata is stored in the third partition, and the third partition is managed by the second partition server module, and then the first partition server module processes the data sent by the client according to the split result, thereby The data processing pressure of the first partition server module. In the embodiment of the present application, the first partition server module can split the first metadata according to the snapshot of the first metadata. Since the snapshot operation is performed on the metadata, the metadata to be split is fixed. Even if the partition server module receives the data sent by the client at this time, the partition can be split according to the fixed metadata in the snapshot, and the client data can be processed normally.

According to the first aspect, in a first implementation manner of the first aspect, the data sent by the client belongs to a data range formed by the data indicated by the second metadata, and the first partition service module is configured according to the split result. Processing the data sent by the client includes:

The first partition server module stores data sent by the client in the second partition, and the second partition is managed by the first partition server module, so the partition splitting succeeds, by the first partition service module Responding to an I/O request corresponding to the data sent by the client.

In this embodiment, a case in which the client transmits data is described, and the implementability of the scheme is increased.

In a second implementation manner of the first aspect, the data sent by the client belongs to a data range formed by the data indicated by the third metadata, and the first partition service module is configured according to the split result. Processing the data sent by the client includes:

Because the second partition server module is not online yet, the first partition server module first writes the data sent by the client to the log, and the log is stored in the third partition, and waits for the second partition server module. After going online, the second partition server module can find the client data according to the log record.

In this embodiment, another case of transmitting data by the client is explained, which increases the flexibility of the scheme.

According to a second implementation manner of the first aspect, the third implementation manner of the first aspect, the log includes data sent by the client;

After the partition is successfully split, the second partition server module may obtain data sent by the client included in the log from the third partition, and respond to the data according to the data sent by the client. I/O request.

In this embodiment, the log may include only the data sent by the client, and the log may also include the data sent by the client and the metadata corresponding to the data sent by the client, which is not limited herein.

In the present embodiment, a case of log content has been described, which increases the flexibility and diversity of the solution embodiment.

Based on the second implementation manner of the first aspect, the fourth implementation manner of the first aspect, the method further includes:

The first partition server module writes a log to the client data, and the first partition server module also stores the data sent by the client in the second partition;

If the partition splitting fails, the first partition server module may call pre-stored data sent by the client from the second partition.

In this embodiment, by storing the client data in the second partition, the first partition server module can quickly roll back the data sent by the client stored in advance after the partition splitting fails, thereby effectively preventing data loss. .

According to the fourth implementation manner of the first aspect, the fifth implementation manner of the first aspect, when the log includes the metadata corresponding to the data sent by the client,

The partitioning is successful, and the second partitioning server module obtains the pre-stored data sent by the client from the second partition according to the metadata corresponding to the data sent by the client, and according to the obtained client. The data sent by the terminal processes the I/O request corresponding to the data sent by the client.

In the present embodiment, another case of log content is illustrated, which increases the flexibility and diversity of the solution embodiment.

The method according to any one of the first to fifth implementation manners of the first aspect, wherein in the sixth implementation manner of the first aspect, the first partition server module is After the snapshot of the first metadata and the preset splitting policy split the first metadata into metadata of at least two pieces of data ranges, the method further includes:

The first partition server module splits the data into data of at least two pieces of data ranges, and the address of the data is indicated by the first metadata, so the data has a corresponding relationship with the first metadata.

The data of each segment of the data in the data of the at least two segments of data is in one-to-one correspondence with the metadata of each segment of the data in the metadata of the at least two segments of data.

The first partition server module stores the second metadata and the data corresponding to the second metadata in the second partition, and the first partition server module compares the third metadata and the first The data corresponding to the ternary data is stored in the third partition.

In this embodiment, the storage situation after the split data is split is described, which increases the integrity of the solution.

The method according to any one of the first to fifth implementation manners of the first aspect, wherein the first partition server module according to the seventh implementation manner of the first aspect, After the snapshot of the first metadata and the preset splitting strategy split the first metadata into metadata of at least two pieces of data ranges, the method further includes:

When the data corresponding to the first metadata is not split, the first partition server module stores the data corresponding to the first metadata and the second metadata in the first partition, and the first partition server The module stores the third metadata in the third partition, and the data corresponds to the first metadata.

In the present embodiment, the storage of data when the data to be split is not split is explained, which increases the diversity of the scheme.

A second aspect of the present application provides a server module, where the server module is a first partition server module, and the first partition server module includes:

a receiving unit, configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the metadata The address used to indicate the data;

a snapshot operation unit, configured to perform a snapshot operation on the first metadata;

a splitting unit, configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, and the element of the at least two pieces of data ranges The data includes second metadata and a third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third partition The third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition;

a processing unit, configured to process data sent by the client according to the split result.

In this embodiment, the splitting method of the partition managed by the first partition server module is described. By applying the splitting method, the first partition server module can process the client data in the splitting process.

Based on the second aspect, in a first implementation manner of the second aspect, the data sent by the client belongs to a data range formed by the data indicated by the second metadata, and the processing unit includes:

a first processing subunit, configured to store data sent by the client in the second partition;

The response subunit is configured to respond to the I/O request corresponding to the data sent by the client if the partition is successfully split.

In this embodiment, a case in which the client transmits data is described, and the implementability of the scheme is increased.

In a second implementation manner of the second aspect, the data sent by the client belongs to a data range formed by the data indicated by the third metadata, and the processing unit includes:

a second processing subunit, configured to write data sent by the client to a log;

The log is stored in the third partition.

In this embodiment, another case of transmitting data by the client is explained, which increases the flexibility of the scheme.

In a second implementation manner of the second aspect, the first partition server module further includes:

a storage unit, configured to store data sent by the client in the second partition;

And a calling unit, configured to call, from the second partition, the pre-stored data sent by the client if the partition splitting fails.

In this embodiment, by storing the client data in the second partition, the first partition server module can quickly roll back the data sent by the client stored in advance after the partition splitting fails, thereby effectively preventing data loss. .

Based on the second aspect and the first to third implementation manners of the second aspect, in a fourth implementation manner of the second aspect, the splitting unit is further configured to split data into at least two pieces of data. a range of data, the data corresponding to the first metadata, the data of each piece of data range in the data of the at least two pieces of data range and the element of each piece of data range in the metadata of the at least two pieces of data range The data corresponds one by one.

The storage unit is further configured to store the data corresponding to the second metadata and the second metadata in the second partition, and corresponding to the third metadata and the third metadata The data is stored in the third partition.

In this embodiment, the storage situation after the split data is split is described, which increases the integrity of the solution.

In a fifth implementation manner of the second aspect and the second aspect, in a fifth implementation manner of the second aspect, the storage unit is further configured to store data and the second metadata in The first partition, and storing the third metadata in the third partition, the data corresponding to the first metadata.

In the present embodiment, the storage of data when the data to be split is not split is explained, which increases the diversity of the scheme.

A third aspect of the present application provides a computing device, including a processor and a memory,

The processor executes program code in the memory to perform any of the methods described in the above aspects.

The first partition server module runs on the computing device, or the computing device operates as the first partition server module.

A fourth aspect of the present application provides a computer readable storage medium having instructions stored therein that, when executed on a computer, cause the computer to perform the methods described in the above aspects.

A fifth aspect of the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the methods described in the various aspects above.

DRAWINGS

1 is a schematic diagram of a function introduction of a module according to an embodiment of the present application;

2(a) is a schematic structural diagram of a method for connecting a partition server module and other modules according to an embodiment of the present application;

FIG. 2(b) is another schematic structural diagram of a method for connecting a partition server module and other modules according to an embodiment of the present application;

2(c) is another schematic structural diagram of a method for connecting a partition server module and other modules according to an embodiment of the present application;

2(d) is a schematic diagram showing another structure of a connection mode between a partition server module and other modules according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an embodiment of a data processing method according to an embodiment of the present application; FIG.

4 is a schematic diagram of a first metadata snapshot operation;

FIG. 5(a) is a schematic diagram of splitting metadata and corresponding data according to an embodiment of the present application;

FIG. 5(b) is a schematic diagram of splitting another metadata and corresponding data according to an embodiment of the present application;

FIG. 6 is a schematic diagram of another embodiment of a data processing method according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a manner of data processing according to an embodiment of the present application; FIG.

FIG. 8 is a schematic diagram of another embodiment of a data processing method according to an embodiment of the present application; FIG.

FIG. 9 is a schematic diagram of another manner of data processing according to an embodiment of the present application; FIG.

10 is a schematic diagram of a second partition server module acquiring client data;

11(a) is a schematic structural diagram of a first partition server module according to an embodiment of the present application;

FIG. 11(b) is another schematic structural diagram of a first partition server module according to an embodiment of the present application;

FIG. 11(c) is another schematic structural diagram of a first partition server module according to an embodiment of the present application;

FIG. 12 is another schematic structural diagram of a first partition server module according to an embodiment of the present application;

FIG. 13 is another schematic structural diagram of a first partition server module according to an embodiment of the present application;

FIG. 14 is another schematic structural diagram of a first partition server module according to an embodiment of the present application.

detailed description

As shown in Figure 1: The client includes an interface to access the database, which can access the database of the partition server module. The control server module can split the data according to the row key of each data and assign it to different partition server modules. Perform management and maintenance, responsible for load balancing of partitioned server modules and fault management of partitioned server modules. The partition server module can manage the partitions allocated by the control server module and process I/O (input/out-put) requests to these partitions. The number of storage partitions managed by each partition server module is not limited herein. Each partition server module may manage only one storage partition as shown in FIG. 1, or one partition server module may manage multiple storage partitions. When a partition server module manages multiple storage partitions, one of the multiple storage partitions needs to be split, and the partition enters the split state. In addition to the data stored in the storage partition managed by each control server module, metadata is stored. The metadata is structured data, and contains address information of the data, and the corresponding data can be found through the metadata. The storage space required by the storage partition is obtained by the control server module from the storage space of the shared storage. It should be noted that the partition splitting of the database is based on the row key to distinguish different data, and then the split point is selected to split the database.

The function module shown in Figure 1, the partition server module, the control server module, and the shared storage may be in the same computer device or on different computer devices. The following is an example of the number of partition server modules: The application scenario is shown below:

As shown in FIG. 2(a), the client, the control server module, the partition server module, and the storage space for storing the partition data are integrated on the same computing device, as shown in FIG. 2(b), the client and the control server. The module is on the same physical device, and the control server module manages multiple partition server modules. The partition server modules are distributed on different computing devices. The storage space of the partition is on the same terminal device as the partition server module. It can be understood that In the implementation scenario shown in 2(b), there is also a possibility that the client and the control server module are on different computing devices. As shown in FIG. 2(c), the partition server module and its corresponding storage space can be distributed on different computing devices, meaning that the partition server module can store data on the storage space of another device after running on one device. As shown in FIG. 2(d), the storage device used for the storage partition can be separated from the computing device by two devices. The storage device and the computing device exchange information through the network, and the partition managed by each partition server is stored in different In the storage device, the storage space of these storage devices together constitute a shared storage.

When the number of partition server modules is N, the second partition server module determined by the control server may be any one of the partition server modules 2 to N in FIG. 2(a) to FIG. 2(d) (FIG. 2 (FIG. 2) a) to FIG. 2(d)) taking N equal to 3 as an example), the storage partition or storage device corresponding to the determined partition server module is the storage space of the third partition. The first partition server module in this embodiment is the partition server module 1 in the structural diagrams of FIG. 2(a) to FIG. 2(d), and the storage partition or storage device corresponding to the partition server module 1 is the first before splitting. The storage space of the partition or the storage space of the second partition after the split.

The terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a particular order or order. It is to be understood that the terms so used are interchangeable under appropriate circumstances, and are merely illustrative of the manner in which the objects of the same attribute are used in the description of the embodiments of the present application. In addition, the terms "comprises" and "comprises" and "comprises", and any variations thereof, are intended to cover a non-exclusive inclusion so that a process, method, system, product, or device comprising a series of units is not necessarily limited to those elements, but may include Other units listed or inherent to these processes, methods, products or equipment.

The embodiment of the present application provides a data processing method for a partition server module to process data sent by a client during a metadata splitting process.

301. The first partition server module receives a split instruction.

The first partition server module processes data of one or more partitions. When the data of one partition is too large and exceeds the processing capability of the first partition server module, the partition server module may be crashed, and the control server module is responsible for managing the processing of each partition service module. The amount of data is balanced, so the control server module needs to split the partition, that is, allocate a part of the data request of the partition to another server for load balancing between the servers.

After the control server module determines the first partition to be split, applies to the shared storage layer for the storage space of the third partition, where the third partition is formed by splitting the first partition, and the first partition is split into at least two partitions, at least two The partition includes a second partition and a third partition. The second partition occupies the storage space of the first partition before the split, and the third partition occupies the storage space requested by the control server in advance. The second partition server module is determined by the control server module, and is configured to manage a part of the data of the split partition. The specific method is to store the metadata corresponding to the part of the data in the third partition, which is managed by the second partition server module. This part of the data and its corresponding metadata, in order to reduce the data processing pressure of the first partition server module. For example, the first metadata is 1-100, and is split into metadata of four data ranges of 1-20, 20-50, 50-70, and 70-100. The first partition service module and the second partition server module may respectively Any one of the four pieces of metadata is managed, wherein the data range formed by the metadata managed by the first partition server module and the second partition server module is different.

Metadata is a structured data that provides information resources or data. It is a structured description of information resources. Metadata includes the formation, relationships, fields, constraints, etc. of the data set. The partition server module can access the corresponding data according to the metadata. The data.

Which piece of metadata is managed by the first partition server module or the second partition server module means that the data corresponding to the piece of metadata is also managed by the first partition server module or the second partition server module. For example, the metadata managed by the first partition server module ranges from 1 to 20, and the data corresponding to the metadata of the range of 1-20 is 1-100. The data of 1-100 is also managed by the first partition server module, and the metadata can be used as The index of the data, so the first partition server module only needs to store the metadata of 1-20 range in the second partition managed by it, and realize the management of the metadata of 1-20 range and the data of 1-100 range. After the split is successful, when the data corresponding to the client I/O request belongs to the 1-100 range, the first partition server module responds to the client request.

In this embodiment, the first partition server module first receives a split instruction, where the split command carries information about the metadata to be split and storage space information that the control server allocates for the third partition in advance, and the first partition server module may Separating the first metadata according to the splitting instruction, and the information of the to-be-split metadata (ie, the first metadata) includes a splitting policy preset by the control server, so the first partition server determines the first metadata. The preset split strategy can then be determined.

302. The first partition server module performs a snapshot operation on the first metadata.

As shown in FIG. 4 below, the first partition server module performs a snapshot operation on the first metadata to generate a snapshot of the first metadata. After the snapshot is executed, the first partition includes the first metadata, the data indicated by the first metadata, and the first metadata snapshot. It can be understood that the first partition further includes storage space information of the third partition carried in the split instruction.

303. The first partition server module splits the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and the preset splitting policy.

The first partition server module splits the first metadata according to the snapshot of the first metadata, and the first partition server module determines a preset splitting policy, where the splitting policy includes split point information. The split point is determined according to the row key of the data, and the split position of the corresponding metadata after determining the split point of the data is also determined.

1. A possible case is that the first partition server module splits the first metadata into two partitions, that is, splits the first metadata into two pieces of metadata, and only one metadata split point. The first partition server module splits the first metadata into the second metadata and the third metadata from the split point location, the second metadata is stored in the second partition, and the third metadata is stored in the third partition.

2. Another possible case is that the first partition server module splits the first metadata into two or more partition metadata, and the two or more partition metadata includes the second metadata and the third element. Data, the split point of the metadata at this time includes at least two. For example, if the data range of the first metadata is 1-100 and the split points are 20 and 60 respectively, the first metadata is split into three partitions, which are 1-20, 20-60, and 60-100, respectively. The metadata of any two of the data ranges of 1-20, 20-60, and 60-100 are referred to as second metadata and third metadata, respectively, and the second metadata is stored in the second partition, the third metadata. Stored in the third partition.

In this embodiment, when the first partition server module splits the first metadata, the first partition server module processes the data to be split corresponding to the first metadata in the following two cases.

A: the first partition server module splits the split data;

As shown in FIG. 5( a ), taking the first metadata into the second metadata and the third metadata as an example, the first partition server module may split the first metadata and may be the first The data indicated by the metadata is also split. The split form is the split point corresponding to the metadata, and the split point is set at the corresponding position of the data for splitting, so the data to be split is split into the second metadata. The indicated data and the data indicated by the third metadata, and the data indicated by the second metadata is stored in the second partition, and the data indicated by the third metadata is stored in the third partition.

B: The first partition server module does not split the split data;

As shown in FIG. 5(b), taking the first metadata into the second metadata and the third metadata as an example, a possible case is that the first partition server module only splits the first metadata. The data corresponding to the metadata is not split. In this case, the data to be split is stored in the second partition. After the splitting is completed, the second partition server module needs to acquire the data indicated by the third metadata from the second partition and store the data in the third partition.

In this embodiment, the manner in which the second partition server module acquires the data indicated by the third metadata from the second partition is: when the second partition server module is online, when the data is sorted, according to the data indicated in the third metadata. The address information successively acquires the data indicated by the third metadata from the storage space of the first partition, and stores the data in the third partition.

304. The first partition server module processes the data sent by the client sent by the client according to the split result.

During the first metadata splitting process, the first partition server module receives the I/O request sent by the client, and the client may be an application on the terminal device, and the client and the first partition server module may be the same device. The function modules on the device can also be on different devices, which are not limited here.

The first partition server module determines a split result of the first metadata according to the preset splitting policy, and determines that the second metadata formed after the splitting is stored in the second partition, and the third metadata is stored in the third partition, The one-part server module processes the data request sent by the subsequent client according to the splitting of the first metadata and the storage of the split metadata.

In this embodiment, the first partition server module performs a snapshot operation on the first metadata, and splits the first metadata according to the snapshot of the first metadata. In the prior art, the client continuously sends I/O. The request causes the first metadata to be split to change continuously, and the partition cannot be split. Therefore, the server needs to stop receiving the client I/O request. In this embodiment, the snapshot is performed after performing the snapshot operation on the metadata, which is equivalent to The split metadata is fixed at a certain time, so the server can process the client's I/O request normally.

In this embodiment, the server splits the first metadata by using a snapshot, and in this way, the normal processing of the I/O request is achieved. In this embodiment, the processing flow of the I/O request has the following two situations: Please refer to FIG. 6 and FIG. 7, and one of the cases will be described below.

601. The first partition server module receives the split instruction.

602. The first partition server module performs a snapshot operation on the first metadata.

603. The first partition server module splits the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and the preset splitting policy.

In this embodiment, the steps 601 to 603 of the embodiment are similar to the steps 301 to 303 of the embodiment shown in FIG. 3, and details are not described herein again.

604. The first partition server module determines, according to the split result, a data range formed by the data indicated by the second metadata.

As shown in FIG. 7 , the data to be split is split into an example, and the first partition server module determines second metadata and third metadata generated after splitting the first metadata according to the split result, and the second metadata. Include one or more metadata, and the first partition server module searches for corresponding data according to the data address indicated by each metadata in the second metadata until all data corresponding to the second metadata is determined to be completed, and second The data corresponding to the metadata may form a data range, for example, the second metadata is 1, 2, 5, 10, 15, 20, and the corresponding data is 5, 10, 15, 20, 25, respectively, and the data is formed. The data range is 5 to 25.

605. The first partition server module stores the data sent by the client in the second partition.

The first partition server module determines the data range to which the data sent by the client belongs, and the data corresponding to the second metadata forms a data range. When the data sent by the client belongs to the data range, the first partition server module sends the data to the client. The data is written into the second partition, that is, the data sent by the client is stored in the storage space of the second partition. As shown in FIG. 7, when the data X belongs to the data range of 1-a, the data of the 1-a data range is stored in the second partition, so the first partition service module writes the data X to the second partition.

606. If the partition splitting succeeds, the first partition service module responds to the I/O request corresponding to the data sent by the client.

After the partition is successfully split, the first metadata is successfully split. At this time, the data managed by the first partition service module changes. The first partition server module only manages the data allocated after the split and the client during the split process. The data written by the terminal, the first partition server module notifies the client partition version update, the client sends an instruction to update the routing table information to the control server, and after the routing table information is updated, the second partition server module goes online, and the pair falls into the second partition. The data managed by the server module is processed by the data allocated by the control server.

The data sent by the client is managed by the first partition server module, so the I/O request corresponding to the data is responded by the first partition server module.

When the partition split fails, the first partition is not split, so the first partition server module manages the data range unchanged, and the data written by the client during the splitting process is also managed by the first partition server module.

In this embodiment, when the client data is processed in the partition splitting process, the first partition server module needs to determine the data range to which the data sent by the client is sent by the client, and the data to which the client sends the data belongs in this embodiment. A description of the scope has increased the feasibility of the solution.

Referring to FIG. 7 and FIG. 8, another situation of the client I/O request processing flow in this embodiment is as follows:

801. The first partition server module receives the split instruction.

802. The first partition server module performs a snapshot operation on the first metadata.

803. The first partition server module splits the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy.

In this embodiment, the steps 801 to 803 of the embodiment are similar to the steps 301 to 303 of the embodiment shown in FIG. 3, and details are not described herein again.

804. The first partition server module determines, according to the splitting result, a data range formed by the data indicated by the third metadata.

The first partition server determines, according to the split result, the third metadata generated after the first metadata split, the third metadata includes one or more metadata, each metadata indicates an address of the data, and the first partition server The module finds the corresponding data according to the data address indicated by each metadata in the third metadata, until all the data corresponding to the third metadata is determined to be completed, and the data corresponding to the third metadata may form a data range, for example, The third metadata is 50, 59, 65, 71, 80, 88, and the corresponding data is 55, 60, 70, 80, 90, respectively, and the data forms a data range of 55 to 90.

805. The first partition server module writes data sent by the client to the log.

806. The first partition server module stores the log in the third partition.

As shown in FIG. 7, since the partition splitting is not completed yet, the second partition server module cannot be online, so the first partition server module cannot directly write the data sent by the client to the third partition, but needs to use the write log. In the manner, the pre-write log is sent to the data sent by the client, as shown in FIG. 10, so that after the partition splitting is completed, the second partition server module traverses the log, obtains the data sent by the client, and stores the data sent by the client. In the third partition, the data sent by the client is managed by the second partition server module, and when the subsequent client sends the I/O request corresponding to the data, the second partition server module responds.

FIG. 9 is an implementation manner of the embodiment when the data X in FIG. 7 belongs to the data range indicated by the third metadata. Referring to FIG. 9, in the embodiment, the first partition server module writes data sent by the client. At the same time of the log, the data sent by the client is also written into the second partition, that is, the data sent by the client is stored in the storage space of the second partition. Its purpose is to facilitate fault handling, but also for decoupling. If the partition split fails, the first metadata is not split at this time. The first partition server module re-executes the split operation. If multiple splits fail, the rollback is initiated. The first partition server module can quickly roll back the data sent by the client stored in advance, effectively preventing data loss.

In this embodiment, there are many possible situations for the content contained in the log. The following two cases will be introduced:

1. The log contains client data.

If the partition is successfully split, after the second partition server module is online, the data is directly obtained from the log, and the data is stored in the third partition;

If the partition split fails, referring to Figure 9, the second partition server module cannot be successfully brought online. At this time, the data in the log cannot be read by the first partition server module, but the first partition server module records the client data at the same time. The client data is also stored in the second partition, so the first partition server module can directly obtain the client data from the second partition, effectively avoiding data loss when the partition split fails.

In this embodiment, the log may include only the data sent by the client, and the log may also include the data sent by the client and the metadata corresponding to the data sent by the client, which is not limited herein.

2. The log only contains the metadata corresponding to the client data, and the client data is stored in the second partition.

If the partition is successfully split, after the second partition server module is online, referring to FIG. 9, the second partition server module calls the data sent by the pre-stored client from the second partition according to the metadata corresponding to the client data, and the client The sent data is stored in the third partition.

If the partition split fails, the same first partition server module can directly obtain the client data from the second partition, effectively avoiding data loss when the partition split fails.

In this embodiment, when the client data is processed in the partition splitting process, the first partition server module needs to determine the data range to which the data sent by the client is sent by the client, and the data to which the client sends the data belongs in this embodiment. Another aspect of the scope is illustrated, increasing the enforceability of the scheme.

A data processing method in the embodiment of the present application is described above with reference to FIG. 1 to FIG. 10. Referring to FIG. 11(a), a server module according to an embodiment of the present application will be described below.

A server module, which is a first partition server module 1100, and may include:

The receiving unit 1101 is configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the element The data is used to indicate the address of the data;

The snapshot operation unit 1102 is configured to perform a snapshot operation on the first metadata.

The splitting unit 1103 is configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third a partition, the third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition;

The processing unit 1104 is configured to process data sent by the client according to the split result.

In this embodiment, the first partition server module can split the first metadata according to the snapshot of the first metadata. Since the snapshot operation is performed on the metadata, the metadata to be split is fixed, and the first Even if the partition server module receives the data sent by the client at this time, the partition can be split according to the fixed metadata in the snapshot, and the client data can be processed normally.

Referring to FIG. 11(b), in FIG. 11(a), a possible case is that when the data sent by the client belongs to the data range formed by the data indicated by the second metadata, the data sent by the client Processed by the first partition server module, so the processing unit includes:

a first processing sub-unit 11041, configured to store data sent by the client in the second partition;

The response subunit 11042 is configured to respond to the I/O request corresponding to the data sent by the client if the partition is successfully split.

In this embodiment, a case in which the client transmits the data range to which the data belongs is described, and the implementability of the scheme is increased.

Referring to FIG. 11(c), another possible case is that when the data sent by the client belongs to the data range formed by the data indicated by the third metadata, the data sent by the client is performed by the second partition server module. Processing, so the processing unit includes:

a second processing sub-unit 11043, configured to write data sent by the client to a log;

The log is stored in the third partition.

In this embodiment, another case in which the client sends the data range to which the data belongs is described, which increases the flexibility of the scheme.

Referring to FIG. 12, the data sent by the client can be stored in the second partition while writing the log to prevent data loss.

The receiving unit 1201 is configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the element The data is used to indicate the address of the data;

a snapshot operation unit 1202, configured to perform a snapshot operation on the first metadata;

The splitting unit 1203 is configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third a partition, the third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition;

The processing unit 1204 is configured to process data sent by the client according to the split result.

a storage unit 1205, configured to store data sent by the client in the second partition;

The calling unit 1206 is configured to invoke pre-stored data sent by the client from the second partition if the partition splitting fails.

The processing unit includes:

The second processing sub-unit 12043 is further configured to write data sent by the client to a log;

The log is stored in the third partition.

In this embodiment, in order to prevent the loss of client data in the case of failure of partition splitting, this embodiment introduces another possible case of data processing for the client, which increases the diversity of the scheme.

Referring to FIG. 13, the first partition server may split the metadata, and may select the split data to be split or not split, as will be described below.

The receiving unit 1301 is configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the element The data is used to indicate the address of the data;

a snapshot operation unit 1302, configured to perform a snapshot operation on the first metadata;

The splitting unit 1303 is configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third a partition, the third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition;

The splitting unit 1303 is further configured to split data into data of at least two pieces of data ranges, where the data corresponds to the first metadata, and data of each piece of data range in the data of the at least two pieces of data ranges The metadata of each of the data ranges in the metadata of the at least two pieces of data ranges are in one-to-one correspondence.

The storage unit 1304 is configured to store the data corresponding to the second metadata and the second metadata in the second partition, and the data corresponding to the third metadata and the third metadata Stored in the third partition.

The processing unit 1305 is configured to process data sent by the client according to the split result.

or,

The receiving unit 1301 is configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the element The data is used to indicate the address of the data;

a snapshot operation unit 1302, configured to perform a snapshot operation on the first metadata;

The splitting unit 1303 is configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third a partition, the third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition;

a storage unit 1304, configured to store data and the second metadata in the first partition, and store the third metadata in the third partition, the data and the first metadata correspond.

The processing unit 1305 is configured to process data sent by the client according to the split result.

In this embodiment, two storage situations of the data to be split are introduced, which increases the flexibility of the implementation of the solution.

FIG. 2 shows several possible connections of the first partition server module and the control server module, the client module, and the storage partition. Referring to FIG. 14, the internal result diagram of the first partition server module will be described below.

FIG. 14 is a schematic structural diagram of a computing device according to an embodiment of the present invention. The computing device 1400 includes one or more central processing units (CPU) 1422 and a memory 1430 (which is a local storage of the computing device 1400). space). The computing device 1400 also includes one or more memories 1430 (eg, media such as memory, HDD or SSD or tape) for storing program code 1442 or data 1444. The memory 1430 may be a transitory storage medium or a persistent storage medium. Program code 1442 may include one or more modules (not shown), each of which may include a series of instruction operations. The central processor 1422 executes the program code 1442 to implement the functions of the first partition server module described above on the computing device 1400, or to perform the steps performed by the first partition server module described above. Still further, central processor 1422 can be arranged to communicate with storage medium 1430 to perform a series of instruction operations in storage medium 1430.

The central processing unit 1422 can perform the steps performed by the first partition server module in the foregoing embodiment according to the instruction operation, including:

Receiving a split instruction, the split instruction is configured to instruct the first partition server module to split the first metadata, where the first metadata is stored in a first partition, and the metadata is used to indicate data address;

Performing a snapshot operation on the first metadata;

And dividing the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the metadata of the at least two pieces of data ranges includes second metadata And the third metadata, the second metadata is stored in the second partition, the second partition is managed by the first partition server module, and the third metadata is stored in the third partition, the third partition Managed by the second partition server module, the second partition and the third partition are formed by splitting the first partition;

The data sent by the client is processed according to the split result.

Computing device 1400 can also include one or more power sources 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458, and/or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM and more.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of cells is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated. Go to another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

An integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, can be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application, in essence or the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. The instructions include a plurality of instructions for causing a computer device (which may be a personal computer, a local client, or a network device, etc.) to perform all or part of the steps of the various embodiments of Figures 2 through 4 of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program code. .

The above embodiments are only used to explain the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the foregoing embodiments can still be implemented. The technical solutions described in the examples are modified, or equivalent to some of the technical features, and the modifications or substitutions do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (16)

A data processing method, comprising: The first partition server module receives the split instruction, where the split instruction is used to instruct the first partition server module to split the first metadata, where the first metadata is stored in the first partition, the metadata The address used to indicate the data; The first partition server module performs a snapshot operation on the first metadata; The first partition server module splits the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, where the at least two pieces of data range The metadata includes second metadata and third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third a partition, the third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition; The first partition server module processes the data sent by the client according to the split result. The method according to claim 1, wherein the data sent by the client belongs to a data range formed by the data indicated by the second metadata, and the first partition service module sends the client to the client according to the split result. The processing of the data includes: The first partition server module stores data sent by the client in the second partition; If the partition splitting succeeds, the first partition service module responds to an I/O request corresponding to the data sent by the client. The method according to claim 1, wherein the data sent by the client belongs to a data range formed by the data indicated by the third metadata, and the first partition service module sends the data to the client according to the split result. The processing of the data includes: The first partition server module writes data sent by the client to a log; The first partition server module stores the log in the third partition. The method according to claim 3, wherein the log includes data sent by the client; After the log is used for the partition splitting, the second partition server module acquires data sent by the client included in the log from the third partition, and the data sent by the client is used by the client. The second partition server module responds to an I/O request corresponding to the data sent by the client. The method of claim 3, wherein the method further comprises: The first partition server module stores data sent by the client in the second partition; If the partition splitting fails, the first partition server module calls pre-stored data sent by the client from the second partition. The method according to claim 5, wherein the log includes metadata corresponding to data sent by the client; If the partition is successfully split, the log is used by the second partition server module to obtain, from the second partition, the pre-stored data sent by the client according to the metadata corresponding to the data sent by the client. The data sent by the client is used by the second partition server module to respond to an I/O request corresponding to the data sent by the client. The method according to any one of claims 1 to 6, wherein the first partition server module splits the first metadata according to a snapshot of the first metadata and a preset splitting policy After the metadata is divided into at least two pieces of data ranges, the method further includes: The first partition server module splits data into data of at least two pieces of data, the data corresponding to the first metadata, and data and data of each piece of data in the data of the at least two pieces of data range The metadata of each piece of data range in the metadata of at least two pieces of data ranges corresponds one-to-one; The first partition server module stores the data corresponding to the second metadata and the second metadata in the second partition, and the first partition server module uses the third metadata and the The data corresponding to the third metadata is stored in the third partition. The method according to any one of claims 1 to 6, wherein the first partition server module splits the first metadata according to a snapshot of the first metadata and a preset splitting policy After the metadata is divided into at least two pieces of data ranges, the method further includes: The first partition server module stores data and the second metadata in the first partition, and the first partition server module stores the third metadata in the third partition, The data corresponds to the first metadata. A server module, wherein the server module is a first partition server module, and the first partition server module includes: a receiving unit, configured to receive a splitting instruction, where the splitting instruction is used to instruct the first partitioning server module to split the first metadata, where the first metadata is stored in the first partition, the metadata The address used to indicate the data; a snapshot operation unit, configured to perform a snapshot operation on the first metadata; a splitting unit, configured to split the first metadata into metadata of at least two pieces of data ranges according to the snapshot of the first metadata and a preset splitting policy, and the element of the at least two pieces of data ranges The data includes second metadata and a third metadata, the second metadata is stored in a second partition, the second partition is managed by the first partition server module, and the third metadata is stored in a third partition The third partition is managed by the second partition server module, and the second partition and the third partition are formed by splitting the first partition; a processing unit, configured to process data sent by the client according to the split result. The first partition server module according to claim 9, wherein the data sent by the client belongs to a data range formed by the data indicated by the second metadata, and the processing unit comprises: a first processing subunit, configured to store data sent by the client in the second partition; The response subunit is configured to respond to the I/O request corresponding to the data sent by the client if the partition is successfully split. The first partition server module according to claim 9, wherein the data sent by the client belongs to a data range formed by the data indicated by the third metadata, and the processing unit comprises: a second processing subunit, configured to write data sent by the client to a log; The log is stored in the third partition. The first partition server module according to claim 11, wherein the first partition server module further comprises: a storage unit, configured to store data sent by the client in the second partition; And a calling unit, configured to invoke pre-stored data sent by the client from the second partition if the partition splitting fails. The first partition server module according to any one of claims 9 to 12, wherein the splitting unit is further configured to split data into data of at least two pieces of data ranges, the data and the Corresponding to the first metadata, the data of each segment of the data in the data of the at least two segments of data is in one-to-one correspondence with the metadata of each segment of the data in the metadata of the at least two segments; The storage unit is further configured to store the data corresponding to the second metadata and the second metadata in the second partition, and corresponding to the third metadata and the third metadata The data is stored in the third partition. The first partition server module according to any one of claims 9 to 12, wherein the storage unit is further configured to store data and the second metadata in the first partition, and The third metadata is stored in the third partition, the data corresponds to the first metadata, and the data corresponds to the first metadata. A computer readable storage medium comprising instructions for causing a computer to perform the method of any one of claims 1 to 8 when the instructions are run on a computer. A computing device, including a processor and a memory, The processor executes program code in the memory to perform the method of any of claims 1-8.

Images