CN111221857B - Method and apparatus for reading data records from a distributed system - Google Patents

Abstract

This application provides a method and device for reading data records from a distributed system. The distributed system includes a partition server and a client. The partition server is used to manage the parent partitions in the data table. The partition server stores the first partition range check The partition range of the first partition range checker is the partition range of the parent partition. The method includes: the partition server receives the first read request sent by the client during the process of splitting the parent partition, and the first read request is used to request Read the first data record from the parent partition; the partition server reads the first data record from the parent partition according to the first read request, and the completion time of reading the first data record is later than the split completion time of the parent partition; the partition server Using the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs is beneficial to improve the accuracy of reading data records.

Description

Method and device for reading data records from a distributed system

technical field

The present application relates to the field of information technology, and more particularly, to a method and apparatus for reading data records from a distributed system.

Background technique

In a distributed system, data records are usually stored in the form of a data table, for example, an Hbase table in a Hadoop distributed system. In order to improve query efficiency, the data table can be divided into multiple partitions (regions), and multiple partitions are divided into multiple partition servers, and each partition server manages its own partition. Among the multiple partition servers, some partition servers process access requests (read requests or write requests) more frequently, and some partition servers process access requests less frequently. In this way, it will lead to load imbalance among multiple partition servers.

In the industry, the load of servers in each partition is usually balanced by splitting partitions. For example, if the partition server with higher load in the distributed system is the target partition server, and the target partition is accessed frequently in the partitions managed by the target partition server, then the target partition can be used as the partition to be split (that is, the parent partition), and then split the parent partition into multiple sub-partitions, and distribute some of the sub-partitions to the partition servers with lower load in the distributed system to reduce the load of the target partition server.

During partition splitting, the partition metadata of the parent partition and the partition range checker of the parent partition need to be modified to apply to the new child partition. However, if during the process of splitting the parent partition, after receiving the first read request requesting to read the first data record from the parent partition, and the completion time of reading the first data record is later than the split completion time of the parent partition , at this time, the partition server of the parent partition has been modified. If the modified partition server is used to check the partition range of the first data record, the first data record is no longer read from the parent partition, but regenerated read in the subpartition, resulting in a reduction in the accuracy of the first data record.

Contents of the invention

The present application provides a method and device for reading data records from a distributed system, so as to improve the accuracy of reading data records.

In a first aspect, a method for reading data records from a distributed system is provided, the distributed system includes a partition server and a client, the partition server is used to manage parent partitions in a data table, and the partition server stores There is a first partition range checker, the partition range of the first partition range checker is the partition range of the parent partition, the method includes: the partition server receives the The first read request sent by the client, the first read request is used to request to read the first data record from the parent partition; the partition server reads from the parent partition according to the first read request Fetching the first data record, the completion time of reading the first data record is later than the split completion time of the parent partition; the partition server uses the first partition range checker to check the first data The partition range of the partition to which the row key value of the record belongs.

The above partition server uses the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs. It can be understood that if the above first read request requests from the parent partition in the first file Reading the first data record, the partition server uses the first partition range checker to check the row key value of the read data record in the first file to obtain the data in the parent partition in the first file record, so that the partition server reads the first data record from the data record of the parent partition in the first file.

In the embodiment of the present application, the first data record read by the first read request is checked by the first partition range checker, which avoids that in the prior art, after the parent partition is successfully split, the first partition range checker This modification is the partition range checker for sub-partitions, which causes the partition range of the read data record to be different from the partition range where the data record expected to be read by the first read request is located, which is conducive to improving the accuracy of reading data records.

In a possible implementation manner, the partition server stores a second partition range checker, the partition range of the second partition checker is the partition range of the first target sub-partition, and the first target sub-partition is The sub-partition obtained by splitting the parent partition, the method further includes: after the split of the parent partition is completed, the partition server receives a second read request, and the second read request is used to request reads a second data record from the first target subpartition; the partition server reads the second data record from the first target subpartition; the partition server checks using the second partition range checker The partition range of the partition to which the row key value of the second data record belongs.

In the embodiment of the present application, the second partition range checker is used to check the data records read by the second read request, which is beneficial to improve the accuracy of reading data records.

In a possible implementation manner, during the process of splitting the parent partition, the value of the timestamp corresponding to the parent partition is the maximum value, and the method further includes: the partition server obtains the first read The timestamp carried in the request; if the timestamp carried in the first read request is smaller than the timestamp corresponding to the parent partition, the partition server selects the first partition range checker to check the row key of the first data record The partition range of the partition to which the value belongs.

In the embodiment of the present application, the time stamp corresponding to the parent partition is compared with the time stamp carried in the read request to select a different partition range checker, which is beneficial to improve the accuracy of selecting a partition range checker.

In a possible implementation manner, after the split of the parent partition is completed, the value of the time stamp corresponding to the parent partition is the split completion time of the parent partition, and the partition server stores a third partition range Inspector, the partition range of the third partition range checker is the partition range of the second target sub-partition, and the second target sub-partition is a sub-partition obtained by splitting the parent partition, and the method further includes : the partition server receives a third read request, the third read request is used to request to read a third data record from the second target sub-partition, the second target sub-partition is to perform The sub-partition obtained by splitting; the partition server obtains the timestamp carried by the third read request; if the timestamp carried by the third read request is greater than the timestamp corresponding to the parent partition, the partition server uses the The third partition range checker checks the partition range of the partition to which the row key value of the third data record belongs.

In the embodiment of the present application, the time stamp corresponding to the parent partition is compared with the time stamp carried in the read request to select a different partition range checker, which is beneficial to improve the accuracy of selecting a partition range checker.

In a possible implementation manner, the partition server stores multiple copies of the partition metadata records of the parent partition, and the method further includes: after splitting the parent partition into the multiple child partitions , the partition server modifies the multiple replicas into partition metadata records of the multiple sub-partitions.

In the embodiment of the present application, the partition metadata of the child partition is obtained by modifying the copy of the partition metadata of the parent partition, which is beneficial to simplify the process of generating the partition metadata of the child partition.

In a second aspect, a partition server for reading data records from a distributed system is provided, and the partition server includes various modules for executing the first aspect or any possible implementation manner of the first aspect.

The functions to be realized by the above modules can be realized by hardware, or can be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions.

In a third aspect, a partitioned server cluster is provided. The partitioned server cluster includes at least one partitioned server, and each partitioned server includes a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the cluster of partitioned servers executes the method in the first aspect above.

According to a fourth aspect, a computer program product is provided, and the computer program product includes: computer program code, when the computer program code is run on a computer, it causes the computer to execute the methods in the above aspects.

It should be noted that all or part of the above computer program code may be stored on the first storage medium, where the first storage medium may be packaged together with the processor, or may be packaged separately with the processor, and this embodiment of the present application does not make any Specific limits.

In a fifth aspect, a computer-readable medium is provided, the computer-readable medium stores program codes, and when the computer program codes are run on a computer, the computer is made to execute the methods in the above aspects.

Description of drawings

FIG. 1 is a schematic diagram of an architecture of a distributed system 100 applicable to an embodiment of the present application.

FIG. 2 is a schematic diagram of a data table 200 in a distributed system.

FIG. 3 is a flowchart of a method for splitting partitions according to an embodiment of the present application.

Fig. 4 is a flowchart of a method for reading data records from a distributed system according to another embodiment of the present application.

Fig. 5 is a flowchart of a method for reading data records from a distributed system according to another embodiment of the present application.

FIG. 6 is a schematic structural diagram of a partition server for writing data records according to an embodiment of the present application.

Fig. 7 is a schematic block diagram of a partitioned server cluster according to another embodiment of the present application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

For ease of understanding, first introduce the system architecture of the distributed system applicable to the embodiment of the present application with reference to FIG. 1 . The distributed storage system 100 shown in FIG. 1 includes a client 110 , a control server 120 , a partition server 130 and a storage server 140 .

The storage server 140 is used to provide shared storage space for data records and is responsible for the final persistence of data records. All partition servers can share data records in the distributed storage system.

The client (client) 110 can be understood as providing an interface for accessing the distributed system. The client can improve the access to the data records by storing the information of the distributed system in the cache, for example, the partition location information of the data table, etc.

The control server 120 is used to manage each partition server, monitor the status of each partition server in real time, and balance the load among each partition server. Usually, the control server also stores metadata records of the data table, partition metadata of each partition in the data table, and a routing table.

The partition metadata of the above partition includes the partition range of the partition described by the partition metadata and the partition server that manages the partition.

The above metadata of the data table is used to describe the row key value range of the row key contained in the data table, and the identifier of the partition contained in the data table.

A region server (region server) 130 is configured to maintain the allocated partitions and process read requests for reading data records from the partitions.

In a distributed system, data records can be stored in the form of files. Data records usually consist of one or more key-value pairs (see Table 1). Each data record contains a row key (row key) represented by "_id" , the key (key) is represented by "key", and the value is represented by "value". These data records can be logically presented in the form of data tables. In order to improve the access performance of data records in a distributed system, the data table can be split into multiple partitions in the horizontal direction. For example, the data table 200 in FIG. 2 can be divided into partition 1, ..., partition m, partition m+1, each partition contains a part of continuous row keys in the data table, and each partition contains the values of the continuous row keys A partition range that can represent a partition.

Table 1

{"_id":1,"key1":"value1","key2":"value2","key3":"value3"} {"_id":2,"key1":"value1","key2":"value2","key3":"value3"} {"_id":3,"key1":"value1","key2":"value2","key3":"value3"} {"_id":4,"key1":"value1","key2":"value2","key3":"value3"} {"_id":5,"key1":"value1","key2":"value2","key3":"value3"} {"_id":6,"key1":"value1","key2":"value2","key3":"value3","key4":"value4"}

In a distributed system, each partition has its own partition scope. The data records in each partition are organized and stored by the distributed system storage engine. The data records under the same partition can be scattered in different data files. Correspondingly, the same data file may contain data records from different partitions in the data table. When a partition is split, the partition range corresponding to the partition will also change. In order to ensure that the data records read by the read request are within the range of the partition, a partition range checker (checker) can be set in the partition server to check The partition range to which the row key value of the read data record belongs.

For example, partition A and partition B in the data table are managed on the partition server, the partition range of partition A is [1-100], the partition range of partition B is [101-200], and the data records of partition A are scattered in the data In the files F1, F2, and F3, at the same time, the data records of the partition B are also scattered in the data files F1, F2, and F3. At this time, if the read request requests to read the data records in partition A, the storage engine will read the data records from the files F1, F2, and F3 in turn. However, as mentioned above, the data files F1, F2, and F3 are stored If there are data records belonging to partition B, the storage engine will read the data records belonging to partition B while reading data files F1, F2, and F3. At this time, you can use the partition range checker of partition A to exclude Data records belonging to Partition B.

Considering the load and performance of the distributed system, it may be necessary to further split the partition on the basis of the existing partition to generate more partitions. For example, the concurrent performance of reading and writing can be improved through partition splitting. For another example, the load of different partition services in the distributed system can be balanced through the mechanism of partition splitting, that is, the parent partition in the target partition server with heavy load is split into multiple sub-partitions, and the sub-partitions are handed over to other partitions. Partition server management to offload the target partition server.

The following describes the method of partition splitting in combination with the architecture of the distributed system shown in FIG. 1 and in combination with FIG. 3 . FIG. 3 is a flowchart of a method for splitting partitions according to an embodiment of the present application. FIG. 3 includes step 310 to step 380 .

310. The control server selects the partition to be split as the parent partition according to the load of each partition server and the access frequency of data records in each partition managed by the partition server.

320. The control server applies to the distributed system for storage space for storing sub-partitions.

330. The control server sends a region split (region split) message to the target partition server where the parent partition is located. The partition split message carries the identification (ID) of the parent partition, the storage space applied for for the child partition, the partition metadata of the parent partition and other information.

340. The target partition server closes the partition service of the parent partition, that is, marks the parent partition as offline. At this time, if the target partition server receives a read request to read data records from the parent partition, the target partition server will report an exception to the client and inform the parent partition that it is not a service partition. Correspondingly, the client can perform compensatory retry later .

350. The target partition server splits the parent partition into two sub-partitions according to the partition metadata of the parent partition and the preset partition splitting strategy, and stores the partition metadata of the first sub-partition of the two sub-partitions to the above storage In the space, the partition metadata of the second child partition may occupy the storage space of the partition metadata of the parent partition.

360, the target partition server configures partition range checkers for both child partitions.

Specifically, after the target partition server splits the parent partition, it will modify the partition range of the partition range checker of the parent partition to the partition range of the first sub-partition as the partition range checker of the first sub-partition. Correspondingly, a partition range checker is regenerated as the partition range checker for the second sub-partition.

370. The target partition server sends a partition split success message to the control server.

380. The control server modifies information such as the partition information of the routing table, the metadata of the partition, and the metadata of the data table, so as to record the partition information of the sub-partition.

From the partition splitting process shown in Figure 3, it can be seen that the partition server will take the parent partition offline before splitting the parent partition, that is, the partition server will no longer process the read requests to read data records from the parent partition. Requests will affect the overall performance of the distributed distributed system and reduce the success rate of clients reading data.

In order to improve the overall performance of the distributed system and improve the success rate of the client to read data, the present application also provides a method for reading data records, which will be introduced in detail below based on the distributed system shown in FIG. 1 and in conjunction with FIG. 4 .

Fig. 4 is a flowchart of a method for reading data records from a distributed system according to another embodiment of the present application. The method shown in FIG. 4 includes step 410 and step 420 .

410. During the process of splitting the parent partition, the target partition server receives the first read request sent by the client, where the first read request is used to request to read the first data record from the parent partition.

That is to say, whether it is in the process of splitting the parent partition or preparing to split the parent partition, the parent partition is always online.

The process of splitting the parent partition by the target partition server may include the process of preparing to split the parent partition and splitting the parent partition. That is, the time period from when the target partition server receives a split request for the parent partition until the split of the parent partition is completed.

420. The target partition server reads the first data record from the parent partition according to the first read request.

In the embodiment of the present application, during the process of splitting the parent partition, the partition server can also read the data records requested by the first read request from the parent partition, avoiding the traditional partition splitting process. Previously, the parent partition had to be offline first, and the first read request for the data records in the parent partition could not be processed, which was conducive to improving the overall performance of the distributed system and improving the success rate of clients reading data.

There is also a situation in the process of processing the read request by the target partition server, that is, the completion time of reading the first data record by the target partition server may be later than the split completion time of the parent partition. For example, the first data record requested by the first read request has a large amount of data, and when the target partition server has read all the data records, the parent partition has been split into multiple sub-partitions. At this time, since the partition range checker of the parent partition has been modified to the partition range checker of the child partition, if the partition range checker of the child partition is used to check the partition to which the row key of the first data record belongs, the first data record In the records, a part of the data records that do not belong to the partition range of the partition range checker of the sub-partition cannot be read, which reduces the accuracy of reading data records from the distributed system.

For example, the partition range of the parent partition and the partition range of the parent partition's partition range checker is [1,100], the partition range of the first child partition and the partition range of the first child partition's partition range checker is [1,50], The partition range of the second sub-partition and the partition range of the partition range checker of the second sub-partition are [51,100], and the row key range of the first data record read by the first read request is [2,90]. When the target partition server is in the process of reading the first data record, after the partition range checker of the parent partition is modified to the partition range checker of the first sub-partition, only the partition range checker of the first sub-partition can be used to check the first sub-partition. A data record, at this time, the data record whose row key value does not belong to the partition range [1,50] in the first data record will be regarded as an error data record, and will not be returned to the client, reducing the accuracy of querying data records.

In order to avoid the above situation, the partition range checker of the parent partition may be reserved in the target partition server to check whether the row key value of the above first data record belongs to the parent partition.

That is, the time when the partition service completes the first read request is later than the split completion time of the parent partition, and the target partition server stores a first partition range checker, and the first partition range checker The partition range is the partition range of the parent partition, and the method further includes:

430. After the parent partition is split, the target partition server uses the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs.

The above partition server uses the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs. It can be understood that if the above first read request requests from the parent partition in the first file Reading the first data record, the partition server uses the first partition range checker to check the row key value of the read data record in the first file to obtain the data in the parent partition in the first file record, so that the partition server reads the first data record from the data record of the parent partition in the first file.

It should be noted that the above-mentioned first partition range checker (that is, the partition range checker of the parent partition) may always be stored in the target partition server. However, in order to save resources of the partition server, the first partition range checker may be deleted after the target partition server finishes processing the first read request.

That is to say, in this embodiment of the application, the read request can be associated with the partition range checker, so that it is of the same type as the first read request, that is, it is received during the splitting process of the parent partition, and the processing time is later than that of the parent partition The read request at the time of the split completion of can be associated with the first partition range checker.

For example, the partition range of the parent partition and the partition range of the parent partition's partition range checker is [1,100], the partition range of the first child partition and the partition range of the first child partition's partition range checker is [1,50], The partition range of the second sub-partition and the partition range of the partition range checker of the second sub-partition are [51,100], and the row key range of the first data record read by the first read request is [2,90]. Before the target partition server finishes reading the first data record, the first partition range checker will be saved in the target partition server, so that the first data record can be checked by the first partition range checker.

Correspondingly, after the split of the parent partition is completed, the read request received by the target partition server can be associated with the partition range checker of the child partition.

That is, the target partition server stores a second partition range checker, the partition range of the second partition checker is the partition range of the first target sub-partition, and the first target sub-partition is obtained by splitting the parent partition child partition, the method further includes: after the parent partition is split, the target partition server receives a second read request, and the second read request is used to request to read from the first target child partition second data record; the target partition server reads the second data record from the first target sub-partition; the target partition server checks the second data record with the second partition range checker The partition range of the partition to which the row key value belongs.

The above-mentioned first target sub-partition may be any one of multiple sub-partitions obtained by splitting the parent partition. The above-mentioned first target sub-partition may also be a sub-partition that needs to be managed by the target partition server among the multiple sub-partitions. Correspondingly, for load balancing reasons, other sub-partitions among the multiple sub-partitions may be assigned to other partition servers for management.

In order to simplify the generation process of the subpartition range checker (second partition range checker), you can first take a snapshot of the first partition range checker to get a copy of the first partition range checker, by modifying the first partition range checker A copy of the checker gets the second partition range checker. Of course, the second partition range checker can also be obtained by modifying the first partition range checker.

It should be noted that if the target partition server receives a read request to read data records from the parent partition after the parent partition is split, it can directly return a partition version error message to the client to notify the client Update partition version.

The target partition server can determine the relationship between the time of receiving the read request and the time of completion of partition splitting of the parent partition in various ways. For example, it can be judged by means of a preset time period. That is, the start time of partition splitting of the parent partition is taken as the start time of the preset time period, and the read requests received within the preset time period are the read requests that can be associated with the first partition range checker. Read requests received after a preset time period can be associated with the second partition range checker. For another example, it can also be judged by means of time stamp, and the specific judgment method will be introduced below.

In the process of splitting the parent partition, the value of the timestamp corresponding to the parent partition of the target partition server is the maximum value, and the method further includes: the target partition server according to the timestamp carried by the first read request and the timestamp corresponding to the parent partition, determining that the timestamp carried by the first read request is smaller than the timestamp corresponding to the parent partition; the target partition server uses the first partition range checker to check the first The partition range of the partition to which the row key value of the data record belongs.

It should be noted that the target partition server can maintain a timestamp for the parent partition (that is, the timestamp corresponding to the parent partition), and each pair of data records read from the read request of the parent partition needs to be compared with the timestamp of the parent partition. Compare to determine the partition range checker associated with the read request.

After the split of the parent partition is completed, the value of the time stamp corresponding to the parent partition is the split completion time of the parent partition, and the target partition server stores the partition range of the third partition checker as the second target The partition range of the sub-partition, the second target sub-partition is a sub-partition obtained by splitting the parent partition, and the method further includes: the target partition server receives a third read request, and the third read request It is used to request to read a third data record from the second target sub-partition, where the second target sub-partition is a sub-partition obtained by splitting the parent partition; the target partition server determines that the third The time stamp carried in the read request is greater than the time stamp corresponding to the parent partition; the target partition server uses the third partition range checker to check the partition range of the partition to which the row key value of the third data record belongs.

In the process of partition splitting, the target partition server needs to generate partition metadata of child partitions. In order to simplify the process of generating partition metadata, you can first take a snapshot of the partition metadata of the parent partition to generate partition metadata of multiple parent partitions. A copy of the data, such that the partition metadata of the parent partition can be modified to generate the partition metadata of the child partition. That is, the target partition server stores multiple copies of the partition metadata records of the parent partition, and the method further includes: after splitting the parent partition into the multiple sub-partitions, the target partition server will The multiple replicas are modified as partition metadata records for the multiple sub-partitions.

The multiple copies include the original partition metadata of the parent partition, and the number of the multiple copies may be equal to the number of the multiple sub-partitions, so as to generate the partition metadata of each sub-partition.

For example, if the parent partition is split into the first sub-partition and the second sub-partition, the partition range of the first sub-partition is from the start row key (start rowkey) of the parent partition to the row key corresponding to the split point, and the second sub-partition The partition range of a partition is from the row key corresponding to the split point to the end row key (end rowkey) of the parent partition. Then, the partition metadata of a parent partition can be regenerated by means of a snapshot, and then the partition metadata of the second child partition in the original partition metadata of the parent partition can be deleted, which is used to describe the row corresponding to the split point in the data table The metadata of the partition from the key to the end row key, as the partition metadata of the first child partition, is used to describe the start of the data table by deleting the partition metadata of the first child partition in the copy of the parent partition's partition metadata The metadata of the partition from the row key to the row key corresponding to the split point is used as the partition metadata of the second sub-partition.

It should be noted that the snapshot of the partition metadata of the parent partition and the snapshot of the partition range of the parent partition described above may or may not be executed at the same time, which is not limited in this embodiment of the present application.

In order to facilitate the understanding of the present application, the method in the embodiment of the present application is introduced below in combination with specific examples. It should be understood that the following examples are only for illustrating the methods of the embodiments of the present application, and do not limit the scope of the embodiments of the present application.

Fig. 5 is a flowchart of a method for reading data records from a distributed system according to another embodiment of the present application. The method shown in FIG. 5 includes steps 510 to . It should be noted that the preparation work for partition splitting can refer to the introduction in FIG. 3 . Method flow for processing read requests.

Assume that the parent partition to be split is partition A, and the partition server managing partition A is the target partition server.

510. The target partition server performs preparatory work before splitting the partition, including adjusting the timestamp corresponding to the parent partition to the maximum value, taking a snapshot of the partition metadata of partition A and the partition range checker of partition A, and obtaining the partition A's Partition metadata and a copy of the partition range checker for partition A.

520. The target partition server receives a first read request, where the first read request is used to request to read data records from partition A.

530. The target partition server judges the relationship between the timestamp carried in the first read request and the timestamp corresponding to the parent partition.

Since the value of the timestamp corresponding to the parent partition is the maximum value during the split process, the value of the timestamp carried in the first read request received during the split process is smaller than the value of the timestamp corresponding to the parent partition.

540. The target partition server associates the partition range checker of partition A with the first read request. That is, the partition range checker of partition A is used to check the partition to which the row key of the data record read by the first read request belongs.

550. The target partition server splits partition A into subpartition A' and subpartition B.

Specifically, the target partition server modifies the partition metadata of partition A to the partition metadata of sub-partition A', and modifies the copy of the partition metadata of partition A to the partition metadata of sub-partition B. The target partition server modifies the partition range checker of partition A to the partition range checker of subpartition A', and modifies the copy of partition A's partition range checker of subpartition B's partition range checker.

In addition, if the target partition server detects that the data read by the first read request has not been processed, the target partition server will also retain the partition range checker of partition A to check the rows of the data records read by the first read request. The partition to which the key belongs.

560. The target partition server adjusts the time stamp corresponding to partition A to the splitting completion time of partition A.

570. The target partition server receives the second read request.

It should be noted that if the partition version number carried in the second read request is different from the partition version number maintained by the target partition server for partition A, it means that the partition version stored by the client sending the second read request is expired. At this time, the target partition server Return a partition version error to the client to notify the client to update the partition version. If the partition version number carried in the second read request is the same as the partition version number maintained by the target partition server for partition A, and the second read request requests to read data records from sub-partition A', then perform step 580.

580. The target partition server judges the relationship between the timestamp carried in the second read request and the timestamp corresponding to partition A.

The timestamp carried by the second read request is greater than the timestamp corresponding to partition A, indicating that the second read request is sent after partition A is split. A', then go to step 590.

590. The target partition server processes the second read request.

The method of the embodiment of the present application is described in detail above with reference to FIG. 1 to FIG. 5 , and the apparatus for reading data records from a distributed system according to the embodiment of the present application is described in detail below in conjunction with FIG. 6 to FIG. 7 . It should be noted that the devices shown in FIG. 6 to FIG. 7 can implement each step in the above method, and for the sake of brevity, details are not repeated here.

FIG. 6 is a schematic structural diagram of a partition server for writing data records according to an embodiment of the present application. The partition server 600 shown in FIG. 6 includes: a receiving module 610 and a processing module 620 . The partition server is used to manage the parent partition in the data table, the partition server stores a first partition range checker, the partition range of the first partition range checker is the partition range of the parent partition,

The partition servers include:

The receiving module 610 is configured to receive a first read request sent by the client during the process of splitting the parent partition, where the first read request is used to request to read a first data record from the parent partition ;

The processing module 620 is configured to read the first data record from the parent partition according to the first read request, and the completion time of reading the first data record is later than the split completion time of the parent partition ;

The processing module 620 is further configured to use the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs.

Optionally, as an embodiment, the partition server stores a second partition range checker, the partition range of the second partition checker is the partition range of the first target sub-partition, and the first target sub-partition is For the child partition obtained by splitting the parent partition, the receiving module 610 is further configured to, after the split of the parent partition is completed, the partition server receive a second read request, and the second read request is used for Request to read a second data record from the first target sub-partition; the processing module 620 is further configured to read the second data record from the first target sub-partition; the processing module 620, It is also used to check the partition range of the partition to which the row key value of the second data record belongs by using the second partition range checker.

Optionally, as an embodiment, during the process of splitting the parent partition, the value of the timestamp corresponding to the parent partition is the maximum value, and the processing module 620 is further configured to: obtain the first read request The timestamp carried; if the timestamp carried by the first read request is smaller than the timestamp corresponding to the parent partition, select the first partition range checker to check the partition to which the row key value of the first data record belongs partition range.

Optionally, as an embodiment, after the split of the parent partition is completed, the value of the time stamp corresponding to the parent partition is the split completion time of the parent partition, and the partition server stores a third partition range Inspector, the partition range of the third partition range checker is the partition range of the second target sub-partition, the second target sub-partition is a sub-partition obtained by splitting the parent partition, and the receiving module 610 , is also used to receive a third read request, the third read request is used to request to read a third data record from the second target sub-partition, the second target sub-partition is to split the parent partition the sub-partition obtained; the processing unit 620 is further configured to obtain the timestamp carried by the third read request; the processing unit 620 is also configured to obtain the timestamp carried by the third read request greater than the For the timestamp corresponding to the parent partition, use the third partition range checker to check the partition range of the partition to which the row key value of the third data record belongs.

Optionally, as an embodiment, the partition server stores multiple copies of the partition metadata records of the parent partition, and the processing module 620 is further configured to: split the parent partition into the multiple After sub-partitions, modify the multiple replicas into partition metadata records of the multiple sub-partitions.

In an optional embodiment, the receiving module 610 may be an input/output interface 730, the processing module 620 may be a processor 720, and the partition server may further include a memory 710, as specifically shown in FIG. 7 .

Fig. 7 is a schematic block diagram of a partitioned server cluster according to another embodiment of the present application. The partition server cluster 700 shown in FIG. 7 may include at least one partition server, and each partition server includes: a memory 710 , a processor 720 and an input/output interface 730 . Wherein, the memory 710, the processor 720 and the input/output interface 730 are connected through an internal connection path, the memory 710 is used to store instructions, and the processor 720 is used to execute the instructions stored in the memory 720 to control the input/output interface 730 to receive Input data and information, output operation results and other data.

It should be noted that the above partition server cluster may include one partition server, and may also include multiple partition servers. When the partition server cluster includes multiple partition servers, the multiple partition servers cooperate with each other to implement the functions of the partition servers in the method shown in Figure 1-5. The specific structure of the partition server cluster is shown in Figure 7, that is, the partition server cluster can include multiple memory, multiple processors and multiple input/output interfaces. When the partition server cluster includes one partition server, the specific structure of the partition server can refer to the partition server structure in FIG. 7 , that is, the partition server cluster can include a memory, a processor and an input/output interface.

In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 720 or instructions in the form of software. The methods disclosed in the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory 710, and the processor 720 reads the information in the memory 710, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

It should be understood that in the embodiment of the present application, the processor may be a central processing unit (central processing unit, CPU), and the processor may also be other general processors, digital signal processors (digital signal processor, DSP), application specific integrated circuits ( application specific integrated circuit (ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

It should also be understood that in the embodiment of the present application, the memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

It should be understood that in this embodiment of the present application, "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B may also be determined according to A and/or other information.

It should be understood that the term "and/or" in this article is only an association relationship describing associated objects, which means that there may be three relationships, for example, A and/or B may mean: A exists alone, and A and B exist at the same time , there are three cases of B alone. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

It should be understood that, in various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the order of execution, and the execution order of the processes should be determined by their functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be read by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium, (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc (digital video disc, DVD)) or a semiconductor medium (for example, a solid state disk (solid state disk, SSD) )wait.

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be determined by the protection scope of the claims.

Claims (13)

1. A method for reading data records from a distributed system, wherein the distributed system includes a partition server and a client, the partition server is used to manage the parent partition in the data table, and the partition server stores having a first partition range checker whose partition range is the partition range of the parent partition, The methods include: During the process of splitting the parent partition, the partition server receives a first read request sent by the client, where the first read request is used to request to read a first data record from the parent partition; The partition server reads the first data record from the parent partition according to the first read request, and the completion time of reading the first data record is later than the split completion time of the parent partition; The partition server uses the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs. 2. The method according to claim 1, wherein the partition server stores a second partition range checker, the partition range of the second partition checker is the partition range of the first target sub-partition, and the The first target sub-partition is a sub-partition obtained by splitting the parent partition, The method also includes: After the parent partition is split, the partition server receives a second read request, where the second read request is used to request to read a second data record from the first target child partition; the partition server reads the second data record from the first target sub-partition; The partition server uses the second partition range checker to check the partition range of the partition to which the row key value of the second data record belongs. 3. The method according to claim 1, wherein during the process of splitting the parent partition, the value of the timestamp corresponding to the parent partition is the maximum value, and after the split of the parent partition is completed, The value of the time stamp corresponding to the parent partition is the splitting completion time of the parent partition. 4. The method of claim 3, further comprising: The partition server obtains the timestamp carried by the first read request; If the timestamp carried by the first read request is smaller than the timestamp corresponding to the parent partition, the partition server selects the first partition range checker to check the partition of the partition to which the row key value of the first data record belongs scope. 5. The method according to claim 3 or 4, wherein the partition server stores a third partition range checker, and the partition range of the third partition range checker is the partition range of the second target sub-partition , the second target sub-partition is a sub-partition obtained by splitting the parent partition, The method also includes: The partition server receives a third read request, the third read request is used to request to read a third data record from the second target sub-partition, the second target sub-partition is to split the parent partition The sub-partitions obtained; The partition server obtains the timestamp carried by the third read request; If the timestamp carried by the third read request is greater than the timestamp corresponding to the parent partition, the partition server uses the third partition range checker to check the partition of the partition to which the row key value of the third data record belongs scope. 6. The method according to any one of claims 1-5, wherein the partition server stores multiple copies of the partition metadata records of the parent partition, the method further comprising: After splitting the parent partition into the plurality of sub-partitions, the partition server modifies the plurality of copies into partition metadata records of the plurality of sub-partitions. 7. A partition server for reading data records from a distributed system, characterized in that, the distributed system includes the partition server and the client, the partition server is used to manage the parent partition in the data table, the The partition server stores a first partition range checker, the partition range of the first partition range checker is the partition range of the parent partition, The partition servers include: A receiving module, configured to receive a first read request sent by the client during the process of splitting the parent partition, where the first read request is used to request to read a first data record from the parent partition; A processing module, configured to read the first data record from the parent partition according to the first read request, and the completion time of reading the first data record is later than the split completion time of the parent partition; The processing module is further configured to use the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs. 8. The partition server according to claim 7, wherein the partition server stores a second partition range checker, and the partition range of the second partition checker is the partition range of the first target sub-partition, so The first target sub-partition is a sub-partition obtained by splitting the parent partition, The receiving module is further configured to, after the parent partition is split, the partition server receive a second read request, and the second read request is used to request to read the second target sub-partition from the first target child partition. data record; The processing module is further configured to read the second data record from the first target sub-partition; The processing module is further configured to use the second partition range checker to check the partition range of the partition to which the row key value of the second data record belongs. 9. The partition server according to claim 7, wherein during the process of splitting the parent partition, the value of the timestamp corresponding to the parent partition is the maximum value, and after the split of the parent partition is completed, , the value of the timestamp corresponding to the parent partition is the splitting completion time of the parent partition. 10. The partition server according to claim 9, wherein the processing module is further used for: Obtain the timestamp carried by the first read request; If the timestamp carried by the first read request is smaller than the timestamp corresponding to the parent partition, select the first partition range checker to check the partition range of the partition to which the row key value of the first data record belongs. 11. The partition server according to claim 9 or 10, wherein the partition server stores a third partition range checker, and the partition range of the third partition range checker is the partition of the second target sub-partition range, the second target sub-partition is a sub-partition obtained by splitting the parent partition, The receiving module is further configured to receive a third read request, the third read request is used to request to read a third data record from the second target sub-partition, the second target sub-partition is for the The child partition obtained by splitting the parent partition; The processing unit is further configured to obtain the timestamp carried by the third read request; The processing unit is further configured to use the third partition range checker to check whether the row key value of the third data record belongs to if the timestamp carried by the third read request is greater than the timestamp corresponding to the parent partition. The partition range of the partition. 12. The partition server according to any one of claims 7-11, wherein the partition server stores multiple copies of the partition metadata records of the parent partition, and the processing module is further configured to: After the parent partition is split into the plurality of sub-partitions, the plurality of copies are modified as partition metadata records of the plurality of sub-partitions. 13. A partition server cluster for reading data records from a distributed system, wherein the distributed system includes the partition server cluster and a client, the partition server cluster includes at least one partition server, and each partition The server includes a processor and a memory, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the partitioned server cluster executes any one of claims 1-6. Methods.

Images