CN116010412A - Database full synchronization breakpoint continuous transmission method, system and storage medium - Google Patents

Abstract

The present invention relates to the field of information technology, and in particular to a method, system and storage medium for full synchronous breakpoint resume transmission of a database. times, sort the batches; record the start value and end value, and store them persistently in external storage; establish a subtask thread pool and a sending window that matches the number of subtask threads; when a breakpoint recovery occurs, read from the external storage Get the start value and end value of the segmentation field corresponding to the batch; construct the query condition, and query all the data satisfying the start value and end value of the segmentation field from the corresponding table in the database, as the restored batch, the restored Batches are submitted to subtask threads for synchronization. The beneficial technical effects of the present invention include: the granularity of retransmission can be reasonably controlled, and the balance between the amount of retransmitted data and the amount of concurrency can be achieved; the speed of breakpoint recovery is fast, and it is easy to practice.

Description

A method, system, and storage medium for full-volume synchronous breakpoint resuming of database

technical field

The present invention relates to the field of information technology, in particular to a method, system and storage medium for full-scale synchronous breakpoint resume transmission of databases.

Background technique

The current mainstream database full-volume synchronization technology cannot implement resuming transmission from breakpoints or control the granularity of resuming transmission from breakpoints well, and cannot achieve a balance between the degree of concurrency and the amount of retransmitted data. The traditional full database synchronization method re-enters the database by retrying when some data fails to be stored in the database. It is helpless for scenarios such as program process crashes and machine restarts, and cannot achieve resuming transmission from breakpoints. Although there is a public technology, one of the ways to realize the resuming of the full database synchronization breakpoint is to mark the synchronization progress in real time during the synchronization process, and continue the transmission from the marked breakpoint when restoring. However, this method has the problems of difficulty in implementing multi-threaded concurrent synchronization scenarios and low efficiency in single-threaded synchronization scenarios. Therefore, it is necessary to study database synchronization technology that can improve the efficiency of data synchronization.

The prior art discloses a high-speed, high-stability data synchronization method based on the HTTP protocol, which includes a central system data synchronization module and a sub-node data synchronization module. The data entry management module of the central system is used to process the updated business data, and call the data update notification module of the central system after processing and storage. The notification module sends a notification to the data update monitoring module of the child node through the message queue link, and the data update monitoring module The module then calls the data synchronization request module of the child node to initiate a request to the synchronization data download module of the central system, obtains the requested version data, and recycles the request until the maximum version number of the business data of the child node and the maximum version number of the central system Stop data synchronization after consistency, wait for the next update notification from the central system or arrive at the timing period, and then request new version data from the synchronization data download module of the central system. Its technical solution can quickly, efficiently and stably perform incremental data synchronization between a central system and multiple sub-nodes even when the network is unstable. However, its technical solution cannot realize the control of the granularity of resuming transmission from breakpoints, and is not suitable for multi-threaded parallel execution.

Contents of the invention

The technical problem to be solved by the present invention is the lack of a database synchronization solution that supports resumable transfer from breakpoints and is suitable for multi-thread parallel execution. A method, system, and storage medium for full synchronous breakpoint resume transmission of databases are proposed, which can realize breakpoint resume transmission, and the breakpoint granularity is controllable, which is suitable for multi-thread parallel execution.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solution: a method for resuming data from a breakpoint in full synchronization of a database, comprising the following steps:

Select the segmentation field of each table of the database, divide the table data into several batches according to the segmentation field, and sort the batches;

Record the start value and end value of each batch corresponding to the segmentation field, and store the sort number of the batch and the corresponding table name, start value and end value in an external storage persistently;

Set up a subtask thread pool and a sending window matched with the number of subtask threads, and submit subtask threads within the sending window for synchronization, and update the sending window position after the batch synchronization is completed;

When a breakpoint recovery occurs, read the start value and end value of the segmentation field of the corresponding batch from the external storage;

Construct a query condition, query all data satisfying the start value and end value of the segmentation field from the corresponding table of the database, and submit the recovered batch to the subtask thread for synchronization as a restored batch.

Preferably, the method for selecting the segmentation field includes:

If the table of the database is a single primary key table, then select the segmentation field of the table as the primary key;

If the table of the database is a joint primary key, the field with the highest degree of discrimination is selected as the splitting field of the table.

As a preference, the method for calculating the degree of discrimination of a field is: degree of discrimination=count(distinct field name)/count(field name), count(distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, count (field name) refers to the number of entries in the table.

Preferably, the method of dividing the table data into several batches according to the division field includes:

Set the number of divided batches to n, and read the minimum value change step m of the segmented field;

Calculate the division step step, step=max(count(segmentation field)/n,m);

Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch.

Preferably, the method for establishing the sending window includes:

Setting the size of the sending window equal to the number of subtask threads;

The starting position of the sending window is set in the first batch, and an ack pointer is established to point to the starting position of the sending window, and the ack pointer is persistently stored in an external storage.

As a preference, the external storage also stores the synchronization state of the batches, the initial value of the synchronization state is waiting for scheduling, after the subtask thread pool is established, the synchronization state of the batches submitted to the subtask threads is set as sending, After the batch synchronization is completed, update the synchronization status of the corresponding batch as successful, and if the batch synchronization is interrupted, retry the synchronization immediately until the synchronization is completed. The method for updating the position of the sending window is: when the sending window After the batch synchronization at the starting position is successful, the ack pointer is incremented by 1 until the end position of the sending window reaches the last batch.

Preferably, when a breakpoint recovery occurs, restore the position of the sending window according to the ack pointer, determine the current batch that needs to be restored according to the restored position of the sending window, and submit the restored batch to the child Task threads are synchronized.

A system for resuming full-scale synchronous breakpoint transfer of databases, which is used to perform the above-mentioned method for resuming full-scale synchronous breakpoint transfer of databases, including a cluster of management nodes, a cluster of coordinators, and a cluster of synchronization nodes. The cluster of management nodes includes at least one management nodes, the coordinator cluster includes at least one coordinator, the sync node cluster includes at least two sync nodes,

The management node selects the segmentation field of each table of the database of the data source, divides the table data into several batches according to the segmentation field, sorts the batches, and records the corresponding Segment the start value and end value of the field, and store the batch sort number and the corresponding table name, start value, and end value persistently to the external storage.

The two synchronization nodes run respectively on the host where the source database and the target database are located, and the synchronization node corresponding to the source database establishes a subtask thread pool and a sending window matching the number of subtask threads, and will be in the sending window The batch submission subtask thread in the batch synchronization is performed. After the batch synchronization is completed, the synchronization node corresponding to the source database updates the position of the sending window. When a breakpoint recovery occurs, the synchronization node corresponding to the source database The node reads the start value and end value of the segmentation field corresponding to the batch from the external storage, and the synchronization node corresponding to the source database constructs a query condition, and queries the data satisfying the segmentation field from the corresponding table of the database. All the data of the start value and the end value are used as a restored batch, and the synchronization node corresponding to the source database submits the restored batch to the subtask thread for synchronization,

The coordinator establishes a network connection between the management node and several synchronization nodes.

Preferably, the synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database, and the subtask thread of the synchronization node corresponding to the target database completes the batch After the transmission, the digital fingerprints are checked, and if the verification is consistent, it is fed back that the batch synchronization is successful; otherwise, if the verification is inconsistent, it is fed back that the batch synchronization fails.

As a preference, when the management node selects the segmentation field of each table of the database of the data source, the following steps are performed:

If the table of the database is a single primary key table, then select the segmentation field of the table as the primary key;

If the table of the database is a joint primary key, select the field with the highest degree of discrimination as the segmentation field of the table, and the method of calculating the degree of discrimination of the field is: degree of discrimination=count(distinct field name)/count(field name), count(distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, and count(field name) refers to the number of entries in the table.

Preferably, when the management node divides the table data into several batches according to the segmentation fields, the following steps are performed:

Set the number of divided batches to n, and read the minimum value change step m of the segmented field;

Calculate the division step step, step=max(count(segmentation field)/n,m);

Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch.

A computer system, the computer system includes a memory, a processor, and a computer program stored in the memory and operable on the processor, when the computer program is executed by the processor, the aforementioned one A database full synchronization breakpoint resume upload method.

A computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for full-volume synchronous resumable download of a database is implemented.

The beneficial technical effects of the present invention include: the data volume of each batch of data obtained by segmentation during full synchronization is approximately equal, and can be segmented according to character fields without being affected by character sets; the granularity of retransmission can be reasonably controlled, Reach the balance between the amount of retransmitted data and the amount of concurrency; only need to store data segmentation and a small amount of state information to the external storage system, the speed of recovery from breakpoints is fast, and it is easy to practice.

Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings.

Description of drawings

Below in conjunction with accompanying drawing, the present invention will be further described:

FIG. 1 is a schematic flowchart of a method for synchronous breakpoint resume transmission according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of a method for dividing table data into several batches according to an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a synchronous breakpoint resume transmission system according to an embodiment of the present invention.

FIG. 4 is a schematic structural diagram of a computer system according to an embodiment of the present invention.

Among them: 10. Management node cluster, 20. Coordinator cluster, 30. Synchronization node cluster, 40. Computer system, 41. Memory, 42. Computer program, 43. Processor.

Detailed ways

The technical solutions of the embodiments of the present invention will be explained and described below in conjunction with the accompanying drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all of them. Based on the examples in the implementation manners, other examples obtained by those skilled in the art without making creative efforts all belong to the protection scope of the present invention.

In the following description, terms such as "inner", "outer", "upper", "lower", "left", "right" etc. appearing to indicate orientation or positional relationship are only for the convenience of describing the embodiment and simplifying the description, rather than Nothing indicating or implying that a referenced device or element must have a particular orientation, be constructed, and operate in a particular orientation should therefore not be construed as limiting the invention.

A method for resuming a full-volume synchronization of a database, please refer to Figure 1, comprising the following steps:

Step A01) Select the segmentation field of each table of the database, divide the table data into several batches according to the segmentation field, and sort the batches;

Step A02) Record the start value and end value of each batch corresponding to the segmentation field, and store the batch sort sequence number and corresponding table name, start value, and end value persistently in external storage;

Step A03) Establishing a subtask thread pool and a sending window matching the number of subtask threads, synchronizing the batch submission subtask threads within the sending window, and updating the position of the sending window after the batch synchronization is completed;

Step A04) When a breakpoint recovery occurs, read the start value and end value of the segmentation field of the corresponding batch from the external storage;

Step A05) Construct the query condition, query all the data satisfying the start value and end value of the segmentation field from the database corresponding table, and submit the restored batch to the subtask thread for synchronization as a restored batch. Construct a batch of start-stop interval information and store it persistently in an external system. Each interval represents a synchronous subtask that can be executed concurrently. In this way, the granularity of resuming transmission from breakpoints can be controlled, and it is suitable for data synchronization methods executed in parallel by multiple threads. Each thread can be responsible for the synchronization of multiple subtasks, and the subtasks can be sorted in time.

The method of selecting the split field includes: if the database table is a single primary key table, select the split field of the table as the primary key; if the database table is a joint primary key, select the field with the highest degree of discrimination as the split field of the table .

The method of calculating the discrimination degree of a field is: discrimination degree=count(distinct field name)/count(field name), count(distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, count(field name ) refers to the number of entries in the table.

Please refer to Figure 2, the method of dividing the table data into several batches according to the segmentation field includes:

Step B01) Set the number of divided batches to n, and read the minimum value change step m of the segmented field;

Step B02) Calculate the division step step, step=max(count(segmentation field)/n,m);

Step B03) Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch.

Take the segmentation of numeric fields as an example:

Taking 100 records as an example, the values of this field are 1, 2, 3...99,5000, the minimum value is 1, and the maximum value is 5000. The goal is to split into 5 parts. Traditional or other patented methods: Calculate the maximum and minimum values of the field and construct an arithmetic sequence to segment the data. The segmentation results are [1,1000],[1001,2000],[2001,3000],[3001,4000],[ 4001,5000]. The data segmentation of the method in this embodiment: the result is [1,20], [21,40], [41,60], [61,80], [81,5000].

It can be seen that in the case of uneven distribution of table data, there are large errors in the segmentation of traditional or other patented methods, which will cause a large amount of data to accumulate in a certain batch, and the amount of data in some batches is so small that it is impossible to achieve data The relative balance of distribution. The data segmentation method in the method of this embodiment can solve the problem of data skew.

Take the segmentation of character fields as an example:

Take 1000 randomly generated 6-digit string records containing only 0-9, a-z as an example. The minimum value of this field is 001222 and the maximum value is ffe732 in lexicographical order. The goal is to divide into 10 parts. Traditional method: cannot be segmented. Segmentation method of the open source database full synchronization software dataX: do string mapping, the specific process is to traverse the string from left to right, the length of the string is k, and the product construction of the acsii value of the i-th string and 128^(k-i) A real number, then accumulated and transformed into a numeric field for segmentation, and reversely mapped back to a string after the segmentation is completed. The method in this embodiment: use the natural order of the index to scan the index for fast segmentation without returning to the table.

As can be seen from the segmentation results, the segmentation method used by dataX has a mapping between strings and numbers, and the resulting intervals are mapped back to strings and sorted under different character sets. The strings in the previous interval are not necessarily smaller than the strings in the previous interval. That is, affected by the character set, there are interval repetitions and data skew problems. The segmentation result of the method in this embodiment has nothing to do with the character set, and the visualization is good.

The data segmentation method makes full use of the index of the database, and does not need to return to the table when only scanning the index, and the speed is very fast. Compared with the method of calculating the maximum and minimum values of fields and constructing an arithmetic sequence to segment data, the data segmentation method in this embodiment can avoid the problem of data skew in the case of unbalanced table data distribution, and realize the data in each batch. The amount of data is roughly equal; compared with the traditional method, it solves the problem that it cannot be segmented according to the string type field; compared with the way of mapping strings to numbers and then segmenting, this embodiment uses the natural order of the index The feature solves the problem of being inaccurately affected by the character set.

Wherein, the method for setting up the sending window in this embodiment includes: setting the size of the sending window equal to the number of subtask threads; setting the starting position of the sending window in the first batch, setting up the ack pointer to point to the starting position of the sending window, The ack pointer is persistently stored in external storage.

The external storage also stores the synchronization state of the batch. The initial value of the synchronization state is waiting for scheduling. After the subtask thread pool is established, the synchronization state of the batch submitted to the subtask thread is set as sending. After the batch synchronization is completed, update The synchronization status of the corresponding batch is successful, if the batch synchronization is interrupted, retry the synchronization immediately until the synchronization is completed. The method of updating the position of the sending window is: when the batch synchronization at the starting position of the sending window is successful, the ack pointer will be incremented 1, until the end of the sending window reaches the last batch.

During the synchronization process, according to the set subtask task thread pool size, a sending window of the same size is constructed, and the corresponding numbered subtasks in the sending window are submitted to the thread pool concurrently. If the subtask fails or times out, it will rejoin the task thread pool for retrying. Whenever a subtask is completed, update the sign of the number array in the sending window, and try to update the ack pointer and slide the sending window backward. The ack pointer is currently The position before and after indicates the task number that has been completed. The location of the pointer is persisted to the storage system after each update of the ack pointer.

When a breakpoint recovery occurs, the position of the sending window is restored according to the ack pointer, the batches to be restored are determined according to the position of the restored sending window, and the restored batches are submitted to the subtask thread for synchronization.

The beneficial technical effects of this embodiment include: the data volume of each batch of data obtained during full synchronization is approximately equal, and can be segmented according to character fields without being affected by character sets; the granularity of retransmission can be reasonably controlled , to achieve a balance between the amount of retransmitted data and the amount of concurrency; only need to store data segmentation and a small amount of state information to the external storage system, the speed of recovery from breakpoints is fast, and it is easy to practice.

A database full-volume synchronous resuming breakpoint transfer system, used to execute the above-mentioned full-scale database synchronous breakpoint resuming method, please refer to the accompanying drawing 3, including a management node cluster 10, a coordinator cluster 20 and a synchronization node cluster 30, The management node cluster 10 includes at least one management node, the coordinator cluster 20 includes at least one coordinator, and the synchronization node cluster 30 includes at least two synchronization nodes,

The management node selects the segmentation field of each table in the database of the data source, divides the table data into several batches according to the segmentation field, sorts the batches, and records the start of each batch corresponding to the segmentation field value and end value, to store the batch sort number and corresponding table name, start value and end value persistently to external storage,

Two synchronization nodes run on the host where the source database and the target database are located respectively. The synchronization node corresponding to the source database establishes a subtask thread pool and a sending window matching the number of subtask threads, and submits the batches within the sending window to the subtask thread After the batch synchronization is completed, the synchronization node corresponding to the source database updates the position of the sending window. When a breakpoint recovery occurs, the synchronization node corresponding to the source database reads the initial value and The end value, the synchronization node corresponding to the source database constructs the query condition, and queries all the data satisfying the start value and end value of the segmentation field from the corresponding table of the database. As the restored batch, the synchronization node corresponding to the source database will restore the batch submitted to the subtask thread for synchronization,

The coordinator establishes network connections between the management node and several synchronization nodes.

The synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database. After the subtask thread of the synchronization node corresponding to the target database completes the batch transmission, it checks the digital fingerprint. If the verification is consistent, the batch is fed back If the synchronization is successful, otherwise, if the verification is inconsistent, the feedback batch synchronization fails.

When the management node selects the segmentation fields of each table in the database of the data source, perform the following steps:

If the database table is a single primary key table, the split field of the selected table is the primary key;

If the table of the database is a joint primary key, select the field with the highest degree of discrimination as the splitting field of the table, and the method of calculating the degree of discrimination of the field is: degree of discrimination=count(distinct field name)/count(field name), count( distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, and count (field name) refers to the number of entries in the table.

When the management node divides the table data into several batches according to the segmentation field, perform the following steps:

Set the number of divided batches to n, and read the minimum value change step m of the segmented field;

Calculate the division step step, step=max(count(segmentation field)/n,m);

Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch.

On the other hand, this embodiment provides a verification example of the method for resuming the breakpoint and full synchronization of the database.

The source database is a mysql database, the target database business system is a postgresql type database, and the table creation statement of the test.table table in the demo library is as follows:

CREATE TABLE `test_table` (

`sid` varchar (6) NOT NULL,

`name` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

`type` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

`city` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,

PRIMARY KEY (`id`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Randomly generate 1000 6-digit string records containing only 0-9, a-z as the value of the field sid, and the contents of other fields are random values. The value of the sid field is lexicographically the minimum value is 001222, the maximum value is ffe732, the number of split pre-batches is set to 10, the minimum step size is 10000, and the thread pool size of the synchronization subtask is 5. Now perform a full synchronization operation on this table.

Implementation steps:

1. Use this patented system to synchronize table data, fill in the corresponding source database information and target server information, and then automatically generate a synchronization mapping relationship with one click;

2. Select the segmentation field. Since this table is a single primary key table, select the sid field as the segmentation field.

3. Execute the character segmentation algorithm, execute select sid from test_table wheresid > '001222' limit 100,1 for the first query; get the result 1b52a7, and execute select sid from test_table where sid > '1b52a7' limit 100,1 for the second query ; And so on, iterating until the result set is empty. Collect the results of each query and construct the synchronization interval [001222,1b52a7),[1b52a7,332ab6),[332ab6,50799a),[50799a,69a3d2),[69a3d2,84bcfb),[84bcfb,a1a75d),[a1a75d,b8398c ),[b8398c,d35be2),[d35be2,ecae00),[ecae00,ffe732]. The numbers are 1, 2..., 10 respectively, and the interval information is persisted to the external storage system. Each synchronization interval represents a subtask that can be executed concurrently, and the subtasks are independently responsible for data synchronization within the interval.

4. Initialize the ack pointer to the position of 0, construct a sending window with a size of 5, and concurrently submit 5 subtasks to the thread pool for full synchronization. If a subtask fails or times out, it will rejoin the task thread pool for retrying. Whenever a subtask is completed, update the sign of the number array in the sending window, and try to update the ack pointer and slide the sending window backward. The current and previous positions of the ack pointer indicate the number of the task that has ended. The location of the pointer is persisted to the storage system after each update of the ack pointer. After t1 seconds, the subtask numbered 3 completes the synchronization. At this time, the ack pointer does not meet the update condition and still points to the position of 0. If the subtask numbered 2 fails after t2 seconds, the subtask will be added to the task thread pool for retrying. After t3 seconds, the subtask numbered 2 has completed the synchronization. At this time, the ack pointer does not meet the update condition and still points to the position of 0. After t4 seconds, the subtask numbered 1 completes the synchronization. At this time, the ack pointer meets the update condition, the pointer is moved to the position of 3, and the pointer position is persisted to the external storage. Slide the sending window back 3 units.

5. After t5 seconds, a power failure event occurs on the machine where the synchronization system is located, and the synchronization is interrupted.

6. After the machine restarts, initiate full synchronization again, read the interval information and the position of the ack pointer from the external storage system, and reconstruct the running status array. The starting position of the reconstructed sending window is at number 4, and the position of the ack pointer is at number 3. Concurrently submit a total of 5 subtasks numbered 4-8 to the subtask thread pool, and then repeat the process of "synchronization and status marking" until the data synchronization of all batches is completed.

On the other hand, the embodiment of the present application provides a computer system, referring to FIG. 4 , the computer system 40 includes a memory 41, a processor 43, and a computer program 42 stored in the memory 41 and operable on the processor 43, When the computer program 42 is executed by the processor 43, it realizes the above-mentioned method for full data synchronization and resuming transmission from breakpoints.

The computer system can be a general purpose computer system or a special purpose computer system. In a specific implementation, the computer system may be a server cluster including multiple servers, such as a blockchain system including multiple nodes. Those skilled in the art can understand that FIG. 4 is only an example of a computer system, and does not constitute a limitation to the computer system. It may include more or less components than those shown in the illustration, or combine certain components, or different components, such as It may also include input and output devices, network access devices, etc.

The processor 43 can be a central processing unit (Central Processing Unit, CPU), and the processor 43 can also be other general processors 43, a digital signal processor 43 (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general purpose processor 43 may be a microprocessor 43 or any conventional processor 43 .

The memory 41 may be an internal storage unit of the computer system in some embodiments, such as a hard disk or memory of the computer system. The memory 41 may also be an external storage device of the computer system in other embodiments, such as a plug-in hard disk equipped on the computer system, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash memory Card (Flash Card), etc. Further, the memory 41 may also include both an internal storage unit of the computer system and an external storage device. The memory 41 is used to store operating system, application program, boot loader (Boot Loader), data and other programs, etc. The memory 41 can also be used to temporarily store data that has been output or will be output.

On the other hand, the embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program 42. When the computer program 42 is executed by the processor 43, it realizes the above-mentioned full-scale database synchronization breakpoint continuation. pass method.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto, and those skilled in the art should understand that the present invention includes but is not limited to the accompanying drawings and the description in the above specific embodiments content. Any modifications that do not depart from the functional and structural principles of the present invention will be included in the scope of the claims.

Claims (13)

1. A method for full synchronous breakpoint resume transmission of a database, characterized in that, Include the following steps: Select the segmentation field of each table of the database, divide the table data into several batches according to the segmentation field, and sort the batches; Record the start value and end value of each batch corresponding to the segmentation field, and store the sort number of the batch and the corresponding table name, start value and end value in an external storage persistently; Set up a subtask thread pool and a sending window matched with the number of subtask threads, and submit subtask threads within the sending window for synchronization, and update the sending window position after the batch synchronization is completed; When a breakpoint recovery occurs, read the start value and end value of the segmentation field of the corresponding batch from the external storage; Construct a query condition, query all data satisfying the start value and end value of the segmentation field from the corresponding table of the database, and submit the recovered batch to the subtask thread for synchronization as a restored batch. 2. a kind of database full amount synchronous breakpoint resume transmission method according to claim 1, is characterized in that, Methods for selecting split fields include: If the table of the database is a single primary key table, then select the segmentation field of the table as the primary key; If the table of the database is a joint primary key, the field with the highest degree of discrimination is selected as the splitting field of the table. 3. a kind of database full amount synchronous breakpoint resume transmission method according to claim 2, is characterized in that, The method of calculating the discrimination degree of a field is: discrimination degree=count(distinct field name)/count(field name), count(distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, count(field name ) refers to the number of entries in the table. 4. A method for resuming transmission of a full amount of database synchronously according to any one of claims 1 to 3, characterized in that, The methods of dividing table data into several batches according to the division fields include: Set the number of divided batches to n, and read the minimum value change step m of the segmented field; Calculate the division step step, step=max(count(segmentation field)/n,m); Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch. 5. A method for resuming transmission of a full amount of database synchronously according to any one of claims 1 to 3, characterized in that, Methods for establishing a send window include: Setting the size of the sending window equal to the number of subtask threads; The starting position of the sending window is set in the first batch, and an ack pointer is established to point to the starting position of the sending window, and the ack pointer is persistently stored in an external storage. 6. a kind of database full amount synchronous breakpoint resume transmission method according to claim 5, is characterized in that, The external storage also stores the synchronization state of the batch, the initial value of the synchronization state is waiting for scheduling, after the subtask thread pool is established, the synchronization state of the batch submitted to the subtask thread is set as sending, and the batch After the second synchronization is completed, update the synchronization status of the corresponding batch as successful, and if the batch synchronization is interrupted, retry the synchronization immediately until the synchronization is completed. The method for updating the position of the sending window is: when the starting position of the sending window After the batch synchronization is successful, the ack pointer is incremented by 1 until the end position of the sending window reaches the last batch. 7. a kind of database full amount synchronous breakpoint resume transmission method according to claim 5, is characterized in that, When a breakpoint recovery occurs, restore the position of the sending window according to the ack pointer, determine the batches that currently need to be restored according to the restored position of the sending window, and submit the restored batches to the subtask thread for execution. Synchronize. 8. A system for full-volume synchronous breakpoint resuming of databases, which is used to execute a method for full-scale synchronous breakpoint resuming of databases according to any one of claims 1 to 7, wherein: Including a management node cluster, a coordinator cluster and a synchronization node cluster, the management node cluster includes at least one management node, the coordinator cluster includes at least one coordinator, and the synchronization node cluster includes at least two synchronization nodes, The management node selects the segmentation field of each table of the database of the data source, divides the table data into several batches according to the segmentation field, sorts the batches, and records the corresponding Segment the start value and end value of the field, and store the batch sort number and the corresponding table name, start value, and end value persistently to the external storage. The two synchronization nodes run respectively on the host where the source database and the target database are located, and the synchronization node corresponding to the source database establishes a subtask thread pool and a sending window matching the number of subtask threads, and will be in the sending window The batch submission subtask thread in the batch synchronization is performed. After the batch synchronization is completed, the synchronization node corresponding to the source database updates the position of the sending window. When a breakpoint recovery occurs, the synchronization node corresponding to the source database The node reads the start value and end value of the segmentation field corresponding to the batch from the external storage, and the synchronization node corresponding to the source database constructs a query condition, and queries the data satisfying the segmentation field from the corresponding table of the database. All the data of the start value and the end value are used as a restored batch, and the synchronization node corresponding to the source database submits the restored batch to the subtask thread for synchronization, The coordinator establishes a network connection between the management node and several synchronization nodes. 9. A kind of database full amount synchronous breakpoint resume transmission system according to claim 8, is characterized in that, The synchronization node corresponding to the source database extracts the digital fingerprint of the batch and sends it to the synchronization node corresponding to the target database, and after the subtask thread of the synchronization node corresponding to the target database completes the batch transmission, The digital fingerprints are checked, and if the verification is consistent, it is fed back that the batch synchronization is successful; otherwise, if the verification is inconsistent, the batch synchronization failure is fed back. 10. A kind of database full synchronization breakpoint resume transmission system according to claim 8 or 9, characterized in that, When the management node selects the segmentation fields of each table of the database of the data source, the following steps are performed: If the table of the database is a single primary key table, then select the segmentation field of the table as the primary key; If the table of the database is a joint primary key, select the field with the highest degree of discrimination as the segmentation field of the table, and the method of calculating the degree of discrimination of the field is: degree of discrimination=count(distinct field name)/count(field name), count(distinct field name) refers to the number of entries in the table after deduplicating the field values in the table, and count(field name) refers to the number of entries in the table. 11. A kind of database full amount synchronous breakpoint resume transmission system according to claim 10, is characterized in that, When the management node divides the table data into several batches according to the segmentation fields, the following steps are performed: Set the number of divided batches to n, and read the minimum value change step m of the segmented field; Calculate the division step step, step=max(count(segmentation field)/n,m); Divide the value of the segmentation field into several intervals according to the division step step, and the table entries corresponding to each interval are regarded as a batch. 12. A computer system, characterized in that the computer system comprises a memory, a processor, and a computer program stored in the memory and operable on the processor, the computer program being executed by the processor Realize a kind of database full amount synchronous breakpoint resume transmission method as described in any one of claims 1 to 7 at the same time. 13. A computer-readable storage medium, characterized in that, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, one of the methods described in any one of claims 1 to 7 is implemented. The method of resuming the breakpoint and full synchronization of the database.

Images