CN110427422B - Data consistency check method, device and storage medium when data synchronization is abnormal - Google Patents

Abstract

The invention discloses a data consistency check method, device and storage medium when data synchronization is abnormal. The method includes: firstly inserting a transaction ID of a synchronization transaction into an auxiliary table in a target database; then when an error notification is received , to determine what kind of operation the current synchronization transaction is in; when it is a transaction change operation, determine whether the error notification belongs to the preset error type, and if it is, roll back the synchronization operation that has been performed and re-insert the transaction ID of the synchronization transaction Re-synchronize the transaction in the auxiliary table. When submitting the operation for the transaction, query whether the transaction ID of the synchronization transaction is contained in the auxiliary table, and if not, insert the transaction ID of the transaction into the auxiliary table and re-synchronize the operation. . In the synchronization process of the present invention, the transaction label is first registered in the auxiliary table at the beginning of the synchronization transaction, and if there is an error in the synchronization process, different judgment processing is performed according to the current operation to ensure the consistency of the synchronization data.

Description

Data consistency check method, device and storage medium when data synchronization is abnormal

technical field

The invention relates to the technical field of database synchronization, in particular to a data consistency check method, device and storage medium when data synchronization is abnormal.

Background technique

Among the database data real-time synchronization technologies, the data real-time synchronization technology based on database log analysis has the characteristics of supporting heterogeneous operating systems and database platforms, less intrusiveness to the source database system, and low data synchronization delay. Extensive research and application. The log format and log content between heterogeneous database systems are quite different, so the log content of the source database for data synchronization cannot be directly applied to the target database. The general principle of the real-time data synchronization technology based on log analysis is to capture the incremental log content of the source database, and then send it to the target through the network, and use the reverse SQL generation technology on the target to restore the transaction operation SQL statement of the source database, and then pass Common database access interfaces such as ODBC, OCI, etc., execute the restored SQL statement to realize real-time synchronization of data between the source and the target.

Further, the database data real-time synchronization device constructed by using the above-mentioned data real-time synchronization technology mainly includes the following components. At the source end of the synchronization system, the synchronization software log reading module reads the source end database log file in real time and captures the Transaction incremental log content; the synchronization software log analysis module parses the incremental log content in real time, extracts the transaction ID, operation table ID, transaction operation type such as INSERT, UPDATE, DELETE, transaction operation timestamp, log LSN value, operation type in the log Data, etc., and then assembled into messages in a specific format and stored in the sending queue; the synchronization software log sending module sends the messages in the sending queue to the target end of the synchronization system through the TCP/IP network.

At the target end of the synchronization system, the synchronization software log receiving module receives the message sent by the source end, parses the message content, and caches the operations belonging to the same transaction according to the transaction ID. When the COMMIT message of the transaction is received, it is sent to the log. The execution module performs synchronous execution; the synchronous software log execution module constructs the SQL statement according to the transaction operation content and executes it synchronously in the target database.

In the above real-time database data synchronization technology, when the target-side data synchronization service performs data storage on the target-side database, there are network abnormalities between the target-side synchronization service and the target-side database, or the target-side database fails and crashes. It may cause repeated execution of synchronous transactions and affect the consistency of synchronous data.

Therefore, the existing technology still needs to be improved and improved.

SUMMARY OF THE INVENTION

In view of the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a data consistency check method, device and storage medium when data synchronization is abnormal, so as to solve the problem of inconsistent synchronization data caused by abnormality in the current data synchronization process. question.

In order to achieve the above object, the present invention has adopted the following technical solutions:

A method for verifying data consistency when data synchronization is abnormal, comprising the following steps:

Receive the log message package sent by the source, insert the transaction ID of the synchronization transaction in the log message package into the auxiliary table created in the target database, and then perform the transaction synchronization operation;

When receiving an error notification from the target database, determine what operation the current synchronization transaction is in, wherein the operation includes a transaction change operation and a transaction commit operation performed in sequence;

When the current operation is a transaction change operation, determine whether the error notification belongs to the preset error type, roll back the synchronization operation performed when the error notification belongs to the preset error type, and re-insert the transaction ID of the synchronization transaction into the auxiliary table in and re-run the transaction synchronization operation;

When the current operation is a transaction commit operation, reconnect to the target database, and query whether the transaction ID of the synchronous transaction is contained in the auxiliary table, and if not, insert the transaction ID of the synchronous transaction into the auxiliary table and re-synchronize the transaction operate.

Preferably, in the data consistency check method when data synchronization is abnormal, the log message packet sent by the source end is received, and the transaction ID of the synchronization transaction in the log message packet is inserted into the auxiliary table created in the target end database The steps to perform transaction synchronization operations include:

Parse the log message package where the current synchronization transaction is located and obtain the transaction ID of the synchronization transaction in the log message package;

An insert operation is performed in the auxiliary table created in the target database to insert the transaction ID into the auxiliary table.

Preferably, in the data consistency check method when the data synchronization is abnormal, when the current operation is a transaction change operation, it is determined whether the error notification belongs to a preset error type, and when the error notification belongs to the preset error type The steps of rolling back the synchronization operation and re-inserting the transaction ID of the synchronization transaction into the auxiliary table and performing the transaction synchronization operation again include:

When the current operation is a transaction change operation, identify the type of error according to the error code returned by the target database;

Determine whether the error type is a preset error type;

When the error type is the preset error type, disconnect the connection with the target database and roll back the synchronization operation that has been performed, and re-insert the transaction ID of the synchronization transaction into the auxiliary table after the rollback is completed and repeat the operation Transaction synchronization operation.

Preferably, in the method for verifying data consistency when data synchronization is abnormal, when the error type is a preset error type, the connection with the target database is disconnected and the synchronization operation that has been performed is rolled back. , and re-insert the transaction ID of the synchronous transaction into the auxiliary table after the rollback is completed, and the steps of re-synchronizing the transaction also include:

When the error type is not the preset error type, the preset synchronous transaction error processing flow is executed.

Preferably, in the method for verifying data consistency when data synchronization is abnormal, the preset error types include network abnormal errors and database system-level errors.

Preferably, in the data consistency check method when data synchronization is abnormal, when the current operation is a transaction commit operation, reconnect to the target database, and query whether the auxiliary table contains the transaction ID of the synchronization transaction, and The steps for inserting the transaction ID of the synchronized transaction into the auxiliary table and re-synchronizing the transaction when not included include:

When the current operation is a transaction commit operation, disconnect from the target database and reconnect to the target database;

Call the auxiliary table in the target database, and query whether there is a transaction ID registered at the beginning of the synchronous transaction in the auxiliary table;

When the transaction ID does not exist in the auxiliary table, insert the synchronization transaction ID into the auxiliary table in the target database and restart the transaction synchronization operation.

Preferably, in the method for verifying data consistency when data synchronization is abnormal, when the transaction ID does not exist in the auxiliary table, insert the synchronization transaction ID into the auxiliary table in the target database and start again The steps of performing a transaction synchronization operation also include:

When the transaction ID exists in the auxiliary table, the next synchronization transaction is executed.

Preferably, in the method for verifying data consistency when data synchronization is abnormal, the transaction change operation includes an insert operation, an update operation and a delete operation.

A data consistency check device when data synchronization is abnormal, comprising: a processor, a memory and a communication bus;

A computer-readable program executable by the processor is stored on the memory;

The communication bus implements connection communication between the processor and the memory;

When the processor executes the computer-readable program, the steps in the above-mentioned method for verifying data consistency when data synchronization is abnormal is implemented.

A computer-readable storage medium, the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors, so as to realize the abnormality of data synchronization as described above. Steps in a data consistency check method.

In the method, device and storage medium for data consistency verification when data synchronization is abnormal, the method includes: firstly receiving a log message packet sent by the source end, and inserting the transaction ID of the synchronization transaction in the log message packet into the target The transaction synchronization operation is performed after the auxiliary table created in the end database; then when an error notification from the target end database is received, it is judged which operation the current synchronization transaction is in, wherein the operation includes the transaction change operation and transaction commit performed in sequence Operation; when the current operation is a transaction change operation, determine whether the error notification belongs to the preset error type, and when the error notification belongs to the preset error type, roll back the performed synchronization operation and re-insert the transaction ID of the synchronization transaction Re-run the transaction synchronization operation in the auxiliary table. When the current operation is a transaction commit operation, reconnect to the target database, and query whether the transaction ID of the synchronization transaction is contained in the auxiliary table, and if it does not contain the transaction ID of the synchronization transaction The ID is inserted into the auxiliary table and the transaction synchronization operation is repeated. In the synchronization process of the present invention, the transaction label is first registered in the auxiliary table at the beginning of the synchronization transaction. If there is an error in the synchronization process, it is judged whether the transaction label has been registered according to the current operation, so as to further determine whether the synchronization has been completed. Ensure the consistency of synchronized data.

Description of drawings

FIG. 1 is a flowchart of a method for verifying data consistency when data synchronization is abnormal provided by the present invention.

FIG. 2 is a flowchart of the step S100 in the method for verifying data consistency when data synchronization is abnormal provided by the present invention.

FIG. 3 is a flowchart of the step S300 in the method for verifying data consistency when data synchronization is abnormal provided by the present invention.

FIG. 4 is a flowchart of the step S400 in the method for verifying data consistency when data synchronization is abnormal provided by the present invention.

FIG. 5 is a schematic diagram of the running environment of the preferred embodiment of the data consistency check program when the data synchronization is abnormal according to the present invention.

6 is a functional block diagram of a preferred embodiment of a system for installing a data consistency check program when data synchronization is abnormal according to the present invention.

Detailed ways

In view of the problem that synchronization data inconsistency is prone to occur when an exception occurs in the data synchronization process in the prior art, the purpose of the present invention is to provide a data consistency check method, device and storage medium when the data synchronization is abnormal, which can ensure that the data synchronization process is There is no problem of synchronization data inconsistency when an exception occurs in the system.

In order to make the objectives, technical solutions and effects of the present invention clearer and clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Referring to FIG. 1, the method for verifying data consistency when data synchronization is abnormal provided by the present invention includes the following steps:

S100. Receive the log message package sent by the source end, and insert the transaction ID of the synchronization transaction in the log message package into the auxiliary table created in the target end database, and then perform the transaction synchronization operation.

Specifically, the source side sends the tables that need to be synchronized in the form of log message packets. Therefore, the source side also needs to generate log message packets before sending log message packets. In a heterogeneous database system environment, The incremental log information generated by the source database cannot be directly applied to the target database. Therefore, it is necessary to parse the log information of the source database, and then generate a message packet in a specific format inside the synchronization service, which is stored in the sending queue and waiting for the log. The sending module sends. When implemented, the log capture module of the source data synchronization service captures the incremental log data of the source database system in real time based on the log capture technology, and then transmits the incremental log data to the log analysis module for analysis. The log parsing module of the source-side data synchronization service parses the incremental log into an internal specific message packet format and stores it in the sending queue. Preferably, the log message package includes transaction ID, transaction operation type, operation table name, column information and data value, and the like. When the target end performs data synchronization, it classifies the received synchronization transactions according to the transaction ID. Afterwards, the log sending module of the source data synchronization service, based on the thread synchronization mechanism, sends the log message packets in the internal sending queue to the target data synchronization service system through the TCP/IP network. Preferably, in order to ensure the reliability and data integrity of data transmission, the log sending module at the source end adopts a complete message response mechanism. The log sending module only considers the data transmission complete after receiving the confirmation message from the target end, otherwise it will automatically retransmit Data, after receiving the log message packet, the target end will serially number each log message packet and need to return confirmation information.

Further, the target-side data synchronization service interacts with the target-side database through the database driver, so the synchronization service and the database may not be on the same machine. After they are arranged separately, it is necessary to consider the possibility of grid failure between them; in addition, during the synchronization process, the target database may crash due to a bug or other reasons. In these cases, the synchronization When the service performs synchronization on the target database, these possible faults need to be considered to prevent repeated execution or less execution of the synchronization transaction. Therefore, the present invention first creates an auxiliary table in the target database, and the auxiliary table is used to store The transaction tag of the synchronization transaction. In this embodiment, the transaction tag is the transaction ID. When the synchronization transaction starts, the transaction ID of the current synchronization transaction is firstly registered in the auxiliary table, and then when an error occurs during the synchronization process, the current synchronization operation To do different judgment processing to ensure the consistency of synchronization data; after the transaction ID is registered, the synchronization transaction starts. A specific synchronization transaction is as follows:

BEGIN

INSERT INTO T VALUES(1);

COMMIT;

END;

The structure of a specific auxiliary table is shown in the following table:

field name Field Type Field Description TRXID NUMBER Register the transaction ID executed during data synchronization

Please refer to FIG. 2, which is a flowchart of the step S100, including the following steps:

S101, parse the log message package in which the current synchronization transaction is located and obtain the transaction ID of the synchronization transaction in the log message package;

S102. Perform an insert operation in the auxiliary table created in the target database to insert the transaction ID into the auxiliary table.

For the above specific operation transactions, before executing the synchronous transaction operation, perform an INSERT operation on the auxiliary table to insert the transaction ID into the auxiliary table. The specific example is as follows:

INSERT INTO DMHS_TRXID(TRXID)VALUES(12345);

S200. When receiving an error notification from the target database, determine which operation the current synchronization transaction is in, wherein the operation includes a transaction change operation and a transaction commit operation performed in sequence.

In this embodiment, when no error notification from the target database is received, the synchronization transaction is executed according to the process, and then the next synchronization transaction is executed. When an error notification from the target database is received (that is, an error occurs during the synchronization process) ), it is necessary to determine what operation the current synchronization transaction is in. Generally speaking, when a synchronization transaction is performed, a transaction change operation needs to be performed first to perform data conversion, and then a transaction commit operation is performed after the conversion is completed to transfer the converted data to the In the target database, the present invention needs to specifically provide the processing method after the error by judging the operation type. In different operation stages, the processing methods are different. In the transaction change operation stage, the data conversion has not been completed, so it needs to be re-executed. Data conversion, in the transaction commit operation stage, the data conversion has been completed but may not be successfully stored, so it is necessary to verify the transaction ID to ensure the consistency of synchronized data. Preferably, the transaction change operation includes an insert operation (INSERT operation), an update operation Operation (UPDATE operation) and delete operation (DELETE operation), data can be inserted, updated or deleted during synchronization.

In addition, since there are many types of errors, in order to distinguish errors, the present invention sets an error code (errorcode) to indicate the type of errors. Identify and display errors through the display of error codes, and the system can quickly find the type of errors through error code identification.

S300. When the current operation is a transaction change operation, determine whether the error notification belongs to a preset error type, and when the error notification belongs to the preset error type, roll back the performed synchronization operation and re-insert the transaction ID of the synchronization transaction Auxiliary table and redo the transaction synchronization operation.

In this embodiment, the type of error is first identified according to the error notification (error code in this embodiment), and if the type of error belongs to a preset type, step S100 is re-executed. The specific flowchart of step S300 is shown in FIG. 3 shown.

Please refer to FIG. 3, which is a flowchart of the step S300, including the following steps:

S301. When the current operation is a transaction change operation, identify the type of error according to the error code returned by the target database;

S302. Determine whether the error type belongs to a preset error type;

S303. When the error type is a preset error type, disconnect the connection with the target database and roll back the synchronization operation that has been performed, and re-insert the transaction ID of the synchronization transaction into the auxiliary table after the rollback is completed. Retry the transaction synchronization operation.

In this embodiment, the preset error types include abnormal network errors and database system-level errors, which are presented in the form of error codes when reporting errors. If such errors occur, it means that when performing an insert operation, an update operation, or a delete operation , the data synchronization is not completed correctly. At this time, in order to avoid the import of abnormal data into the target database, it is necessary to disconnect the connection with the target database first, and then perform the synchronization operation again. The present invention adopts the rollback method to realize the resynchronization of the synchronization operation. Rollback refers to the act of restoring the program or data to the last correct state through a program or data processing error. Through the rollback method, the data can be synchronized and restored to the correct state, so as to achieve the purpose of ensuring the consistency of synchronized data. .

Further, the step S303 also includes:

When the error type is not the preset error type, the preset synchronous transaction error processing flow is executed.

Specifically, because other errors may occur in the synchronization process, the present invention sets a synchronization transaction error processing flow for non-preset errors, and the error processing flow can be set according to actual needs. The transaction error handling process is not limited. For example, when a non-preset error occurs, the synchronization of the current transaction is directly canceled, and the synchronization operation of the next transaction is performed, etc., so as to ensure that the system will not stagnate when synchronizing transactions, and Affects the synchronization process.

For the synchronization transaction in the above-mentioned specific embodiment, when the synchronization transaction operation INSERTINTO T VALUES(1) is performed on the target database, if an error is reported synchronously, the error code returned by the target database needs to be judged to identify the type of error. For network exceptions or database system-level errors, it is necessary to disconnect the connection to roll back the previous synchronization operation, and then resynchronize the erroneous transaction.

S400. When the current operation is a transaction commit operation, reconnect to the target database, and query whether the transaction ID of the synchronous transaction is included in the auxiliary table, and if not included, insert the transaction ID of the synchronous transaction into the auxiliary table and perform the operation again. Transaction synchronization operation.

Specifically, when a commit operation is performed to the target-side database, a network failure occurs while waiting for the return of the commit operation, resulting in an execution error. The remote target-side database may have completed the commit operation but cannot return to the target-side synchronization service. Here In this case, the target-side synchronization service needs to reconnect to the target-side database, and query the registered transaction ID in the auxiliary table to confirm whether the transaction has been successfully submitted when the last failure occurred. When the operation returns, the database is abnormally down, resulting in an execution error. When the database is down, it is also possible that the commit operation has been completed but has not been returned to the target-side synchronization service. In this case, the target-side synchronization service needs to reconnect to the target side. In the database, the registered transaction ID is queried in the auxiliary table to confirm whether the transaction has been successfully submitted during the last failure. Therefore, the present invention checks whether the transaction ID is included in the auxiliary table, and can confirm whether the synchronization transaction has been submitted successfully when the failure occurs. If unsuccessful, re-synchronize the transaction to ensure the consistency of synchronized data.

Please refer to FIG. 4, which is a flowchart of the step S400, including the following steps:

S401, when the current operation is a transaction commit operation, disconnect the connection with the target database and reconnect the target database;

S402, call the auxiliary table in the target database, and query whether there is a transaction ID registered when the synchronization transaction starts in the auxiliary table;

S403. When the transaction ID does not exist in the auxiliary table, insert the synchronization transaction ID into the auxiliary table in the target database and restart the transaction synchronization operation.

The present invention confirms whether the synchronization data has been successfully submitted by querying the transaction ID, and when the auxiliary table and the table to be synchronized are in the same transaction, the ACID characteristics of the transaction (Atomicity, Consistency, Consistency, Isolation (Isolation), Durability (Durability) to ensure the consistency of synchronized data in the event of grid or database failure.

Preferably, the step S403 further includes:

When the transaction ID exists in the auxiliary table, the next synchronization transaction is executed.

In other words, when judging that the transaction ID exists in the auxiliary table, the present invention can confirm that the synchronization has been successful, thereby ending the synchronization of the current transaction and transferring to the synchronization of the next transaction, so as to ensure the consistency of the synchronization data.

For the synchronous transaction in the above-mentioned specific embodiment, when executing the transaction commit operation on the target database, if an error is reported, it is necessary to disconnect and reconnect to the target database, and then query whether the auxiliary table contains transactions, Execute the query SELECT TRXID FROM DMHS_TRXID WHERE TRXID=12345; if a result set with a transaction ID of 12345 is found, it means that the transaction has been successfully synchronized, and the synchronization of the current transaction is ended and the synchronization of the next transaction is performed; otherwise, the transaction is resynchronized.

As shown in FIG. 5 , based on the above-mentioned data consistency check method when data synchronization is abnormal, the present invention also provides a data consistency check device when data synchronization is abnormal, and the data consistency check device when data synchronization is abnormal It can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palmtop computer and a server. The device for verifying data consistency when data synchronization is abnormal includes a processor 10 , a memory 20 and a display 30 . FIG. 5 only shows some components of the data consistency check device when the data synchronization is abnormal, but it should be understood that it is not required to implement all the shown components, and more or less components may be implemented instead.

In some embodiments, the memory 20 may be an internal storage unit of the data consistency check device when the data synchronization is abnormal, for example, a hard disk or a memory of the data consistency check device when the data synchronization is abnormal. In other embodiments, the memory 20 may also be an external storage device of the data consistency check device when the data synchronization is abnormal, for example, a pluggable hard disk equipped on the data consistency check device when the data synchronization is abnormal. , smart memory card (SmartMedia Card, SMC), secure digital (Secure Digital, SD) card, flash memory card (Flash Card) and so on.

Further, the memory 20 may also include both an internal storage unit of the data consistency check device when the data synchronization is abnormal and an external storage device. The memory 20 is used for storing application software and various types of data installed in the data consistency check device when the data synchronization is abnormal, such as program codes of the data consistency check device when the data synchronization is abnormal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a data consistency check program 40 when the data synchronization is abnormal, and the data consistency check program 40 can be executed by the processor 10 when the data synchronization is abnormal, so as to realize the various embodiments of the present application. The data consistency check method when the data synchronization is abnormal.

In some embodiments, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a microprocessor or other data processing chips, which are used to execute program codes or process data stored in the memory 20, such as Execute the data consistency check method when the data synchronization is abnormal, and the like.

In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, and the like. The display 30 is used for displaying the information of the data consistency checking device when the data synchronization is abnormal and for displaying a visual user interface. When the data synchronization is abnormal, the components 10-30 of the data consistency checking device communicate with each other through the system bus.

In one embodiment, when the processor 10 executes the data consistency check program 40 when the data synchronization in the memory 20 is abnormal, the following steps are implemented:

Receive the log message package sent by the source, insert the transaction ID of the synchronization transaction in the log message package into the auxiliary table created in the target database, and then perform the transaction synchronization operation;

When receiving an error notification from the target database, determine what operation the current synchronization transaction is in, wherein the operation includes a transaction change operation and a transaction commit operation performed in sequence;

When the current operation is a transaction change operation, determine whether the error notification belongs to the preset error type, roll back the synchronization operation performed when the error notification belongs to the preset error type, and re-insert the transaction ID of the synchronization transaction into the auxiliary table in and re-run the transaction synchronization operation;

When the current operation is a transaction commit operation, reconnect to the target database, and query whether the transaction ID of the synchronous transaction is contained in the auxiliary table, and if not, insert the transaction ID of the synchronous transaction into the auxiliary table and re-synchronize the transaction operate.

The transaction modification operations include insert operations, update operations and delete operations.

Further, the step of receiving the log message package sent by the source and inserting the transaction ID of the synchronization transaction in the log message package into the auxiliary table created in the target database includes:

Parse the log message package where the current synchronization transaction is located and obtain the transaction ID of the synchronization transaction in the log message package;

An insert operation is performed in the auxiliary table created in the target database to insert the transaction ID into the auxiliary table.

Further, when the current operation is a transaction change operation, it is judged whether the error notification belongs to a preset error type, and when the error notification belongs to the preset error type, the synchronization operation that has been performed is rolled back and the synchronization transaction is reset. The steps for inserting the transaction ID into the secondary table and re-synchronizing the transaction include:

When the current operation is a transaction change operation, identify the type of error according to the error code returned by the target database;

Determine whether the error type is a preset error type;

When the error type is the preset error type, disconnect the connection with the target database and roll back the synchronization operation that has been performed, and re-insert the transaction ID of the synchronization transaction into the auxiliary table after the rollback is completed and repeat the operation Transaction synchronization operation.

Further, when the error type is a preset error type, disconnect the connection with the target database and roll back the synchronization operation that has been performed, and re-insert the transaction ID of the synchronization transaction into the auxiliary after the rollback is completed. The steps for re-synchronizing the transaction in the table also include:

When the error type is not the preset error type, the preset synchronous transaction error processing flow is executed.

Further, the preset error types include abnormal network errors and database system-level errors.

Further, when the current operation is a transaction commit operation, reconnect the target database, and query whether the transaction ID of the synchronous transaction is contained in the auxiliary table, and if not included, insert the transaction ID of the synchronous transaction into the auxiliary table. And the steps for re-synchronizing the transaction include:

When the current operation is a transaction commit operation, disconnect from the target database and reconnect to the target database;

Call the auxiliary table in the target database, and query whether there is a transaction ID registered at the beginning of the synchronous transaction in the auxiliary table;

When the transaction ID does not exist in the auxiliary table, insert the synchronization transaction ID into the auxiliary table in the target database and restart the transaction synchronization operation.

Further, when the transaction ID does not exist in the auxiliary table, the steps of inserting the synchronization transaction ID into the auxiliary table in the target database and restarting the transaction synchronization operation further include:

When the transaction ID exists in the auxiliary table, the next synchronization transaction is executed.

Please refer to FIG. 6 , which is a functional block diagram of a preferred embodiment of a system for installing a data consistency check program when data synchronization is abnormal according to the present invention. In this embodiment, the system for installing the data consistency check program when the data synchronization is abnormal can be divided into one or more modules, the one or more modules are stored in the memory 20, and the one or more modules are composed of one or more modules. A processor (in this embodiment, the processor 10) is executed to complete the present invention. For example, in FIG. 6 , the system for installing the data consistency check program when data synchronization is abnormal can be divided into a transaction label insertion module 21, an operation judgment module 22, a change operation error processing module 23 and a commit operation error processing module 24. The module referred to in the present invention refers to a series of computer program instruction segments capable of completing specific functions, and is more suitable for describing the data consistency check program when the data synchronization is abnormal than the program. Data consistency check when the data synchronization is abnormal The execution process in the device. The following description will specifically describe the functions of the modules 21-24.

The transaction label insertion module 21 is used for receiving the log message package sent by the source end, and inserting the transaction ID of the synchronization transaction in the log message package into the auxiliary table created in the target end database to perform transaction synchronization operation;

The operation judging module 22 is used for judging what kind of operation the current synchronous transaction is in when receiving the error notification of the target database, wherein the operation includes a transaction change operation and a transaction commit operation performed in turn;

The change operation error processing module 23 is used to determine whether the error notification belongs to a preset error type when the current operation is a transaction change operation, and when the error notification belongs to the preset error type, roll back the synchronization operation that has been performed and re-register the operation. The transaction ID of the synchronization transaction is inserted into the auxiliary table and the transaction synchronization operation is performed again;

The commit operation error processing module 24 is used to reconnect the target database when the current operation is a transaction commit operation, and query whether the transaction ID of the synchronous transaction is contained in the auxiliary table, and if not contained, the transaction ID of the synchronous transaction Insert into the secondary table and re-sync the transaction.

Wherein, the transaction change operation includes an insert operation, an update operation and a delete operation.

Further, the transaction tag insertion module 21 specifically includes:

The transaction label acquisition unit is used to parse the log message package in which the current synchronization transaction is located and obtain the transaction ID of the synchronization transaction in the log message package;

The transaction label insertion unit is used to perform an insert operation in the auxiliary table created in the target database to insert the transaction ID into the auxiliary table.

The change operation error processing module 23 specifically includes:

The error receiving unit is used to identify the type of error according to the error code returned by the target database when the current operation is a transaction change operation;

An error type judgment unit, used to judge whether the error type belongs to a preset error type;

The first error processing unit is used for disconnecting the connection with the target database and rolling back the synchronization operation performed when the error type is a preset error type, and re-transaction of the synchronization transaction after the rollback is completed The ID is inserted into the auxiliary table and the transaction synchronization operation is repeated.

Preferably, the first error processing unit is further configured to execute a preset synchronization transaction error processing flow when the error type is not a preset error type.

Preferably, the preset error types include abnormal network errors and database system-level errors.

The commit operation error handling module 24 specifically includes:

The connection disconnection unit is used to disconnect the connection with the target database and reconnect to the target database when the current operation is a transaction commit operation;

a query unit, used to call an auxiliary table in the target database, and query whether there is a transaction ID registered at the beginning of the synchronous transaction in the auxiliary table;

The second error processing unit, when the transaction ID does not exist in the auxiliary table, inserts the synchronization transaction ID into the auxiliary table in the target database and restarts the transaction synchronization operation.

Preferably, the second error processing unit is further configured to execute the next synchronization transaction when the transaction ID exists in the auxiliary table.

To sum up, in the data consistency check method, device and storage medium provided by the present invention when data synchronization is abnormal, the method includes: firstly receiving the log message package sent by the source end, and converting the synchronization transaction in the log message package After the transaction ID is inserted into the auxiliary table created in the target database, the transaction synchronization operation is performed; then when an error notification from the target database is received, it is judged what operation the current synchronization transaction is in, wherein the operation includes the transactions executed in sequence Change operation and transaction commit operation; when the current operation is a transaction change operation, determine whether the error notification belongs to the preset error type, and roll back the synchronization operation performed when the error notification belongs to the preset error type and re-synchronize the Insert the transaction ID of the transaction into the auxiliary table and perform the transaction synchronization operation again. When the current operation is a transaction commit operation, reconnect to the target database, and query whether the transaction ID of the synchronization transaction is contained in the auxiliary table, and if not The transaction ID of the synchronization transaction is inserted into the auxiliary table and the transaction synchronization operation is performed again. In the synchronization process of the present invention, the transaction label is first registered in the auxiliary table at the beginning of the synchronization transaction. If there is an error in the synchronization process, it is judged whether the transaction label has been registered according to the current operation, so as to further determine whether the synchronization has been completed. Ensure the consistency of synchronized data.

Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware (such as processors, controllers, etc.) through a computer program, and the programs can be stored in a In the computer-readable storage medium, when the program is executed, the processes of the above-mentioned method embodiments may be included. The storage medium may be a memory, a magnetic disk, an optical disk, or the like.

It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above descriptions, and all these improvements and transformations should belong to the protection scope of the appended claims of the present invention.

Claims (7)

1. A data consistency checking method when data synchronization is abnormal is characterized by comprising the following steps:

receiving a log message packet sent by a source end, inserting a transaction ID of a synchronous transaction in the log message packet into an auxiliary table created in a target end database, and then performing transaction synchronization operation;

when an error report notification of a target end database is received, judging which operation the current synchronous transaction is in, wherein the operation comprises a transaction change operation and a transaction commit operation which are executed in sequence;

when the current operation is the transaction change operation, judging whether the error notification belongs to a preset error type, rolling back the performed synchronization operation when the error notification belongs to the preset error type, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again; the preset error types comprise network abnormal errors and database system level errors;

when the current operation is a transaction submitting operation, the target end database is reconnected, a result set corresponding to the transaction ID is inquired after the transaction ID of the synchronous transaction registered in the auxiliary table is inquired, and when the result set of the transaction ID is not inquired, the transaction ID of the synchronous transaction is reinserted into the auxiliary table and the transaction synchronizing operation is carried out again;

the receiving source end sends a log message packet, and the step of performing transaction synchronization operation after inserting the transaction ID of the synchronization transaction in the log message packet into an auxiliary table created in a target end database comprises the following steps:

analyzing a log message packet where the current synchronous transaction is located and acquiring a transaction ID of the synchronous transaction in the log message packet;

performing an insert operation in an auxiliary table created in a target-side database to insert the transaction ID into the auxiliary table;

when the current operation is a transaction submitting operation, the target end database is reconnected, the result set corresponding to the transaction ID is inquired after the transaction ID of the synchronous transaction registered in the auxiliary table is inquired, and when the result set of the transaction ID is not inquired, the transaction ID of the synchronous transaction is reinserted into the auxiliary table and the transaction synchronizing operation is carried out again, the steps comprise:

when the current operation is a transaction submitting operation, disconnecting the target end database and reconnecting the target end database;

calling an auxiliary table in a target end database, and inquiring a result set corresponding to the transaction ID after inquiring the transaction ID registered when the synchronous transaction starts in the auxiliary table;

and when the result set of the transaction ID is not inquired, reinserting the transaction ID of the synchronous transaction into the auxiliary table and performing the transaction synchronization operation again.

2. The method according to claim 1, wherein the step of determining whether the error notification belongs to a preset error type when the current operation is a transaction change operation, and rolling back the performed synchronization operation and inserting the transaction ID of the synchronized transaction into the auxiliary table again and performing the transaction synchronization operation again when the error notification belongs to the preset error type comprises:

when the current operation is a transaction change operation, identifying the type of an error according to an error code returned by a target end database;

judging whether the error type belongs to a preset error type or not;

and when the error type is the preset error type, disconnecting the target end database, rolling back the performed synchronization operation, and after the rolling back is completed, inserting the transaction ID of the synchronization transaction into the auxiliary table again and performing the transaction synchronization operation again.

3. The method for checking data consistency when data synchronization is abnormal according to claim 2, wherein the step of disconnecting the target database and rolling back the performed synchronization operation when the error type is a preset error type, and reinserting the transaction ID of the synchronized transaction into the auxiliary table and performing the transaction synchronization operation again after the rolling back is completed further comprises:

and when the error type is not the preset error type, executing a preset error processing flow of the synchronous transaction.

4. The method for checking data consistency when data synchronization is abnormal according to claim 1, wherein the step of reinserting the transaction ID of the synchronized transaction into the auxiliary table and resuming the transaction synchronization operation when the result set of the transaction ID is not queried further comprises:

when the result set of the transaction ID is queried, the next synchronized transaction is executed.

5. The method for checking data consistency when data synchronization is abnormal according to claim 1, wherein the transaction change operation comprises an insert operation, an update operation and a delete operation.

6. A data consistency check device when data synchronization is abnormal is characterized by comprising: a processor, a memory, and a communication bus;

the memory has stored thereon a computer readable program executable by the processor;

the communication bus realizes connection communication between the processor and the memory;

the processor, when executing the computer readable program, implements the steps of the data consistency checking method when data synchronization is abnormal according to any one of claims 1 to 5.

7. A computer-readable storage medium, storing one or more programs, which are executable by one or more processors, to implement the steps of the data consistency checking method when data synchronization is abnormal according to any one of claims 1 to 5.

Images