CN116501539A - Data processing method and related device - Google Patents

Abstract

The application discloses a data processing method and related devices. The first device acquires a transaction table, which is used to indicate the state of the transaction and the storage address of the log chain. Each transaction corresponds to a log chain, and the log chain includes at least one chain node, each chain node indicates the storage address of a tuple in the transaction; the first device obtains the log chain of the transaction to be restored according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be restored; the first device obtains the transaction according to The log chain of the transaction to be restored determines the storage address of the tuple of the transaction to be restored, so as to obtain the tuple of the transaction to be restored; the first device deletes the log chain and the tuple of the transaction to be restored. In this application, the transaction to be recovered can be queried through the transaction table, and the tuple of the transaction to be recovered can be obtained according to the log chain of the transaction to be recovered, so that there is no need to perform a full table scan on the first device, which improves data recovery, The efficiency of data processing.

Description

A data processing method and related device

technical field

The embodiments of the present application relate to the field of computer technologies, and in particular, to a data processing method and related devices.

Background technique

Write behind logging (WBL) is a new logging and recovery protocol designed for non-volatile memory (NVM). The key idea is to record which parts of the database have changed data, not how the data has changed. Using WBL requires persisting the transaction before persisting the log, which is used to track the data modified by the transaction for undoing uncommitted transactions.

Zen's no-log policy is an implementation of WBL. In the Zen mechanism, the log information is written into each tuple of the transaction, reducing the write operations on the log. However, when facing data recovery scenarios, Zen needs to scan the data of the entire table in the database twice to identify uncommitted transactions and roll back the data.

Since the database stores a lot of data content, scanning the data of the entire table twice requires a large system overhead, and the data recovery time is too long.

Contents of the invention

The embodiments of the present application provide a data processing method and a related device, which are used to improve the efficiency of data processing.

In the first aspect, the embodiment of the present application provides a data processing method. The first device obtains a transaction table, which is used to indicate the state of the transaction and the storage address of the log chain. Each transaction corresponds to a log chain. The log chain includes At least one chain node, each chain node indicates the storage address of a tuple in the transaction; the first device obtains the log chain of the transaction to be restored according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be restored; A device determines the storage address of the tuple of the transaction to be restored according to the log chain of the transaction to be restored, so as to obtain the tuple of the transaction to be restored; the first device deletes the log chain and the tuple of the transaction to be restored.

In this application, the transaction to be recovered can be queried through the transaction table, and the tuple of the transaction to be recovered can be obtained according to the log chain of the transaction to be recovered, so that there is no need to perform a full table scan on the first device, which improves data recovery, The efficiency of data processing. On the other hand, since the log chain in this application does not carry the tuple information of the transaction itself, it can avoid the overhead of additional write operations in the process of using the log chain, thus compared with the log file carrying the transaction The write-ahead logging (WAL) mechanism of tuples also has great performance advantages.

Based on the first aspect, in an optional implementation manner, the log chain includes multiple serial chain nodes, and there is an association relationship among the multiple serial chain nodes.

Based on the first aspect, in an optional implementation manner, the storage address of the log chain is the storage address of the head node of the log chain.

Based on the first aspect, in an optional implementation manner, the first device obtains the log chain of the transaction to be restored according to the transaction table, including:

According to the transaction table, the first device determines that the transaction in the undo state is the transaction to be recovered;

The first device determines the storage address of the log chain of the transaction to be recovered according to the transaction table;

The first device obtains the log chain of the transaction to be restored according to the storage address of the log chain of the transaction to be restored.

Based on the first aspect, in an optional implementation manner, the method is applied to the primary and standby database systems, the first device is the primary database, the second device is the standby database, and the transaction table is also used to indicate the commit time of the transaction. The method also includes :

The first device determines the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest backup time of the primary and standby database systems;

The first device obtains the log chain of the incremental transaction according to the transaction table;

The first device determines the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

The first device synchronizes the log chain and tuple of the incremental transaction to the second device.

Based on the first aspect, in an optional implementation manner, the first device obtains the log chain of the incremental transaction according to the transaction table, including:

The first device determines the storage address of the log chain of the incremental transaction according to the transaction table;

The first device acquires the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction.

In the second aspect, the embodiment of the present application provides a data processing method, which is characterized in that the method is applied to the primary and secondary database systems, the first device is the primary database, and the second device is the secondary database, and the method includes:

The first device obtains a transaction table, the transaction table is used to indicate the commit time of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, the log chain includes at least one chain node, and each chain node indicates a tuple in the transaction storage address;

The first device determines the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest data synchronization time of the active and standby database systems;

The first device obtains the log chain of the incremental transaction according to the transaction table;

The first device determines the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

The first device synchronizes the log chain and tuple of the incremental transaction to the second device.

Based on the second aspect, in an optional implementation manner, the log chain includes multiple serial chain nodes, and there is an association relationship between the multiple serial chain nodes.

Based on the second aspect, in an optional implementation manner, the storage address of the log chain is the storage address of the head node of the log chain.

Based on the second aspect, in an optional implementation manner, the first device obtains the log chain of the incremental transaction according to the transaction table, including:

The first device determines the storage address of the log chain of the incremental transaction according to the transaction table;

The first device acquires the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction.

In a third aspect, the embodiment of the present application provides a data processing device, which is characterized in that it includes:

The obtaining unit is used to obtain the transaction table, the transaction table is used to indicate the state of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, and each chain node indicates an element in the transaction The storage address of the group;

The obtaining unit is also used to obtain the log chain of the transaction to be recovered according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be recovered;

The acquisition unit is also used to determine the storage address of the tuple of the transaction to be recovered according to the log chain of the transaction to be recovered, so as to obtain the tuple of the transaction to be recovered;

The deletion unit is used to delete the log chain and tuple of the transaction to be recovered.

Based on the third aspect, in an optional implementation manner, the log chain includes multiple serial chain nodes, and there is an association relationship between the multiple serial chain nodes.

Based on the third aspect, in an optional implementation manner, the storage address of the log chain is the storage address of the head node of the log chain.

Based on the third aspect, in an optional implementation manner, the acquiring unit is specifically configured to:

According to the transaction table, it is determined that the transaction in the undo state is the transaction to be recovered;

Determine the storage address of the log chain of the transaction to be recovered according to the transaction table;

Obtain the log chain of the transaction to be restored according to the storage address of the log chain of the transaction to be restored.

Based on the third aspect, in an optional implementation manner, the data processing device is applied to the active and standby database systems, the data processing device is the main database, the second device is the standby database, and the transaction table is also used to indicate the commit time of the transaction, and the data The processing device also includes a determination unit and a synchronization unit,

The determination unit is used to determine the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest backup time of the primary and standby database systems;

The obtaining unit is also used to obtain the log chain of the incremental transaction according to the transaction table;

The obtaining unit is also used to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

The synchronization unit is used to synchronize the log chain and tuple of the incremental transaction to the second device.

Based on the third aspect, in an optional implementation manner, the acquiring unit is specifically configured to:

Determine the storage address of the log chain of the incremental transaction according to the transaction table;

According to the storage address of the log chain of the incremental transaction, the log chain of the incremental transaction is obtained.

In a fourth aspect, the embodiment of the present application provides a data processing device. The data processing device is applied to the primary and backup database systems. The data processing device is the primary database, and the second device is the standby database. The data processing device includes:

The obtaining unit is used to obtain the transaction table, the transaction table is used to indicate the commit time of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, the log chain includes at least one chain node, and each chain node indicates one of the transactions The storage address of the tuple;

The determination unit is used to determine the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest data synchronization time of the main and standby database systems;

The obtaining unit is also used to obtain the log chain of the incremental transaction according to the transaction table;

The determination unit is also used to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

The synchronization unit is used to synchronize the log chain and tuple of the incremental transaction to the second device.

Based on the fourth aspect, in an optional implementation manner, the log chain includes multiple serial chain nodes, and there is an association relationship between the multiple serial chain nodes.

Based on the fourth aspect, in an optional implementation manner, the storage address of the log chain is the storage address of the head node of the log chain.

Based on the fourth aspect, in an optional implementation manner, the acquiring unit is specifically configured to:

Determine the storage address of the log chain of the incremental transaction according to the transaction table;

According to the storage address of the log chain of the incremental transaction, the log chain of the incremental transaction is obtained.

In a fifth aspect, an embodiment of the present invention provides a computer device, including a memory, a communication interface, and a processor coupled to the memory and the communication interface; the memory is used to store instructions, and the processor is used to execute the instructions , the communication interface is used to communicate with other devices under the control of the processor; wherein, when the processor executes the instructions, it executes the data processing method described in any one of the above aspects.

In the sixth aspect, the embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when it is run on a computer, the computer executes the data program described in any one of the above-mentioned aspects. The method of processing.

In the seventh aspect, the embodiment of the present application provides a computer program product or computer program, the computer program product or computer program includes computer instructions, and when it is run on a computer, it causes the computer to perform the data processing described in any of the above aspects Methods.

It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

The application discloses a data processing method and related devices. The first device acquires a transaction table, which is used to indicate the state of the transaction and the storage address of the log chain. Each transaction corresponds to a log chain, and the log chain includes at least one chain node, each chain node indicates the storage address of a tuple in the transaction; the first device obtains the log chain of the transaction to be restored according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be restored; the first device obtains the transaction according to The log chain of the transaction to be restored determines the storage address of the tuple of the transaction to be restored, so as to obtain the tuple of the transaction to be restored; the first device deletes the log chain and the tuple of the transaction to be restored. In this application, the transaction to be recovered can be queried through the transaction table, and the tuple of the transaction to be recovered can be obtained according to the log chain of the transaction to be recovered, so that there is no need to perform a full table scan on the first device, which improves data recovery, The efficiency of data processing. On the other hand, since the log chain in this application does not carry the tuple information of the transaction itself, it can avoid the overhead of additional write operations in the process of using the log chain, thus compared with the log file carrying the transaction The write-ahead logging (WAL) mechanism of tuples also has great performance advantages.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present application, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.

FIG. 1 is a schematic diagram of a pre-write log and a post-write log in a database system;

Figure 2 is a schematic diagram of the Zen-based system architecture;

FIG. 3 is a schematic diagram of the logging mechanism provided by the present application;

Fig. 4 is a schematic flow chart of the data processing method in the present application;

Fig. 5 is a schematic diagram of the architecture of the active and standby database systems in the present application;

Figure 6 is a schematic structural diagram of the log chain in the present application;

FIG. 7 is a schematic diagram of a scene of data recovery in an embodiment of the present application;

FIG. 8 is a schematic diagram of the data synchronization process in this application;

FIG. 9 is a schematic diagram of a scenario of data synchronization of the active and standby database systems in the embodiment of the present application;

FIG. 10 is a schematic diagram of another scenario of data synchronization of the active and standby database systems in the embodiment of the present application;

FIG. 11 is a schematic diagram of the application of the log chain mechanism in the data recovery scenario and the data synchronization scenario in the embodiment of the present application;

FIG. 12 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

FIG. 13 is a schematic structural diagram of another data processing device provided in the embodiment of the present application;

FIG. 14 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

Detailed ways

The embodiments of the present application provide a data processing method and a related device, which are used to improve the efficiency of data processing.

Embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain specific examples of the present invention, and are not intended to limit the present invention. Those of ordinary skill in the art know that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

In this application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there can be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of practice in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The security of the database involves various aspects, and the loss or tampering of data in the database will bring immeasurable losses, so the security of the database is particularly important. At present, the data in the database is mainly analyzed through the log (logging) of the database, and then corresponding measures are taken according to the analysis results, such as data recovery or data synchronization.

Next, several common logging mechanisms are introduced.

A: Write ahead logging (WAL) is a common method in database systems to ensure the atomicity and persistence of data operations. In computer science, WAL is a family of techniques used in relational database systems to provide atomicity and durability. In a database system using WAL, all modifications must be written to the log (logging) before committing.

Log files usually include redo information and undo information. The purpose of this can be illustrated by an example. Assuming that a program is in the process of performing some operations, the machine is powered off, then when restarting, the program may need to know whether the operation performed at that time was successful, partially successful or failed. If the WAL is used, the program can examine the log file and compare what was planned to be done with what was actually done when there was a sudden power loss. On the basis of this comparison, the program can decide whether to undo the operation that has been done, continue to complete the operation that has been done, or keep it as it is.

However, a problem with WAL technology is that the log file already contains log information and data information, and the same data will be saved in the database file and log file respectively. The data is written twice, and there is a problem of write amplification. Especially in the scenario of the active and standby database systems, when the active database device is synchronizing data, the WAL log will cause a huge overhead for write operations.

B: Write behind logging (WBL) is a new logging and recovery protocol designed for non-volatile memory (non-volatile memory, NVM) databases. The advantages of NVM are byte addressability, high performance close to memory, little gap between sequential access and random access, and, when the current is turned off, the data stored in NVM will not disappear.

Please refer to FIG. 1, which is a schematic diagram of a write-ahead log and a write-behind log in a database system. As shown in Figure 1, in the WAL mechanism, the modification of each transaction to the database will be written sequentially in the WAL log. Before the data of the transaction is persisted, the corresponding log must be persisted first, and the new value and the old value are recorded at the same time. On the critical path of submission, only the log needs to be persisted in the form of sequential writing; the WBL mechanism is different from the WAL mechanism, and its key idea is to record which parts of the database data have changed, rather than how the data has changed. Using WBL requires persisting the transaction before persisting the log, which is used to track the data modified by the transaction for undoing uncommitted transactions.

C: Zen system, which is a high-throughput log-free online transaction processing (on line transaction processing, OLTP) engine for NVM. Zen's no-log policy is an implementation of WBL. In the Zen mechanism, log information is written into each tuple of the transaction. Please refer to FIG. 2 , which is a schematic diagram of the Zen-based system architecture. As shown in Figure 2, when the transaction is committed, the tuple in memory is written into NVM, and the (dirty-bit, LP) value of the last tuple written by the transaction is updated, which means that the transaction data has been completely persistent change. When restoring data in the database, the entire table will be scanned to check all tuples. If the LP bits of all tuples corresponding to a certain transaction are not set to 1, it means that the transaction has not been successfully persisted. Zen uses the ts-commit variable to save the maximum time stamp encountered by the current scan during the scanning of each area. Every time it encounters a tuple with an LP value of 1, the time stamp will be updated. If a tuple If the timestamp ≤ ts-commit, Zen considers that the associated transaction has been committed. If the timestamp of a tuple > ts-commit, it will be placed in the queue waiting for the second scan. After the entire area is scanned, the maximum ts-commit will be obtained, and then it will be processed in the queue waiting for the second scan.

To sum up, the above three log mechanisms all have certain defects. Specifically, for the WAL mechanism, the log file already contains log information and data information, and the same data will be saved in the database file and log file respectively, so additional duplication overhead will be incurred when performing write operations; for The WBL mechanism, after performing data recovery, must determine whether the currently accessed data is dirty data each time it is accessed, which also brings a certain amount of additional overhead; Zen needs to perform data recovery on all tables in the database. The data is scanned twice to identify uncommitted transactions, so that the data can be rolled back. Since the database stores a lot of data content, scanning the data of the entire table twice requires a large system overhead, and the data recovery time is too long.

In view of this, in the embodiment of the present application, a log mechanism is provided, which is applied to the data processing method and related equipment in the present application, so as to improve the efficiency of data recovery. Please refer to FIG. 3 , which is a schematic diagram of the log mechanism provided by this application. As shown in Figure 3, in the Zen mechanism, the log information is written into each tuple of the transaction; while in the log mechanism provided by this application, the log information is expressed in the form of a "log chain", and each transaction is Corresponds to a log chain, and the transaction tuple and the log chain are not fused together, in other words, the transaction tuple and the log chain are stored in different storage addresses. Specifically, each log chain includes at least one chain node, and each tuple of a transaction will have its corresponding chain node. That is, in the log chain of the transaction, each chain node indicates the storage address of its corresponding tuple. Exemplarily, in Figure 3, transaction 1 has 3 tuples (data 1, data 2 and data 3), and the log chain corresponding to transaction 1 includes 3 chain nodes (chain node 1, chain node 2 and chain node 3 ). Among them, link node 1 indicates that data 1 is stored in position 1 in table 1, link node 2 indicates that data 2 is stored in position 2 in table 1, and link node 3 indicates that data 3 is stored in position 1 in table 2.

It should be noted that in this application, the storage location of the log chain is not limited. On the one hand, an independent table can be configured to centrally store all the log chains in the database, so as to facilitate the centralized management of the log chains of each transaction; On the one hand, the log chain can also be stored in the table where each tuple is located, which is not limited here.

The data processing method of the present application can be applied to the data recovery process of the database, and can also be applied to the data synchronization process in the active and standby database systems. Next, the data recovery process in this application is firstly introduced. Please refer to FIG. 4 , which is a schematic flowchart of the data processing method in this application. As shown in Figure 4, the data processing method in the embodiment of the present application includes:

101. The first device acquires a transaction table.

Since NVM has the advantages of 256B read and write granularity, asymmetric read and write performance, and a specific optimal number of concurrency, in order to better implement the data processing method of this application, preferably, the data processing of this application can be The method is applied to the NVM database.

It should be noted that the first device in this application can be an independent physical server database, or a server cluster database composed of multiple physical servers, a distributed system database, or a master database in a master-standby database system. It can also provide cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, content delivery network (content delivery network, CDN), big data or manual Cloud server databases for basic cloud computing services such as smart platforms, which are not limited here.

Taking the application of the data processing method of the present application to the active and standby database systems as an example, please refer to FIG. 5 , which is a schematic diagram of the architecture of the active and standby database systems in this application. As shown in FIG. 5 , the master database (equivalent to an independent physical server database) includes a free space management module, a transaction table, a log chain, and table data. Specifically, in the log mechanism provided by this application, the log information is expressed in the form of "log chain", and each transaction corresponds to a log chain, and the tuple of the transaction and the log chain are not fused together. In other words, transaction tuples and log chains are stored in different storage addresses. Specifically, each log chain includes at least one chain node, and each tuple of a transaction will have its corresponding chain node. That is, in the log chain of the transaction, each chain node indicates the storage address of its corresponding tuple. When there are multiple tuples in a transaction, that is, the log chain corresponding to the transaction will also have the same number of chain nodes as the number of tuples in the transaction, and each chain node in the same log chain will be associated in a serial manner , so when a chain node in a certain log chain is obtained, other chain nodes associated with it belonging to the same log chain can also be traced back, which improves the management efficiency of the log chain.

In the first device, when a transaction is created, a tuple is created for recording basic information (data) of the transaction. When a transaction is committed, it is necessary to first write to each chain node of the log chain corresponding to the transaction, and to write each tuple. When all the chain nodes of the log chain of the transaction and all the tuples of the transaction have been persisted, the entire transaction submission process is completed.

Since there are a large number of transactions stored in the database (the first device), in order to centrally manage and query the log chains of these transactions, this application configures a transaction table for these log chains, and the transaction table stores the first device The status of all transactions in and the storage addresses of the log chains corresponding to these transactions. By obtaining the state of the transaction in the transaction table, the first device can know whether the transaction has been committed, or whether it needs to be rolled back, or whether it needs to perform data synchronization; and by obtaining the storage address of the log chain of the transaction, the first device can then You can know where these log chains are stored. Further, the transaction table may also indicate the commit time of each transaction and the like. Since the log chain includes several chain nodes, in this application, only the storage address where the head node of each log chain is located can be used as the storage address of the log chain. Each chain node in the same log chain is serially associated, and by obtaining the storage address of the chain head node, all associated chain nodes in the entire log chain can be obtained. Therefore, there is no need to record the storage address of each chain node in the log chain in the transaction table, which saves storage resources and improves the management efficiency of the log chain.

It can be seen from the above that the number of transactions stored in the database (the first device) is huge, and each transaction corresponds to a log chain. Therefore, in this application, the log chains of these transactions can be further connected in series. The head node of the log chain is connected to the tail of the previous log chain, so that the arrangement of the log chains stored in the first device is clear and easy to manage. For ease of understanding, please refer to FIG. 6, which is a schematic structural diagram of the log chain in this application. As shown in Figure 6, two transactions are indicated in the transaction table of the first device, namely 0x001 transaction and 0x002 transaction, and both of them have been committed (in committed state), and the storage address of the chain head node of 0x001 transaction is 0x001, 0x002 The storage address of the chain head node of the transaction is 0x002. Through the storage address of the chain head node indicated by the transaction table, the chain head node of the log chain of the 0x001 transaction and the chain head node of the log chain of the 0x002 transaction can be obtained. Further, other chain nodes serially associated with the chain head node can be obtained through the chain head node, so as to obtain the complete log chain, and then use each chain node in the log chain to clarify the storage where each tuple of the transaction is located Location. As shown in Figure 6, the log chain of the 0x001 transaction indicates that the tuple of the transaction is stored in the position 1a, the position 1b in the table (table) A, the position 2a in the table B, and the position 3a in the table C, and the 0x002 transaction The log chain for indicates that the tuple for this transaction is stored at location 1c in table A and at location 2b in table B.

The free space management module needs to manage these log chains. When the transaction is rolled back, the corresponding log chain will be deleted. By moving the head node of the log chain to the end of the free chain where the log chain is located, the free space is completed. Recycling of space. Table data is used to store transaction tuples.

102. The first device acquires the log chain of the transaction to be restored according to the transaction table.

When the first device goes down, the tuples of the transactions that are being persisted often cannot be fully persisted. In order to ensure the atomicity and durability of transactions, it is necessary to roll back such interrupted transactions that fail to persist all tuples. Such transactions belong to undo transactions. Since states of all transactions are recorded in the transaction table, after obtaining the transaction table, the first device can filter out transactions in the undo state in the transaction table as transactions to be restored. Since the transaction table also records the storage addresses of the log chains of each transaction, the first device can find the storage addresses of the log chains corresponding to the The storage address of the log chain to obtain the log chain of the transaction to be restored.

103. The first device determines the storage address of the tuple of the transaction to be restored according to the log chain of the transaction to be restored, so as to obtain the tuple of the transaction to be restored.

In this application, since each chain node in the log chain indicates the storage address of a tuple of the transaction. Therefore, after obtaining the log chain of the transaction to be restored, the first device obtains all tuples of the transaction to be restored according to the indications of all chain nodes in the log chain of the transaction to be restored.

104. The first device deletes the log chain and tuple of the transaction to be recovered.

The first device deletes the log chain and tuple of the transaction to be recovered, and completes the data recovery process for this time.

For ease of understanding, please refer to FIG. 7 , which is a schematic diagram of a data recovery scenario in an embodiment of the present application. As shown in Figure 7, in the data recovery scenario after a downtime, the transaction table indicates that the status of the transaction numbered 0x004 is "active", which means that the transaction is in the undo state. However, according to the indication of the log chain of transaction 0x004, it can be determined that the tuple at position 1d in Table 1 failed to be persisted. Therefore, the log chain and all tuples of the transaction need to be deleted to complete the transaction rollback.

In this application, the transaction to be recovered can be queried through the transaction table, and the tuple of the transaction to be recovered can be obtained according to the log chain of the transaction to be recovered, so that there is no need to perform a full table scan on the first device, which improves data recovery, The efficiency of data processing. On the other hand, since the log chain in this application does not carry the tuple information of the transaction itself, it can avoid the overhead of additional write operations in the process of using the log chain, thus compared with the log file carrying the transaction The tuple WAL mechanism also has great performance advantages.

Next, the data synchronization process in this application is introduced. In the active-standby database system, the primary database device is responsible for read and write operations, while the standby database device is only responsible for read operations. Therefore, after the primary database device completes the write operation of the transaction, the data needs to be synchronized to the standby database device. The log chain mechanism in this application can also be applied to the data synchronization process of the primary and secondary database systems. The following takes the first device as the primary database device and the second device as the standby database device as an example for description. Please refer to FIG. 8 . FIG. 8 is a schematic diagram of a data synchronization process in the data processing method of the present application. As shown in Figure 8, the data synchronization method in the embodiment of the present application includes:

201. The first device acquires a transaction table.

Step 201 is similar to the aforementioned step 101 shown in FIG. 4 , and details are not repeated here. It should be noted that, in order to facilitate subsequent determination of incremental transactions, in the process of data synchronization, the commit time of each transaction should be recorded in the transaction table.

202. The first device determines incremental transactions according to the transaction table.

In practical applications, the primary and standby database systems generally need to perform data synchronization on a regular basis, that is, periodically synchronize the tuples of new transactions in the first device to the standby database system. Transactions are incremental transactions in this application, and these transactions have not yet performed data synchronization. Since the commit time of each transaction is recorded in the transaction table, the first device can determine, according to the transaction table, a transaction whose commit time is after the latest data synchronization time of the primary and standby database systems as an incremental transaction.

203. The first device acquires the log chain of the incremental transaction according to the transaction table.

Step 203 is similar to the aforementioned step 102 shown in FIG. 4 , and details are not repeated here.

204. The first device determines the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction.

Step 204 is similar to the aforementioned step 103 shown in FIG. 4 , and details are not repeated here.

205. The first device synchronizes the log chain and tuple of the incremental transaction to the second device.

Please refer to FIG. 9 . FIG. 9 is a schematic diagram of a scenario of data synchronization of the active and standby database systems in the embodiment of the present application. As shown in Figure 9, the first device sends the log chain and tuple of the incremental transaction to the second device, and the second device persists the log chain and tuple of the incremental transaction locally to complete the data synchronization process.

Further, please refer to FIG. 10 . FIG. 10 is a schematic diagram of another scenario of data synchronization of the primary and secondary database systems in the embodiment of the present application. As shown in Figure 10, in this application, the number of database devices participating in data synchronization is not limited. The primary database device synchronizes the tuples and log chains of incremental transactions to multiple standby database devices, that is, the second device can refer to multiple A standby database device.

In the WAL mechanism, the log file includes the tuple of the transaction, so during the data synchronization process, the persistence of the log file and the tuple of the transaction is equivalent to writing the tuple of the transaction twice, and the overhead of data synchronization is extremely high. In the WBL mechanism, WAL is regenerated by scanning the entire table for synchronization. Since the position where the transaction writes data is random, to organize the WAL log, the entire table must be scanned, which is very expensive and poor in performance; in the Zen mechanism, the log is additionally organized in memory, which hardly affects performance, but the database After a downtime, the memory logs will be lost, and it will be very costly for the master and standby machines to resynchronize. In this application, the incremental transaction can be queried through the transaction table, and the tuple of the incremental transaction can be obtained according to the log chain of the incremental transaction, so that there is no need to perform a full table scan on the first device, which improves data synchronization, The efficiency of data processing. On the other hand, since the log chain in this application does not carry the tuple information of the transaction itself, it can avoid additional write operation overhead in the process of persisting the log chain in the standby database, thus compared with the log The WAL mechanism in which files carry transaction tuples also has great performance advantages.

In practical applications, the above data recovery method and data synchronization method can be independent of each other, or can be used in conjunction with each other. That is, the database device can only use steps 101 to 104 shown in FIG. 4 for data recovery, or it can only use steps 201 to 205 shown in FIG. 8 for data synchronization, or it can also be integrated in the database system The above data recovery method and data synchronization method. For ease of understanding, please refer to FIG. 11 , which is a schematic diagram of the application of the log chain mechanism in the data recovery scenario and the data synchronization scenario in the embodiment of the present application. As shown in Figure 11, in the data recovery scenario and the data synchronization scenario, the same transaction table and log chain can be shared, that is, the transaction table and transaction log chain can be used for data recovery and data synchronization.

In order to better implement the above solutions of the embodiments of the present application, related equipment for implementing the above solutions is also provided below. Specifically, please refer to FIG. 12. FIG. 12 is a schematic structural diagram of a data processing device provided by an embodiment of the present application. The data processing device includes:

The obtaining unit 301 is used to obtain a transaction table, the transaction table is used to indicate the status of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, the log chain includes at least one chain node, and each chain node indicates one of the transactions The storage address of the tuple;

The obtaining unit 301 is also used to obtain the log chain of the transaction to be restored according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be restored;

The obtaining unit 301 is further configured to determine the storage address of the tuple of the transaction to be restored according to the log chain of the transaction to be restored, so as to obtain the tuple of the transaction to be restored;

The deletion unit 302 is configured to delete the log chain and tuple of the transaction to be recovered.

In a possible design, the log chain includes multiple serial chain nodes, and there is an association relationship among the multiple serial chain nodes.

In a possible design, the storage address of the log chain is the storage address of the head node of the log chain.

In a possible design, the obtaining unit 301 is specifically used to:

According to the transaction table, it is determined that the transaction in the undo state is the transaction to be recovered;

Determine the storage address of the log chain of the transaction to be recovered according to the transaction table;

Obtain the log chain of the transaction to be restored according to the storage address of the log chain of the transaction to be restored.

In a possible design, the data processing device is applied to the active and standby database systems, the data processing device is the main database, the second device is the standby database, the transaction table is also used to indicate the commit time of the transaction, and the data processing device also includes a determination unit 303 and synchronization unit 304,

The determination unit 303 is configured to determine the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest backup time of the active and standby database systems;

The obtaining unit 301 is also used to obtain the log chain of the incremental transaction according to the transaction table;

The obtaining unit 301 is further configured to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

A synchronization unit 304, configured to synchronize the log chain and tuple of the incremental transaction to the second device.

In a possible design, the obtaining unit 301 is specifically used to:

Determine the storage address of the log chain of the incremental transaction according to the transaction table;

According to the storage address of the log chain of the incremental transaction, the log chain of the incremental transaction is obtained.

Please refer to Figure 13. Figure 13 is a schematic structural diagram of another data processing device provided in the embodiment of the present application. The data processing device is applied to the active and standby database systems. Devices include:

The obtaining unit 401 is used to obtain a transaction table, the transaction table is used to indicate the submission time of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, and each chain node indicates a transaction in the transaction The storage address of a tuple;

The determination unit 402 is configured to determine the incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the time of the latest data synchronization of the active and standby database systems;

The obtaining unit 401 is also used to obtain the log chain of the incremental transaction according to the transaction table;

The determining unit 402 is further configured to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction;

A synchronization unit 403, configured to synchronize the log chain and tuple of the incremental transaction to the second device.

Based on the fourth aspect, in an optional implementation manner, the log chain includes multiple serial chain nodes, and there is an association relationship between the multiple serial chain nodes.

Based on the fourth aspect, in an optional implementation manner, the storage address of the log chain is the storage address of the head node of the log chain.

Based on the fourth aspect, in an optional implementation manner, the acquiring unit 401 is specifically configured to:

Determine the storage address of the log chain of the incremental transaction according to the transaction table;

According to the storage address of the log chain of the incremental transaction, the log chain of the incremental transaction is obtained.

The embodiment of the present application also provides a computer device. Please refer to FIG. 14. FIG. 14 is a schematic structural diagram of the computer device provided in the embodiment of the present application. The computer device described in the corresponding embodiment in FIG. 12 or FIG. 13 can be deployed. The data processing device is used to realize the method of the embodiment corresponding to FIG. 4 or FIG. 8. Specifically, the computer equipment is implemented by one or more servers, and the computer equipment may have relatively large differences due to different configurations or performances, and may include a or more than one central processing unit (central processing units, CPU) 522 (for example, one or more processors) and memory 532, one or more storage media 530 for storing application programs 542 or data 544 (for example, one or more mass storage device). Wherein, the memory 532 and the storage medium 530 may be temporary storage or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the computer device. Furthermore, the central processing unit 522 may be configured to communicate with the storage medium 530 , and execute a series of instruction operations in the storage medium 530 on the computer device 500 .

The computer device can also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, and/or, one or more operating systems 541, such as Windows Server ^™ , Mac OS X ^™ , Unix ^™ , Linux ^™ , FreeBSD ^™ , etc.

The embodiment of the present application also provides a computer program product, which, when running on a computer, causes the computer to execute the method described in the embodiment shown in FIG. 4 or FIG. 8 .

An embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a program for signal processing, and when it is run on a computer, the computer executes the program shown in Figure 4 or Figure 8 above. The method described in the example is shown.

The image processing device provided in the embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc. . The processing unit may execute the computer-executable instructions stored in the storage unit, so that the chip executes the method described in the embodiment shown in FIG. 4 or FIG. 8 above. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit may also be a storage unit located outside the chip in the wireless access device, such as only Read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (random access memory, RAM), etc.

It should be noted that the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be It is not a physical unit, that is, it can be located in one place, or it can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course it can also be realized by special hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, Special components, etc. to achieve. In general, all functions completed by computer programs can be easily realized by corresponding hardware, and the specific hardware structure used to realize the same function can also be varied, such as analog circuits, digital circuits or special-purpose circuit etc. However, for this application, software program implementation is a better implementation mode in most cases. Based on this understanding, the essence of the technical solution of this application or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a floppy disk of a computer , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the instructions described in various embodiments of the present application method.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from a website, computer, training device, or data The center transmits to another website site, computer, training device or data center via wired (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer, or a data storage device such as a training device or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, a solid state disk (Solid State Disk, SSD)) and the like.

Claims (23)

1. A method for data processing, comprising: The first device obtains a transaction table, the transaction table is used to indicate the state of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, and each of the chain nodes indicates a transaction The storage address of a tuple in ; The first device obtains the log chain of the transaction to be restored according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be restored; The first device determines the storage address of the tuple of the transaction to be recovered according to the log chain of the transaction to be recovered, so as to obtain the tuple of the transaction to be recovered; The first device deletes the log chain and tuple of the transaction to be recovered. 2 . The method according to claim 1 , wherein the log chain includes a plurality of serial chain nodes, and there is an association relationship among the plurality of serial chain nodes. 3 . 3. The method according to claim 2, wherein the storage address of the log chain is the storage address of the head node of the log chain. 4. The method according to claim 1, 2 or 3, wherein the first device obtains the log chain of the transaction to be restored according to the transaction table, comprising: The first device determines, according to the transaction table, that the transaction in the undo state is a transaction to be recovered; The first device determines the storage address of the log chain of the transaction to be restored according to the transaction table; The first device acquires the log chain of the transaction to be restored according to the storage address of the log chain of the transaction to be restored. 5. The method according to any one of claims 1 to 4, wherein the method is applied to a master database system, the first device is the master database, the second device is the standby database, and the transaction The table is also used to indicate a commit time of the transaction, and the method further includes: The first device determines an incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest backup time of the active and standby database systems; The first device obtains the log chain of the incremental transaction according to the transaction table; The first device determines the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction; The first device synchronizes the log chain and tuple of the incremental transaction to the second device. 6. The method according to claim 5, wherein the first device obtains the log chain of the incremental transaction according to the transaction table, comprising: The first device determines the storage address of the log chain of the incremental transaction according to the transaction table; The first device acquires the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction. 7. A method for data processing, characterized in that, the method is applied to a master database system, the first device is the master database, and the second device is the standby database, the method comprising: The first device obtains a transaction table, the transaction table is used to indicate the commit time of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, each of the chain The node indicates the storage address of a tuple in the transaction; The first device determines an incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest data synchronization time of the primary and standby database systems; The first device obtains the log chain of the incremental transaction according to the transaction table; The first device determines the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction; The first device synchronizes the log chain and tuple of the incremental transaction to the second device. 8 . The method according to claim 7 , wherein the log chain includes a plurality of serial chain nodes, and there is an association relationship among the plurality of serial chain nodes. 9. The method according to claim 8, wherein the storage address of the log chain is the storage address of the head node of the log chain. 10. The method according to claim 7, 8 or 9, wherein the first device obtains the log chain of the incremental transaction according to the transaction table, comprising: The first device determines the storage address of the log chain of the incremental transaction according to the transaction table; The first device acquires the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction. 11. A data processing device, characterized in that it comprises: The obtaining unit is used to obtain a transaction table, the transaction table is used to indicate the state of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, each of the chain nodes Indicates the storage address of a tuple in the transaction; The acquiring unit is further configured to acquire a log chain of a transaction to be recovered according to the transaction table, wherein the transaction in the undo state in the transaction table is the transaction to be recovered; The acquiring unit is further configured to determine the storage address of the tuple of the transaction to be recovered according to the log chain of the transaction to be recovered, so as to acquire the tuple of the transaction to be recovered; The deletion unit is used to delete the log chain and tuple of the transaction to be recovered. 12 . The data processing device according to claim 11 , wherein the log chain includes a plurality of serial chain nodes, and there is an association relationship among the plurality of serial chain nodes. 13 . 13. The data processing device according to claim 12, wherein the storage address of the log chain is the storage address of the head node of the log chain. 14. The data processing device according to claim 11, 12 or 13, wherein the acquisition unit is specifically used for: According to the transaction table, it is determined that the transaction in the undo state is a transaction to be recovered; determining the storage address of the log chain of the transaction to be recovered according to the transaction table; Acquire the log chain of the transaction to be restored according to the storage address of the log chain of the transaction to be restored. 15. The data processing device according to any one of claims 11 to 14, wherein the data processing device is applied to an active and standby database system, the data processing device is a primary database, and the second device is a standby database , the transaction table is also used to indicate the submission time of the transaction, and the data processing device also includes a determination unit and a synchronization unit, The determining unit is configured to determine an incremental transaction according to the transaction table, and the commit time of the incremental transaction is after the latest backup time of the primary and standby database systems; The acquiring unit is further configured to acquire the log chain of the incremental transaction according to the transaction table; The acquiring unit is further configured to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to acquire the tuple of the incremental transaction; The synchronization unit is configured to synchronize the log chain and tuple of the incremental transaction to the second device. 16. The data processing device according to claim 15, wherein the acquiring unit is specifically used for: determining the storage address of the log chain of the incremental transaction according to the transaction table; Acquire the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction. 17. A data processing device, characterized in that, the data processing device is applied to a primary and backup database system, the data processing device is a primary database, and the second device is a standby database, and the data processing device includes: The obtaining unit is used to obtain a transaction table, the transaction table is used to indicate the submission time of the transaction and the storage address of the log chain, each transaction corresponds to a log chain, and the log chain includes at least one chain node, each of the chains The node indicates the storage address of a tuple in the transaction; A determining unit, configured to determine an incremental transaction according to the transaction table, the commit time of the incremental transaction is after the latest data synchronization time of the primary and standby database systems; The acquiring unit is further configured to acquire the log chain of the incremental transaction according to the transaction table; The determining unit is further configured to determine the storage address of the tuple of the incremental transaction according to the log chain of the incremental transaction, so as to obtain the tuple of the incremental transaction; A synchronization unit, configured to synchronize the log chain and tuple of the incremental transaction to the second device. 18 . The data processing device according to claim 17 , wherein the log chain includes multiple serial chain nodes, and there is an association relationship between the multiple serial chain nodes. 19 . 19. The data processing device according to claim 18, wherein the storage address of the log chain is the storage address of the head node of the log chain. 20. The data processing device according to claim 17, 18 or 19, wherein the acquisition unit is specifically used for: determining the storage address of the log chain of the incremental transaction according to the transaction table; Acquire the log chain of the incremental transaction according to the storage address of the log chain of the incremental transaction. 21. A computer device, comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory, so that the computer device executes the method according to any one of claims 1 to 6, or makes the computer device execute the method according to claims 7 to 10 any one of the methods described. 22. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented Or, when the computer program is executed by the processor, the method according to any one of claims 7 to 10 is implemented. 23. A computer program product, characterized in that, computer-readable instructions are stored in the computer program product, and when the computer-readable instructions are executed by a processor, the computer-readable instructions according to any one of claims 1 to 6 are realized. or, when the computer-readable instructions are executed by a processor, implement the method according to any one of claims 7 to 10.