CN117807161A - Compensation transaction realization method for distributed database - Google Patents
Compensation transaction realization method for distributed database Download PDFInfo
- Publication number
- CN117807161A CN117807161A CN202311703142.XA CN202311703142A CN117807161A CN 117807161 A CN117807161 A CN 117807161A CN 202311703142 A CN202311703142 A CN 202311703142A CN 117807161 A CN117807161 A CN 117807161A
- Authority
- CN
- China
- Prior art keywords
- transaction
- compensation
- node
- data
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
技术领域Technical field
本发明属于分布式数据库技术领域,具体的说是一种面向分布式数据库的补偿事务实现方法。The invention belongs to the technical field of distributed databases, and in particular is a compensation transaction implementation method for distributed databases.
背景技术Background technique
分布式系统已成为现代软件架构的中心部分,可以提供高可用性、可扩展性和故障恢复能力,在这些系统中,需要处理跨多个服务的业务操作,这就导致了分布式事务的需求。分布式事务是指事务的参与者、支持事务的服务器,目前分布式事务主要有以下几种模式:AT模式、Saga模式、TCC模式、XA模式和补偿事务模式。AT模式是一种无侵入的分布式事务解决方案,零学习成本,但SQL都由框架托管统一执行,会存在脏写问题;Saga模式是一种长事务解决方案,第一阶段就操作DB,会存在脏读问题;TCC模式适用于核心系统等对性能有很高要求的场景,事务执行太久会锁行,避免了脏读;XA模式是强一致性的解决方案,但性能低而使用较少;而补偿事务的核心思想是通过执行一系列的反向操作来撤销已完成的业务操作,这些反向操作被称为“补偿操作”,它们可以确保在发生错误或异常情况时,系统能够恢复到一致的状态,具有解耦、容错性和灵活性等优点。Distributed systems have become a central part of modern software architecture to provide high availability, scalability and failure recovery. In these systems, business operations need to be processed across multiple services, which has led to the need for distributed transactions. Distributed transactions refer to transaction participants and servers that support transactions. Currently, distributed transactions mainly have the following modes: AT mode, Saga mode, TCC mode, XA mode and compensation transaction mode. AT mode is a non-intrusive distributed transaction solution with zero learning cost, but SQL is managed and executed uniformly by the framework, and there will be dirty write problems; Saga mode is a long transaction solution, and the DB is operated in the first stage. There will be problems with dirty reads; TCC mode is suitable for scenarios with high performance requirements such as core systems. If a transaction is executed for too long, rows will be locked, avoiding dirty reads; XA mode is a strong consistency solution, but its performance is low and it is used Less; the core idea of compensating transactions is to undo completed business operations by performing a series of reverse operations. These reverse operations are called "compensating operations". They can ensure that when an error or abnormal situation occurs, the system Being able to restore to a consistent state has the advantages of decoupling, fault tolerance, and flexibility.
如申请公开号为CN112182103A的中国专利公开了一种分布式数据库及其实现跨节点事务强一致性的方法,其特征是,所述分布式数据库采用存储和计算分离的构架,包括:协调节点、数据节点以及编目节点三种类型节点,每个类型的节点分别可以水平扩展,协调节点负责分发请求到需要参与的数据节点,数据节点负责数据的访问与存储,编目节点存放系统元数据以及分区相关信息,一个事务经由所述协调节点发起,下发到一个或多个数据节点组上进行操作,分布式数据库引入作为参与者的数据节点之间的事务协商与事务补偿机制,在系统局部出现故障导致参与者无法正常收到作为协调者的协调节点的事务控制消息时,通过参与者之间相互状态确认,从而判断事务是应该提交还是回滚,以避免数据的不一致问题。For example, the Chinese patent application with publication number CN112182103A discloses a distributed database and a method for achieving strong consistency in cross-node transactions. It is characterized in that the distributed database adopts a storage and computing separation architecture, including: coordination nodes, There are three types of nodes: data nodes and catalog nodes. Each type of node can be expanded horizontally. The coordination node is responsible for distributing requests to the data nodes that need to participate. The data nodes are responsible for data access and storage. The catalog nodes store system metadata and partition related Information, a transaction is initiated through the coordination node and sent to one or more data node groups for operation. The distributed database introduces transaction negotiation and transaction compensation mechanisms between data nodes as participants, and a local failure occurs in the system. When participants cannot normally receive transaction control messages from the coordination node as the coordinator, the participants confirm each other's status to determine whether the transaction should be submitted or rolled back to avoid data inconsistencies.
如授权公告号为CN110019469B的中国专利公开了分布式数据库数据处理方法、装置、存储介质及电子装置,本发明包括:数据库导入导出管理节点在接收到用于导出分布式数据库中的数据的导出请求后,获取当前时刻活跃事务列表以及导出表分布信息;数据库导入导出管理节点根据导出表分布信息通知对应数据库执行数据导出操作;数据库导入导出管理节点在确定对应数据库执行完所述数据导出操作之后,将活跃事务列表发送给对应数据库的数据库代理节点;数据库导入导出管理节点在接收到数据库代理节点返回的数据一致性反向补偿语句之后,将对应数据库导出的数据导入到预定数据库中,并指示预定数据库执行数据一致性反向补偿语句。For example, the Chinese patent with the authorization announcement number CN110019469B discloses a distributed database data processing method, device, storage medium and electronic device. The present invention includes: the database import and export management node receives an export request for exporting data in the distributed database. Afterwards, obtain the current active transaction list and export table distribution information; the database import and export management node notifies the corresponding database to perform the data export operation according to the export table distribution information; the database import and export management node determines that the corresponding database has completed the data export operation, Send the active transaction list to the database agent node of the corresponding database; after receiving the data consistency reverse compensation statement returned by the database agent node, the database import and export management node imports the data exported by the corresponding database into the scheduled database and instructs the reservation The database executes data consistency reverse compensation statements.
以上现有技术均存在以下问题:1)实现复杂,对数据库性能影响较大,不适合高并发高性能场景;2)节点重启后补偿任务丢失。The above existing technologies all have the following problems: 1) The implementation is complex, has a great impact on database performance, and is not suitable for high-concurrency and high-performance scenarios; 2) Compensation tasks are lost after node restarts.
发明内容Contents of the invention
针对现有技术的不足,本发明提出了一种面向分布式数据库的补偿事务实现方法,具体包括:客户端发起事务请求,执行目录判断事务SQL是否超过限定值,在数据节点接收到事务请求后,判断是事务提交或事务回滚,并将执行结果返回给协调节点进行确认和验证,操作完成,协调节点提交该事务并进行定时清理,将结果返回给客户端,保证了分布式环境下数据的一致性,降低事务处理过程的复杂度。In view of the shortcomings of the existing technology, the present invention proposes a compensation transaction implementation method for distributed databases, which specifically includes: the client initiates a transaction request, executes the directory to determine whether the transaction SQL exceeds the limit value, and after the data node receives the transaction request , determine whether the transaction is committed or rolled back, and return the execution result to the coordination node for confirmation and verification. When the operation is completed, the coordination node submits the transaction and performs scheduled cleaning, and returns the result to the client, ensuring data in a distributed environment. consistency and reduce the complexity of transaction processing.
为实现上述目的,本发明提供如下技术方案:In order to achieve the above objects, the present invention provides the following technical solutions:
一种面向分布式数据库的补偿事务实现方法,包括:A compensation transaction implementation method for distributed databases, including:
步骤S1:客户端发起事务请求,通过执行目录判断事务SQL是否超过限定值,若超过则后端数据库返回错误,同时事务信息存储在事务SQL映射模块;Step S1: The client initiates a transaction request and determines whether the transaction SQL exceeds the limit value through the execution directory. If it exceeds the limit, the back-end database returns an error, and the transaction information is stored in the transaction SQL mapping module;
步骤S2:协调节点为步骤S1中客户端发起的事务分配一个时间戳,同时将该事务的操作发送到数据节点;Step S2: The coordination node assigns a timestamp to the transaction initiated by the client in step S1, and at the same time sends the operation of the transaction to the data node;
步骤S3:数据节点接收到事务请求后,判断是事务提交或事务回滚,并将执行结果返回给协调节点进行确认和验证,若验证失败,则协调节点通知该数据节点并进行补偿操作;Step S3: After receiving the transaction request, the data node determines whether the transaction is committed or rolled back, and returns the execution result to the coordination node for confirmation and verification. If the verification fails, the coordination node notifies the data node and performs compensation operations;
步骤S4:操作完成,协调节点提交该事务,并进行定时清理,将执行结果返回给客户端。Step S4: The operation is completed, the coordination node submits the transaction, cleans it regularly, and returns the execution result to the client.
具体的,所述步骤S1的具体步骤包括:Specifically, the specific steps of step S1 include:
步骤S101:确定事务执行的目录路径并读取包含的SQL脚本文件;Step S101: Determine the directory path for transaction execution and read the included SQL script file;
步骤S102:分析SQL脚本文件内容以确定是否存在可能导致事务超过限定值的SQL语句或操作并输出分析结果,根据分析结果,判断事务的SQL语句是否超过预定的限定值;Step S102: Analyze the content of the SQL script file to determine whether there are SQL statements or operations that may cause the transaction to exceed the limit value and output the analysis results. Based on the analysis results, determine whether the SQL statement of the transaction exceeds the predetermined limit value;
步骤S103:启动事务并监控其执行情况,若事务执行过程中超过限定值,后端数据库响应并立即返回错误;Step S103: Start the transaction and monitor its execution. If the limit value is exceeded during transaction execution, the back-end database responds and returns an error immediately;
步骤S104:接收错误并进行处理,若事务执行成功,则继续处理其他事务或关闭连接。Step S104: Receive the error and process it. If the transaction is executed successfully, continue to process other transactions or close the connection.
具体的,所述步骤S2的具体步骤包括:Specifically, the specific steps of step S2 include:
步骤S201:客户端向协调节点发送一个事务请求,协调节点接收到事务请求后,对其进行解析和处理;Step S201: The client sends a transaction request to the coordination node. After receiving the transaction request, the coordination node parses and processes it;
步骤S202:协调节点根据补偿规则生成一个全局唯一的时间戳,并将生成的时间戳分配给步骤S1中客户端发起的事务;Step S202: The coordination node generates a globally unique timestamp according to the compensation rules, and assigns the generated timestamp to the transaction initiated by the client in step S1;
步骤S203:协调节点将生成的时间戳与事务请求一起转发给相关的数据节点。Step S203: The coordination node forwards the generated timestamp together with the transaction request to the relevant data node.
具体的,所述步骤S202中的时间戳具体包括:事务开始时间、事务提交时间和事务回滚时间。Specifically, the timestamp in step S202 specifically includes: transaction start time, transaction commit time and transaction rollback time.
具体的,所述步骤S3的具体步骤包括:Specifically, the specific steps of step S3 include:
步骤S301:数据节点接收到协调节点转发的事务请求,根据时间戳进行排序和执行,判断是进行读取、写入或其他操作;Step S301: The data node receives the transaction request forwarded by the coordinating node, sorts and executes it according to the timestamp, and determines whether to perform read, write or other operations;
步骤S302:根据判断结果,数据节点对本地数据进行读取或修改操作,并判断事务的状态;Step S302: According to the judgment result, the data node reads or modifies local data and judges the status of the transaction;
步骤S303:数据节点判断事务可以提交,则将操作结果持久化到数据库中,并返回确认消息给协调节点,需要回滚事务,则撤销之前执行的相关操作,并将数据库恢复到事务开始前的状态,然后返回确认消息给协调节点;Step S303: If the data node determines that the transaction can be submitted, it will persist the operation results in the database and return a confirmation message to the coordination node. If the transaction needs to be rolled back, the relevant operations performed previously will be revoked and the database will be restored to the state before the transaction started. status, and then returns a confirmation message to the coordinating node;
步骤S304:协调节点接收到数据节点的确认消息后,利用三阶段提交策略对操作结果进行验证,验证通过,则说明该数据节点的操作成功完成,验证失败,则需要进行回滚。Step S304: After receiving the confirmation message from the data node, the coordination node uses the three-phase submission strategy to verify the operation result. If the verification passes, it means that the operation of the data node is successfully completed. If the verification fails, a rollback is required.
具体的,所述步骤S304中三阶段提交策略具体为:在准备阶段和提交阶段之间增加了一个预提交阶段,使协调节点向所有的参与节点发送预提交请求,根据参与节点的响应判断提交或回滚操作。Specifically, the three-phase submission strategy in step S304 is as follows: a pre-submission phase is added between the preparation phase and the submission phase, so that the coordination node sends a pre-submission request to all participating nodes, and the submission is judged based on the responses of the participating nodes. or rollback the operation.
具体的,所述步骤S304中补偿操作的具体步骤包括:Specifically, the specific steps of the compensation operation in step S304 include:
步骤S3041:确定需要补偿的事务,定义补偿规则;Step S3041: Determine the transactions requiring compensation and define compensation rules;
步骤S3042:根据补偿规则,通过事务补偿映射将分布式事务中的操作映射到补偿操作;Step S3042: According to the compensation rules, map operations in the distributed transaction to compensation operations through transaction compensation mapping;
步骤S3043:通过补偿执行标志映射记录每个操作的补偿执行状态;Step S3043: Record the compensation execution status of each operation through the compensation execution flag mapping;
步骤S3044:补偿操作执行成功则提交数据库,并通过提交成功映射记录每个操作的提交状态。Step S3044: If the compensation operation is successfully executed, the database is submitted, and the submission status of each operation is recorded through the submission success mapping.
具体的,所述步骤S3041中补偿规则包括:反向操作、数据修复、事务回滚、状态恢复、顺序执行、可重复性和一致性保证。Specifically, the compensation rules in step S3041 include: reverse operation, data repair, transaction rollback, state recovery, sequential execution, repeatability and consistency guarantee.
具体的,所述步骤S4的具体步骤包括:Specifically, the specific steps of step S4 include:
步骤S401:协调节点收到所有参与节点的响应后,确认所有操作都已完成;Step S401: After receiving responses from all participating nodes, the coordination node confirms that all operations have been completed;
步骤S402:协调节点将事务提交到数据库或分布式事务管理器,并用提交成功映射记录提交状态;Step S402: The coordination node submits the transaction to the database or distributed transaction manager, and uses the submission success mapping to record the submission status;
步骤S403:在确认事务提交成功后,协调节点定时清理历史数据、释放资源操作;Step S403: After confirming that the transaction is successfully submitted, the coordination node regularly cleans up historical data and releases resource operations;
步骤S404:将事务处理的结果返回给客户端。Step S404: Return the transaction processing result to the client.
一种面向分布式数据库的补偿事务实现系统,包括:事务SQL映射模块、事务提交成功映射模块、事务回滚映射模块、补偿Tmp映射模块、补偿映射模块、补偿执行标志映射模块,A compensation transaction implementation system for distributed databases, including: transaction SQL mapping module, transaction submission success mapping module, transaction rollback mapping module, compensation Tmp mapping module, compensation mapping module, compensation execution flag mapping module,
所述事务SQL映射模块,用于判断事务SQL是否超过上限或提交限定值;The transaction SQL mapping module is used to determine whether the transaction SQL exceeds the upper limit or submits a limit value;
所述事务提交成功映射模块,用于记录分布式事务提交成功的事务映射,并在有定时任务时对数据进行清理;The transaction submission success mapping module is used to record the transaction mapping of distributed transaction submission success, and clean the data when there are scheduled tasks;
所述事务回滚映射模块,用于记录回滚的分布式事务,并在有定时任务时对其中的数据进行清理;The transaction rollback mapping module is used to record rolled-back distributed transactions and clean the data therein when there are scheduled tasks;
所述补偿Tmp映射模块模块,用于记录要进行补偿的临时节点事务;The compensation Tmp mapping module module is used to record temporary node transactions to be compensated;
所述补偿映射模块,用于记录要进行补偿的节点事务;The compensation mapping module is used to record node transactions to be compensated;
所述补偿执行标志映射模块,用于标记某个节点正在补偿事务。The compensation execution flag mapping module is used to mark that a certain node is compensating for a transaction.
与现有技术相比,本发明的有益效果是:Compared with the prior art, the beneficial effects of the present invention are:
1.本发明提出一种面向分布式数据库的补偿事务实现系统,并进行了架构、运行步骤和流程上的优化改进,系统具备流程简单,投资运行费用低廉,生产工作成本低的优点,在保证功能完善、请求方式不变、性能损耗细微的基础上,实现了不影响数据的补偿事务方案。1. The present invention proposes a compensation transaction implementation system for distributed databases, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple processes, low investment and operation costs, and low production work costs, while ensuring On the basis of complete functions, unchanged request method, and subtle performance loss, a compensation transaction plan that does not affect data is implemented.
2.本发明提出一种面向分布式数据库的补偿事务实现方法,在数据方面,通过本地文件进行各个节点事务状态、SQL情况的记录,既能保证SQL补偿效率,又能实现节点重启补偿事务不丢失;在功能方面,当进入补偿事务之后,仅对目录级别进行加锁,最小化对业务的影响,定时线程会自动进行补偿任务,当补偿成功之后会释放锁,这个过程中业务是无感知的。2. The present invention proposes a compensation transaction implementation method for distributed databases. In terms of data, the transaction status and SQL situation of each node are recorded through local files, which can not only ensure the SQL compensation efficiency, but also realize the node restart compensation transaction without interruption. Lost; in terms of function, when entering the compensation transaction, only the directory level is locked to minimize the impact on the business. The scheduled thread will automatically perform the compensation task. When the compensation is successful, the lock will be released. The business is unaware of this process. of.
3.本发明提出一种面向分布式数据库的补偿事务实现方法,使用内存和本地文件结合的方式对需要进行补偿的事务进行记录,既能保证效率又能保证重启后补偿任务不丢失,在保证功能完善、请求方式不变、性能损耗细微的基础上,实现了不影响数据的补偿事务方案。3. The present invention proposes a compensation transaction implementation method for distributed databases, which uses a combination of memory and local files to record transactions that need to be compensated, which can not only ensure efficiency but also ensure that the compensation task is not lost after restarting. On the basis of complete functions, unchanged request method, and subtle performance loss, a compensation transaction plan that does not affect data is implemented.
附图说明Description of drawings
图1为本发明一种面向分布式数据库的补偿事务实现方法流程图;Figure 1 is a flow chart of a compensation transaction implementation method for distributed databases according to the present invention;
图2为本发明一种面向分布式数据库的补偿事务实现方法系统分析流程图;Figure 2 is a system analysis flow chart of a compensation transaction implementation method for distributed databases according to the present invention;
图3为本发明一种面向分布式数据库的补偿事务实现方法三阶段提交协调者状态转换流程图;Figure 3 is a three-stage submission coordinator state transition flow chart of a compensation transaction implementation method for distributed databases of the present invention;
图4为本发明一种面向分布式数据库的补偿事务实现方法三阶段提交参与者状态转换流程图;Figure 4 is a three-stage submission participant state transition flow chart of a compensation transaction implementation method for distributed databases of the present invention;
图5为本发明一种面向分布式数据库的补偿事务实现系统架构图。FIG5 is a diagram showing the architecture of a system for implementing compensation transactions for distributed databases according to the present invention.
具体实施方式Detailed ways
为使本发明实现的技术手段、创作特征、达成目的与功效易于明白了解,在本发明的描述中,需要说明的是,术语“中心”、“上”、“下”、“左”、“右”、“竖直”、“水平”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系,仅是为了便于描述本发明和简化描述,而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作,因此不能理解为对本发明的限制。此外,术语“一号”、“二号”、“三号”仅用于描述目的,而不能理解为指示或暗示相对重要性。下面结合具体实施方式,进一步阐述本发明。In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, in the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "left" The orientations or positional relationships indicated by "right", "vertical", "horizontal", "inner", "outer", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description. It is not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore is not to be construed as a limitation of the invention. Furthermore, the terms "No. 1," "No. 2," and "No. 3" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The present invention will be further described below in conjunction with specific embodiments.
实施例1Example 1
请参阅图1-图4,本发明提供的一种实施例:一种面向分布式数据库的补偿事务实现方法,包括以下步骤:Referring to Figures 1-4, an embodiment provided by the present invention: a compensation transaction implementation method for distributed databases, including the following steps:
步骤S1:客户端发起事务请求,通过执行目录判断事务SQL是否超过限定值,若超过则后端数据库返回错误,同时事务信息存储在事务SQL映射模块;Step S1: The client initiates a transaction request and determines whether the transaction SQL exceeds the limit value through the execution directory. If it exceeds the limit, the back-end database returns an error, and the transaction information is stored in the transaction SQL mapping module;
事务的四大特性包括:原子性、一致性、隔离性和持久性。The four major characteristics of transactions include: atomicity, consistency, isolation and durability.
事务的SQL语句预定的限定值包括:(1)执行时间:设定事务的最大执行时间,如果超过该时间,则认为事务未能在限定时间内完成,可能会返回错误或回滚事务;(2)资源使用情况:可以设定事务在执行过程中使用的资源的最大限制;(3)数据量大小;(4)并发访问数量。The predetermined limit values of the SQL statement of the transaction include: (1) Execution time: Set the maximum execution time of the transaction. If this time is exceeded, the transaction will be considered to have failed to complete within the limited time, and an error may be returned or the transaction may be rolled back; ( 2) Resource usage: You can set the maximum limit on the resources used by the transaction during execution; (3) The amount of data; (4) The number of concurrent accesses.
对事务SQL设置限定值的优点:(1)可以确保事务在执行过程中不会受其它事务干扰,保持数据的一致性;(2)当事务执行过程中出现错误时,可以回滚事务以撤销已经执行的操作,避免数据的不一致性;(3)通过设置事务的锁定级别和并发控制方式,可以控制并发访问时的事务隔离级别和锁竞争情况,提高系统的性能和响应时间;(4)可以通过回滚和提交事务来恢复数据的一致性;(5)通过事务的设置可以控制程序的逻辑流程。The advantages of setting limit values for transaction SQL: (1) It can ensure that the transaction will not be interfered by other transactions during execution and maintain data consistency; (2) When an error occurs during transaction execution, the transaction can be rolled back to cancel it. Operations that have been performed to avoid data inconsistency; (3) By setting the lock level and concurrency control method of the transaction, you can control the transaction isolation level and lock competition during concurrent access, and improve the performance and response time of the system; (4) Data consistency can be restored by rolling back and committing transactions; (5) The logical flow of the program can be controlled through transaction settings.
步骤S2:协调节点为步骤S1中客户端发起的事务分配一个时间戳,同时将该事务的操作发送到数据节点;Step S2: The coordination node assigns a timestamp to the transaction initiated by the client in step S1, and at the same time sends the operation of the transaction to the data node;
在分布式数据库中,协调节点是一种特殊节点,用于协调分片的搜索和聚合操作,当客户端发送搜索请求时,协调节点会将请求转发给存储相关分片的数据节点,并将结果汇总返回给客户端,协调节点不存储数据,也不参与数据的搜索。In a distributed database, a coordinating node is a special node used to coordinate the search and aggregation operations of shards. When a client sends a search request, the coordinating node forwards the request to the data node storing the relevant shards and returns the results to the client. The coordinating node does not store data and does not participate in data searches.
步骤S3:数据节点接收到事务请求后,判断是事务提交或事务回滚,并将执行结果返回给协调节点进行确认和验证,若验证失败,则协调节点通知该数据节点并进行补偿操作;Step S3: After receiving the transaction request, the data node determines whether the transaction is committed or rolled back, and returns the execution result to the coordination node for confirmation and verification. If the verification fails, the coordination node notifies the data node and performs compensation operations;
事务回滚是指将该事务已经完成对数据库的更新操作撤销,在事务中,每个正确的原子都会被顺序执行,知道遇到错误的原子操作。Transaction rollback refers to undoing the update operation of the database that has been completed by the transaction. In the transaction, each correct atom will be executed sequentially until an incorrect atomic operation is encountered.
步骤S4:操作完成,协调节点提交该事务,并进行定时清理,将执行结果返回给客户端。Step S4: When the operation is completed, the coordination node submits the transaction, cleans it regularly, and returns the execution result to the client.
协调节点进行定时清理的步骤如下:The steps for coordinating nodes to perform scheduled cleanup are as follows:
(1)协调节点首先会接收到查询请求,并将请求广播到每一个数据节点,每个数据节点的分片会处理该查询请求,进行数据查询,并将符合条件的数据放在一个优先队列中;(1) The coordination node will first receive the query request and broadcast the request to each data node. The shards of each data node will process the query request, perform data query, and place qualified data in a priority queue. middle;
(2)每个分片进行数据查询后,将文档ID、节点信息、分片信息返回给协调节点,协调节点将所有的结果进行汇总,并进行全局排序;(2) After each shard performs a data query, it returns the document ID, node information, and shard information to the coordination node, which aggregates all the results and performs global sorting.
(3)协调节点向包含这些文档ID的分片发送get请求,对应的分片将文档数据返回给协调节点;(3) The coordination node sends a get request to the shards containing these document IDs, and the corresponding shards return the document data to the coordination node;
(4)最后协调节点将数据返回给客户端。(4) Finally, the coordination node returns the data to the client.
步骤S1的具体步骤包括:The specific steps of step S1 include:
步骤S101:确定事务执行的目录路径并读取包含的SQL脚本文件;Step S101: Determine the directory path for transaction execution and read the included SQL script file;
步骤S102:分析SQL脚本文件内容以确定是否存在可能导致事务超过限定值的SQL语句或操作并输出分析结果,根据分析结果,判断事务的SQL语句是否超过预定的限定值;Step S102: analyzing the content of the SQL script file to determine whether there is an SQL statement or operation that may cause the transaction to exceed the limit value and outputting the analysis result, and judging whether the SQL statement of the transaction exceeds the predetermined limit value according to the analysis result;
步骤S103:启动事务并监控其执行情况,若事务执行过程中超过限定值,后端数据库响应并立即返回错误;Step S103: Start the transaction and monitor its execution. If the limit value is exceeded during transaction execution, the back-end database responds and returns an error immediately;
步骤S104:接收错误并进行处理,若事务执行成功,则继续处理其他事务或关闭连接。Step S104: Receive the error and process it. If the transaction is executed successfully, continue to process other transactions or close the connection.
步骤S2的具体步骤包括:The specific steps of step S2 include:
步骤S201:客户端向协调节点发送一个事务请求,协调节点接收到事务请求后,对其进行解析和处理;Step S201: The client sends a transaction request to the coordination node. After receiving the transaction request, the coordination node parses and processes it;
具体步骤包括:(1)协调节点首先需要解析事务请求,从中提取出需要执行的操作和相关参数;(2)在解析事务请求后,协调节点需要验证操作的合法性;(3)一旦操作被验证为合法,协调节点就需要将操作分配给适当的事务参与节点;(4)在将操作分配给参与者后,协调节点需要监控事务的执行过程。Specific steps include: (1) The coordination node first needs to parse the transaction request and extract the operations and related parameters that need to be performed; (2) After parsing the transaction request, the coordination node needs to verify the legality of the operation; (3) Once the operation is If verified to be legal, the coordination node needs to assign the operation to the appropriate transaction participating node; (4) After assigning the operation to the participant, the coordination node needs to monitor the execution process of the transaction.
步骤S202:协调节点根据补偿规则生成一个全局唯一的时间戳,并将生成的时间戳分配给步骤S1中客户端发起的事务;Step S202: The coordination node generates a globally unique timestamp according to the compensation rules, and assigns the generated timestamp to the transaction initiated by the client in step S1;
步骤S203:协调节点将生成的时间戳与事务请求一起转发给相关的数据节点。Step S203: The coordination node forwards the generated timestamp together with the transaction request to the relevant data node.
步骤S202中的时间戳具体内容包括:事务开始时间、事务提交时间和事务回滚时间。The specific content of the timestamp in step S202 includes: transaction start time, transaction commit time and transaction rollback time.
在分布式数据库中,时间戳的生成和分配需要保证全局唯一性和一致性,以避免出现冲突和数据不一致的问题,常见的时间戳生成方式包括:使用物理时钟、分布式时钟服务器或通过算法生成伪时钟。In a distributed database, the generation and distribution of timestamps need to ensure global uniqueness and consistency to avoid conflicts and data inconsistencies. Common timestamp generation methods include: using physical clocks, distributed clock servers or through algorithms Generate pseudo clock.
步骤S3的具体步骤包括:The specific steps of step S3 include:
步骤S301:数据节点接收到协调节点转发的事务请求,根据时间戳进行排序和执行,判断是进行读取、写入或其他操作;Step S301: The data node receives the transaction request forwarded by the coordination node, sorts and executes it according to the timestamp, and determines whether to perform reading, writing or other operations;
执行时间戳排序时,需要将待排序的数据与相应的时间戳一起传递给排序函数,排序函数会根据时间戳的升序或降序排列数据When performing timestamp sorting, the data to be sorted needs to be passed to the sorting function together with the corresponding timestamp. The sorting function will arrange the data in ascending or descending order according to the timestamp.
步骤S302:根据判断结果,数据节点对本地数据进行读取或修改操作,并判断事务的状态;Step S302: According to the judgment result, the data node reads or modifies local data and judges the status of the transaction;
步骤S303:数据节点判断事务可以提交,则将操作结果持久化到数据库中,并返回确认消息给协调节点,需要回滚事务,则撤销之前执行的相关操作,并将数据库恢复到事务开始前的状态,然后返回确认消息给协调节点;Step S303: If the data node determines that the transaction can be committed, it will persist the operation result in the database and return a confirmation message to the coordination node. If the transaction needs to be rolled back, the previously executed related operations will be canceled and the database will be restored to the state before the transaction started, and then a confirmation message will be returned to the coordination node;
其中,事务提交和回滚事务存储在对应的事务提交成功映射和事务回滚映射模块。事务提交成功映射指的是将事务成功提交的结果映射到某个对象或系统,具体步骤包括:确定映射对象、定义映射规则、执行映射操作、验证映射结果和记录映射日志;事务回滚映射是数据库管理系统中的一种机制,它用于在事务执行过程中发生错误或异常情况时,将数据库状态恢复到事务开始之前的状态,具体步骤包括:错误检测、准备回滚、生成回滚日志、执行回滚和提交事务。Among them, transaction commit and rollback transactions are stored in the corresponding transaction commit success mapping and transaction rollback mapping modules. Successful transaction submission mapping refers to mapping the result of successful transaction submission to an object or system. The specific steps include: determining the mapping object, defining mapping rules, performing mapping operations, verifying mapping results, and recording mapping logs; transaction rollback mapping is A mechanism in the database management system that is used to restore the database state to the state before the transaction started when an error or abnormality occurs during transaction execution. The specific steps include: error detection, preparing for rollback, and generating rollback logs. , perform rollback and commit transactions.
步骤S304:协调节点接收到数据节点的确认消息后,利用三阶段提交策略对操作结果进行验证,验证通过,则说明该数据节点的操作成功完成,验证失败,则需要进行回滚。Step S304: After receiving the confirmation message from the data node, the coordination node uses the three-phase submission strategy to verify the operation result. If the verification passes, it means that the operation of the data node is successfully completed. If the verification fails, a rollback is required.
步骤S304中三阶段提交策略具体为:在准备阶段和提交阶段之间增加了一个预提交阶段,使协调节点向所有的参与节点发送预提交请求,根据参与节点的响应判断提交或回滚操作。The three-phase submission strategy in step S304 is specifically: a pre-commit phase is added between the preparation phase and the submission phase, so that the coordination node sends a pre-commit request to all participating nodes, and determines the submission or rollback operation based on the responses of the participating nodes.
在分布式数据库中,常见的策略包括:两阶段提交和三阶段提交,本发明选择三阶段提交策略,这样可以减少因等待所有参与节点响应而导致的阻塞问题。三阶段提交内容包括:In distributed databases, common strategies include: two-phase submission and three-phase submission. The present invention selects a three-phase submission strategy, which can reduce the blocking problem caused by waiting for responses from all participating nodes. The three-stage submission content includes:
(1)准备阶段:协调节点向所有的参与节点发送准备就绪请求,等待参与节点的响应;(1) Preparation phase: The coordinating node sends a readiness request to all participating nodes and waits for responses from the participating nodes;
(2)预提交阶段:在等待所有参与节点响应的过程中,协调节点根据参与节点的响应情况来决定是否进行预提交操作。如果所有的参与节点都返回了“同意”响应,则协调节点进入提交阶段;否则,协调节点进入回滚阶段;(2) Pre-commit phase: While waiting for responses from all participating nodes, the coordinating node decides whether to perform a pre-commit operation based on the response of the participating nodes. If all participating nodes return an "agree" response, the coordinating node enters the commit phase; otherwise, the coordinating node enters the rollback phase;
(3)提交阶段:在预提交阶段后,如果所有的参与节点都返回了“同意”响应,则协调节点向所有的参与节点发送提交请求;参与节点接收到提交请求后,执行相关的操作并将结果持久化到数据库中;协调节点等待所有的参与节点完成操作并返回确认消息;当所有的参与节点都完成了操作并经过协调节点的确认和验证后,协调节点提交该事务,并向客户端返回成功结果。(3) Submission phase: After the pre-submission phase, if all participating nodes return an "agree" response, the coordinating node sends a submission request to all participating nodes; after receiving the submission request, the participating nodes perform relevant operations and Persist the results into the database; the coordinating node waits for all participating nodes to complete the operation and return a confirmation message; when all participating nodes have completed the operation and have been confirmed and verified by the coordinating node, the coordinating node submits the transaction and reports it to the client The end returns a successful result.
步骤S304中补偿操作的具体步骤包括:The specific steps of the compensation operation in step S304 include:
步骤S3041:确定需要补偿的事务,定义补偿规则;Step S3041: determine the transaction that needs to be compensated and define the compensation rules;
事务补偿规则包括:1)确定补偿点:在事务执行过程中,确定需要补偿的步骤和状态,从而确定需要回滚的范围;2)确定回滚范围;3)根据回滚范围和补偿点,生成相应的回滚命令;4)根据生成的回滚命令,执行相应的回滚操作;5)将补偿操作的信息记录到补偿日志;6)在执行完补偿操作后,检查补偿结果是否符合预期;7)将补偿操作的结果通知给相关的用户或系统。Transaction compensation rules include: 1) Determine the compensation point: During the transaction execution process, determine the steps and status that require compensation, thereby determining the scope of rollback; 2) Determine the rollback scope; 3) Based on the rollback scope and compensation point, Generate the corresponding rollback command; 4) Perform the corresponding rollback operation according to the generated rollback command; 5) Record the compensation operation information to the compensation log; 6) After executing the compensation operation, check whether the compensation result is as expected ;7) Notify the relevant users or systems of the results of the compensation operation.
步骤S3042:根据补偿规则,通过事务补偿映射将分布式事务中的操作映射到补偿操作;Step S3042: According to the compensation rule, the operations in the distributed transaction are mapped to compensation operations through transaction compensation mapping;
补偿映射是指当事务执行失败或出现异常情况时,通过执行补偿操作来还原数据到事务开始前的状态,以保证数据的一致性和完整性。Compensation mapping refers to performing compensation operations to restore data to the state before the start of the transaction to ensure data consistency and integrity when transaction execution fails or an abnormality occurs.
步骤S3043:通过补偿执行标志映射记录每个操作的补偿执行状态;Step S3043: Record the compensation execution status of each operation through the compensation execution flag mapping;
补偿执行标志映射的优点在于它能够精确地记录每个操作的结果和状态,从而在事务失败时能够准确地执行撤销或恢复操作,保证数据的一致性,同时,通过将操作结果和执行标志之间建立映射关系,所以本发明采用补偿执行标志映射记录补偿执行状态。The advantage of compensated execution flag mapping is that it can accurately record the results and status of each operation, so that when a transaction fails, the undo or recovery operation can be accurately performed to ensure data consistency. At the same time, by combining the operation results and execution flags, A mapping relationship is established between them, so the present invention uses compensation execution flag mapping to record the compensation execution status.
补偿执行标志映射的实现方式包括以下步骤:The implementation of compensation execution flag mapping includes the following steps:
(1)为每个操作生成一个唯一的执行标志;(1) Generate a unique execution flag for each operation;
(2)将执行标志与操作结果之间建立映射关系;(2) Establishing a mapping relationship between the execution flag and the operation result;
(3)在事务执行过程中,记录执行标志和操作结果;(3) During transaction execution, record execution flags and operation results;
(4)如果某个操作失败,根据执行标志找到对应的映射关系,然后根据映射关系对已经完成的操作进行撤销或者恢复;(4) If an operation fails, find the corresponding mapping relationship based on the execution flag, and then undo or restore the completed operation based on the mapping relationship;
(5)所有的操作都成功完成,或者已经执行补偿操作,结束事务并通知客户端。(5) All operations are completed successfully, or compensation operations have been performed, the transaction ends and the client is notified.
步骤S3044:补偿操作执行成功则提交数据库,并通过提交成功映射记录每个操作的提交状态。Step S3044: If the compensation operation is successfully executed, the database is submitted, and the submission status of each operation is recorded through the submission success mapping.
步骤S3041中补偿规则包括:反向操作、数据修复、事务回滚、状态恢复、顺序执行、可重复性和一致性保证。The compensation rules in step S3041 include: reverse operation, data repair, transaction rollback, state recovery, sequential execution, repeatability and consistency guarantee.
补偿操作的提交方式包括:直接提交、异步提交和定时提交。Submission methods for compensation operations include: direct submission, asynchronous submission and scheduled submission.
步骤S4的具体步骤包括:The specific steps of step S4 include:
步骤S401:协调节点收到所有参与节点的响应后,确认所有操作都已完成;Step S401: After receiving responses from all participating nodes, the coordination node confirms that all operations have been completed;
步骤S402:协调节点将事务提交到数据库或分布式事务管理器,并用提交成功映射记录提交状态;Step S402: the coordinating node submits the transaction to the database or distributed transaction manager, and records the submission status with a submission success mapping;
步骤S403:在确认事务提交成功后,协调节点定时清理历史数据、释放资源操作;Step S403: After confirming that the transaction submission is successful, the coordination node regularly cleans historical data and releases resource operations;
步骤S404:将事务处理的结果返回给客户端。Step S404: Return the transaction processing result to the client.
实施例2Example 2
请参阅图5,本发明提供的另一种实施例:一种面向分布式数据库的补偿事务实现系统,包括:Please refer to FIG5 , another embodiment provided by the present invention: a compensation transaction implementation system for a distributed database, comprising:
事务SQL映射模块、事务提交成功映射模块、事务回滚映射模块、补偿Tmp映射模块、补偿映射模块、补偿执行标志映射模块,Transaction SQL mapping module, transaction submission success mapping module, transaction rollback mapping module, compensation Tmp mapping module, compensation mapping module, compensation execution flag mapping module,
所述事务SQL映射模块,用于判断事务SQL是否超过上限或提交限定值;The transaction SQL mapping module is used to determine whether the transaction SQL exceeds the upper limit or submission limit value;
所述事务提交成功映射模块,用于记录分布式事务提交成功的事务映射,并在有定时任务时对数据进行清理;The transaction submission success mapping module is used to record the transaction mapping of distributed transaction submission success, and clean the data when there are scheduled tasks;
所述事务回滚映射模块,用于记录回滚的分布式事务,并在有定时任务时对其中的数据进行清理;The transaction rollback mapping module is used to record rolled-back distributed transactions and clean the data therein when there are scheduled tasks;
所述补偿Tmp映射模块模块,用于记录要进行补偿的临时节点事务;The compensation Tmp mapping module module is used to record temporary node transactions to be compensated;
所述补偿映射模块,用于记录要进行补偿的节点事务;The compensation mapping module is used to record node transactions to be compensated;
所述补偿执行标志映射模块,用于标记某个节点正在补偿事务。The compensation execution flag mapping module is used to mark that a certain node is compensating for a transaction.
以上结合附图对本发明的实施例进行了描述,但是本发明并不局限于上述的具体实施方式,上述的具体实施方式仅仅是示意性的,而不是限制性的,本领域的普通技术人员在本发明的启示下,在不脱离本发明宗旨和权利要求所保护的范围情况下,还可做出很多形式,这些均属于本发明的保护之内。The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above-mentioned specific implementations. The above-mentioned specific implementations are only illustrative and not restrictive. Those of ordinary skill in the art will Under the inspiration of the present invention, many forms can be made without departing from the spirit of the present invention and the scope protected by the claims, and these all fall within the protection of the present invention.
Claims (10)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311703142.XA CN117807161A (en) | 2023-12-12 | 2023-12-12 | Compensation transaction realization method for distributed database |
PCT/CN2024/137276 WO2025124287A1 (en) | 2023-12-12 | 2024-12-06 | Distributed database-oriented compensating transaction implementation method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311703142.XA CN117807161A (en) | 2023-12-12 | 2023-12-12 | Compensation transaction realization method for distributed database |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117807161A true CN117807161A (en) | 2024-04-02 |
Family
ID=90419069
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311703142.XA Pending CN117807161A (en) | 2023-12-12 | 2023-12-12 | Compensation transaction realization method for distributed database |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117807161A (en) |
WO (1) | WO2025124287A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2025124287A1 (en) * | 2023-12-12 | 2025-06-19 | 天翼云科技有限公司 | Distributed database-oriented compensating transaction implementation method |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6873995B2 (en) * | 2002-04-23 | 2005-03-29 | International Business Machines Corporation | Method, system, and program product for transaction management in a distributed content management application |
WO2013184712A2 (en) * | 2012-06-04 | 2013-12-12 | Google Inc. | Systems and methods of increasing database access concurrency using granular timestamps |
WO2021107988A1 (en) * | 2020-05-30 | 2021-06-03 | Xfuturewei Technologies, Inc. | Distributed processing of transactions in a network using timestamps |
CN112182103A (en) * | 2020-09-24 | 2021-01-05 | 广州巨杉软件开发有限公司 | Distributed database and method for realizing cross-node transaction strong consistency |
CN117807161A (en) * | 2023-12-12 | 2024-04-02 | 天翼云科技有限公司 | Compensation transaction realization method for distributed database |
-
2023
- 2023-12-12 CN CN202311703142.XA patent/CN117807161A/en active Pending
-
2024
- 2024-12-06 WO PCT/CN2024/137276 patent/WO2025124287A1/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2025124287A1 (en) * | 2023-12-12 | 2025-06-19 | 天翼云科技有限公司 | Distributed database-oriented compensating transaction implementation method |
Also Published As
Publication number | Publication date |
---|---|
WO2025124287A1 (en) | 2025-06-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111338766B (en) | Transaction processing method, apparatus, computer equipment and storage medium | |
CN111143389B (en) | Transaction execution method and device, computer equipment and storage medium | |
CN109977171B (en) | Distributed system and method for ensuring transaction consistency and linear consistency | |
CN111597015B (en) | Transaction processing method and device, computer equipment and storage medium | |
Cowling et al. | Granola:{Low-Overhead} distributed transaction coordination | |
Sciascia et al. | Scalable deferred update replication | |
CN115098229B (en) | Transaction processing method, device, node device and storage medium | |
CN113396407A (en) | System and method for augmenting database applications using blockchain techniques | |
EP4229523B1 (en) | Multi-statement interactive transactions with snapshot isolation in a scale-out database | |
WO2021036768A1 (en) | Data reading method, apparatus, computer device, and storage medium | |
JP7549137B2 (en) | Transaction processing method, system, device, equipment, and program | |
CN112148436B (en) | Decentralized TCC transaction management method, device, equipment and system | |
CN113391885A (en) | A distributed transaction processing system | |
US11003550B2 (en) | Methods and systems of operating a database management system DBMS in a strong consistency mode | |
US12111817B2 (en) | Log execution method and apparatus, computer device and storage medium | |
US20230315713A1 (en) | Operation request processing method, apparatus, device, readable storage medium, and system | |
CN111444027A (en) | Transaction processing method and device, computer equipment and storage medium | |
WO2025124287A1 (en) | Distributed database-oriented compensating transaction implementation method | |
CN119902932A (en) | A database synchronization method, device, equipment and storage medium | |
CN114722121B (en) | A database cluster master-slave method and device | |
CN112749156B (en) | Data processing method, database management system and data processing device | |
Singh et al. | TransEdge: Supporting efficient read queries across untrusted edge nodes | |
CN119513112B (en) | Transaction processing method of distributed transaction processing system and computer readable storage medium | |
WO2025010725A1 (en) | Hybrid database implementations | |
WO2025010735A1 (en) | Hybrid database implementations |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |