Disclosure of Invention
The embodiment of the application provides a data processing method which is used for retrying the execution failure of a storage process and can avoid the repeated execution of executed transactions in the storage process.
A first aspect of the embodiment of the application provides a data processing method, which comprises the steps of responding to a storage process request, executing a storage process, wherein the storage process comprises one or more ordered transactions, recording mark information of a first transaction in the one or more ordered transactions when the execution of the first transaction is finished, and retrying the transaction after the first transaction in the storage process according to the mark information of the first transaction based on the condition that the abnormality occurs in the execution of the storage process and the condition of retrying the storage process is met.
The storage process may contain one or more ordered transactions, and during execution of the storage process, the tag information of the executed transaction in the storage process, such as the location where the transaction ended, is recorded. Therefore, when the execution of the storage process fails to retry, only sentences after the executed transaction can be retried, and the executed transaction does not need to be re-executed, so that the execution efficiency of the retry is improved, and errors caused by repeatedly executing the submitted transaction during the retry are avoided.
In a possible implementation manner of the first aspect, the method further includes recording a request statement that invokes the stored procedure.
The data processing method provided by the embodiment of the application records the request statement for calling the storage process, and can be used for positioning the retry execution position through the request statement and the mark information of the first transaction.
In a possible implementation manner of the first aspect, the marking information includes a line number of an end statement of the first transaction.
The data processing method provided by the embodiment of the application provides a specific implementation mode for recording the marking information of the first transaction, and enhances the operability of the scheme.
In a possible implementation manner of the first aspect, the method further includes recording intermediate state information of the storage procedure, where the intermediate state information includes one or more of an execution statement result, error prompt information, input parameters, output parameters, global variables, and temporary variables of the first transaction.
According to the data processing method provided by the embodiment of the application, the database management system can also record the execution statement result or error prompt information, the input parameter, the output parameter, the global variable and the temporary variable of the first transaction, retry is performed according to the intermediate state information when the first transaction is finished, and the consistency of data can be met.
In a possible implementation manner of the first aspect, the intermediate state information includes an execution statement result or error prompt information of the first transaction, and the method further includes caching the execution statement result or error prompt information of the first transaction, and sending the execution statement result or error prompt information to the user equipment until the first transaction is over.
According to the data processing method provided by the embodiment of the application, the execution statement result or the error prompt information of the first transaction is recorded and then sent to the user equipment after the first transaction is ended, so that the situation that the same execution statement result or the error prompt information is repeatedly sent to the user equipment during retry can be avoided.
In a possible implementation manner of the first aspect, the intermediate state information includes an execution statement result or error prompt information of the first transaction, and the method further includes caching the execution statement result or error prompt information of the first transaction until the storing process is finished, and then sending the execution statement result or error prompt information to the user equipment.
The data processing method provided by the embodiment of the application provides another way for sending the execution statement result or the error prompt information to the user, namely, sending the user after the storage process is finished, so that the same execution statement result or error prompt information can be prevented from being repeatedly sent to the user equipment during retry.
In a possible implementation manner of the first aspect, the first transaction is located in a branch of a branch statement in the storage process, the method further includes recording intermediate state information of the storage process, the intermediate state information including a branch execution state, the branch execution state being used to identify a branch in which the first transaction is located, and retrying the transaction after the first transaction in the storage process includes executing the transaction after the first transaction in the branch in which the first transaction is located based on the branch execution state.
In the data processing method provided by the embodiment of the application, the intermediate state information may further include a branch execution state, where the branch execution state is used to identify a branch of the first transaction in a branch statement, and the branch statement includes an if branch statement or a case branch statement. And when retrying, the branch can be continuously executed according to the branch execution state, so that retry errors are avoided.
In a possible implementation manner of the first aspect, the branch execution state includes a conditional determination result of executing the branch statement in the storing process, where the conditional determination result is used to determine to execute the target branch in the branch statement.
According to the data processing method provided by the embodiment of the application, one specific record content of the branch execution state is the condition judgment result of the branch statement, so that the feasibility of the scheme is enhanced.
In one possible implementation manner of the first aspect, the method further includes the step of retrying the storage process, wherein the condition of the storage process is that the storage process exception is a restorability exception, and the restorability exception includes a network exception, a memory use exception, an input-output exception, a lock wait timeout exception, or a two-phase transaction commit exception.
According to the data processing method provided by the embodiment of the application, if the abnormality is restorable abnormality, retry can be executed.
In a possible implementation manner of the first aspect, the retrying the transaction after the first transaction in the storing process includes retrying the transaction after the first transaction in the storing process after waiting a preset first time period.
According to the data processing method provided by the embodiment of the application, the retry is performed after the scenes such as the main and standby switching wait for the preset duration, so that the success rate of the retry can be improved.
In one possible implementation manner of the first aspect, the method further includes recording tag information of a second transaction of the one or more ordered transactions when execution of the second transaction ends, the second transaction being a transaction executed after the first transaction, and retrying the transaction after the first transaction in the storage includes retrying the transaction after the second transaction in the storage based on the tag information of the second transaction, the second transaction being a last executed transaction in the executed transaction when a condition for retrying the storage is met.
In the data processing method provided by the embodiment of the application, if the storage process comprises a plurality of transactions, the marking information of each transaction can be recorded, and when retrying, the transaction after the last executed transaction is executed, so that the resource waste can be further reduced.
A second aspect of the embodiment of the application provides a database management system, which is characterized by comprising an execution unit, a recording unit and a retry unit, wherein the execution unit is used for responding to a storage process request and executing a storage process, the storage process comprises one or more ordered transactions, the recording unit is used for recording the marking information of a first transaction in the one or more ordered transactions when the execution of the first transaction is finished, and the retry unit is used for retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction based on the condition that the abnormality occurs in the execution of the storage process and the condition of retrying the storage process is met.
In a possible implementation manner of the second aspect, the recording unit is further configured to record a request statement that invokes the stored procedure.
In a possible implementation manner of the second aspect, the marking information includes a line number of an end statement of the first transaction.
In a possible implementation manner of the second aspect, the database management system further includes a recording unit, configured to record intermediate state information of the storage procedure, where the intermediate state information includes one or more of an execution statement result, error prompt information, an input parameter, an output parameter, a global variable, and a temporary variable of the first transaction.
In a possible implementation manner of the second aspect, the intermediate state information includes an execution statement result or error prompt information of the first transaction, and the database management system further includes a buffer unit, configured to buffer the execution statement result or error prompt information of the first transaction, and send the execution statement result or error prompt information to the user device until the first transaction ends.
In a possible implementation manner of the second aspect, the intermediate state information includes an execution statement result or an error prompt information of the first transaction, and the database management system further includes a buffer unit, configured to buffer the execution statement result or the error prompt information of the first transaction, and send the execution statement result or the error prompt information to the user device until the storage process is finished.
In a possible implementation manner of the second aspect, the first transaction is located in a branch of a branch statement in the storage process, and the recording unit is further configured to record intermediate state information of the storage process, where the intermediate state information includes a branch execution state, and the branch execution state is used to identify a branch in which the first transaction is located, and the retry unit is specifically configured to execute a transaction after the first transaction in the branch in which the first transaction is located based on the branch execution state.
In a possible implementation manner of the second aspect, the branch execution state includes a conditional determination result of executing the branch statement in the storing process, where the conditional determination result is used to determine to execute the target branch in the branch statement.
In one possible implementation manner of the second aspect, the retry of the storage procedure is conditional on the storage procedure being a restorable exception, the restorable exception including a network exception, a memory use exception, an input-output exception, a lock wait timeout exception, or a two-phase transaction commit exception.
In a possible implementation manner of the second aspect, the retry unit is specifically configured to retry a transaction after the first transaction in the storing process after waiting for a preset first duration.
In a possible implementation manner of the second aspect, the recording unit is further configured to record, when execution of a second transaction of the one or more ordered transactions ends, marking information of the second transaction, where the second transaction is a transaction executed after the first transaction, and the retry unit is specifically configured to retry a transaction after the second transaction in the storage process according to the marking information of the second transaction, where the second transaction is a transaction executed last in the executed transaction when a condition for retrying the storage process is satisfied.
A third aspect of the embodiments of the present application provides a data processing apparatus comprising a memory for storing instructions and a processor for executing the instructions in the memory, causing the data processing apparatus to perform the steps of the foregoing first aspect or implementations of the first aspect.
A fourth aspect of the embodiments of the present application provides a computer program product, characterized in that the computer program product comprises instructions which, when run on a computer, cause the computer to perform the steps of the foregoing first aspect or implementations of the first aspect.
A fifth aspect of the embodiments of the present application provides a computer readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the foregoing first aspect or implementations of the first aspect.
A sixth aspect of the embodiment of the present application provides a database system, which is characterized by comprising a network interface for receiving a storage procedure request sent by a user equipment, and a data processing device for executing the method according to the foregoing first aspect or any one of the implementation manners of the first aspect in response to the storage procedure request.
According to the technical scheme, the embodiment of the application has the advantages that the data processing method provided by the embodiment of the application realizes the execution failure retry of the storage process, supports the execution failure retry of the storage process containing the transaction, improves the execution efficiency of the retry because the finished transaction does not need to be re-executed, and avoids errors caused by repeatedly executing the executed transaction during the retry.
Detailed Description
The embodiment of the application provides a data processing method which is used for retrying the execution failure of a storage process and can avoid repeated execution of transactions in the storage process.
Embodiments of the present application will now be described with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the present application. As one of ordinary skill in the art can know, with the development of technology and the appearance of new scenes, the technical scheme provided by the embodiment of the application is also applicable to similar technical problems.
The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or modules is not necessarily limited to those steps or modules that are expressly listed or inherent to such process, method, article, or apparatus. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be executed according to the time/logic sequence indicated by the naming or numbering, and the execution sequence of the steps in the flow that are named or numbered may be changed according to the technical purpose to be achieved, so long as the same or similar technical effects can be achieved.
The data processing method, the database management system and the database server provided by the embodiment of the application can be applied to a stand-alone database system or a cluster database system, and are not limited in particular. A database system (database system) is a data processing system developed to accommodate the data processing requirements, and referring to fig. 1a-1c, a schematic diagram of the architecture of the database system according to an embodiment of the present application is shown below, and briefly described below:
FIG. 1a is a schematic diagram of a stand-alone database system according to an embodiment of the present application, where the stand-alone database system includes a database management system and a data store (data store), and the database management system is used to provide services such as query and modification of a database, and the database management system stores data in the data store. In a stand-alone database system, the database management system and data storage are typically located on a single server, such as a Symmetric Multiprocessor (SMP) server. The SMP server includes a plurality of processors, all of which share resources such as buses, memory, and i/o systems. The functions of the database management system may be implemented by one or more processors executing programs in memory.
Fig. 1b is a schematic diagram of an architecture of a clustered database system according to an embodiment of the present application, where the clustered database system is a clustered database system adopting a shared-storage architecture, and includes a plurality of nodes (e.g., nodes 1 to N in the figure), each node is configured with a database management system, which provides services such as querying and modifying a database for a user, respectively, and the plurality of database management systems store shared data in a shared data storage, and perform read-write operations on the data in the data storage through a switch. The shared data storage may be a shared disk array. Nodes in the clustered database system may be physical machines, such as database servers, or virtual machines running on abstract hardware resources. If the node is a physical machine, the switch is a storage area network (storage area network, SAN) switch, ethernet switch, fabric switch, or other physical switching device. If the node is a virtual machine, the switch is a virtual switch.
Fig. 1c is a schematic diagram of another architecture of a clustered database system according to an embodiment of the present application, where the clustered database system is a clustered database system adopting a shared-nothing architecture, and includes a plurality of nodes (e.g., nodes 1 to N in the figure), each node having a hardware resource (e.g., a data storage device), an operating system, and a database that are individually and independently shared, and the nodes communicate with each other via a network. Under the system, data are distributed to each node according to the database model and the application characteristics, the query task is divided into a plurality of parts, the query task is executed in parallel on all nodes, the query task is calculated in cooperation with each other, the database service is provided as a whole, and all communication functions are realized on a high-broadband network interconnection system. As with the shared-storage architecture clustered database system described in clustered database system 1, the nodes herein may be either physical machines or virtual machines.
The nodes in the cluster database system shown in fig. 1b and fig. 1c may be set as a coordinating node (coordinator) or a data node (data node) according to actual use requirements, where the coordinating node is configured to receive the SQL sent by the client or store a process request, and after analysis, may send the SQL to the data node that needs to participate in data processing. The data node finishes data extraction, the query result is returned to the coordination node, and the coordination node gathers the data and sends the data to the client.
In all embodiments of the application, the data store (data store) of the database system includes, but is not limited to, a Solid State Disk (SSD), a disk array, or other type of non-transitory computer readable medium. Although the database is not shown in fig. 1a-1c, it should be understood that the database is stored in a data store. Those skilled in the art will appreciate that a database system may include fewer or more components than those shown in fig. 1a-1c, or may include different components than those shown in fig. 1a-1c, with fig. 1a-1c only showing components more relevant to the implementations disclosed in this embodiment. For example, while 4 nodes have been described in the two clustered database systems of fig. 1b and 1c, one skilled in the art will appreciate that one clustered database system may contain any number of nodes. The database management system functions of each node may be implemented by an appropriate combination of software, hardware, and/or firmware running on each node, respectively.
For ease of understanding and description, by way of example and not limitation, the solution of an embodiment of the present application is described below with reference to a shared-nothing architecture, clustered database system, such as a database server. It will be apparent to those skilled in the art from this disclosure that the methods of embodiments of the present application are equally applicable to clustered database systems of shared-storage architecture, stand-alone database systems, and any type of relational database system.
Please refer to fig. 2, which is a schematic diagram illustrating a database server according to an embodiment of the present application.
As shown in FIG. 2, an embodiment of the present application provides a database server 100 comprising at least one processor 104, a non-transitory computer readable medium (non-transitory computer-readable medium) 106 storing executable code, and a database management system 108. The executable code, when executed by the at least one processor 104, is configured to implement the components and functions of the database management system 108. The non-transitory computer readable medium 106 may include one or more non-volatile memories including, by way of example, semiconductor memory devices such as erasable programmable read-only memory (erasable programmable read only memory, EPROM), electrically erasable read-only memory (ELECTRICALLY ERASABLE PROGRAMMABLE READ ONLY MEMORY, EEPROM) and flash memory (flash memory), magnetic disks such as internal hard disk (INTERNAL HARD DISK) or removable magnetic disk (removable disk), magneto-optical disk (magneto optical disk), and CD-ROMs and DVD-ROMs. In addition, the non-transitory computer-readable medium 106 may also include any device configured as a main memory (main memory). The at least one processor 104 may comprise any type of general purpose computing circuit or special purpose logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The at least one processor 104 may also be one or more processors, such as a CPU, coupled to one or more semiconductor substrates.
The database management system 108 may be a relational database management system (relational database MANAGEMENT SYSTEM, RDBMS). The database management system 108 supports a structured query language (structured query language, SQL). Generally, SQL refers to a specialized programming language that is used exclusively to manage data stored in a relational database. SQL may refer to various types of data-dependent languages including, for example, data definition languages and data manipulation languages, where the scope of SQL may include data insertion, querying, updating and deleting, schema creation and modification, and data access control. Further, in some examples, SQL may include descriptions related to various language elements.
A query is a request to view, access and/or manipulate data stored in a database. Database management system 108 may receive queries in SQL format (referred to as SQL queries) from database clients 102. Typically, the database management system 108 generates query results corresponding to the query by accessing relevant data from the database and manipulating the relevant data, and returns the query results to the database client 102. A database is a collection of data organized, described, and stored in a mathematical model, and may include one or more database structures or formats, such as row and column stores. The database is typically stored in a data store, such as external data store 120 in FIG. 2, or non-transitory computer readable medium 106. When the database is stored on the non-transitory computer readable medium 106, the database management system 108 is an in-memory database management system.
Database client 102 may include any type of device or application configured to interact with database management system 108. In some examples, database client 102 includes one or more application servers.
The database management system 108 includes an SQL engine 110, an execution engine 122, and a storage engine 134. The SQL engine 110 generates a corresponding execution plan from SQL statements, e.g., queries (Query), submitted by the client 102, and the execution engine 122 operates according to the execution plan of the statements to generate Query results. The storage engine 134 is responsible for managing the table data, the index, and the actual content on the file system, and also managing the Cache, buffer, transaction, log, and other data at runtime. For example, storage engine 134 may write the execution results of execution engine 122 to data store 120 via physical I/O. The SQL engine 110 comprises a parser (not shown) for performing syntactic and semantic analysis of SQL statements, expanding and partitioning the views in the query into small query blocks, and an optimizer (not shown). The optimizer generates a set of execution plans for the sentence that may be used, estimates the cost of each execution plan, compares the costs of the plans, and finally selects an execution plan with the smallest cost.
The following describes important concepts related to the embodiments of the present application:
A stored procedure (stored procedure) is a set of statements stored in a database system that can perform a specific function, and the stored procedure is called again after the first compilation without need for recompilation, and the user can run the stored procedure by specifying the name of the stored procedure.
A transaction is a sequence of database operations defined by a user in a database system, which operations are either all or none, and are an integral unit of work. Transactions have four properties, atomic, consistency, isolation, and persistence (durability). In a relational database, a transaction may be a structured query language (structured query language, SQL) statement, a set of SQL statements, or an entire program. Statements defining transactions in SQL include BEGIN (BEGIN), COMMIT (COMMIT), ROLLBACK (ROLLBACK).
Transactions typically BEGIN with BEGIN and end with COMMIT or ROLLBACK. COMMIT indicates that all operations of the transaction are committed, specifically, all updates to the database in the transaction are written to disk, and the transaction is normally ended. ROLLBACK represents a ROLLBACK, i.e., some failure occurs during the execution of a transaction, the transaction cannot continue, and the system rolls back all completed operations to the database in the transaction (UNDO) to the state at the beginning of the transaction.
In the prior art, a database server executes a storage process according to a request of a client, and when the execution of the storage process fails, the whole storage process can only be retried completely, and the control granularity is the storage process level.
When the stored procedure includes one or more transactions, if the stored procedure is in error, then there is a transaction that ends with a COMMIT, which represents that all updates to the database in the transaction have been written to disk. When retrying the storage process, the transaction is repeatedly executed, which will cause data errors. If a transaction ends with a ROLLBACK before the storage process is in error, which means that a fault occurs in the process of running the transaction, the transaction cannot be continuously executed, and when the storage process is retried, the ROLLBACK transaction is repeatedly executed, so that resources are wasted.
In addition, prior art retries need to take into account the driving support of different systems, such as open database connection (open database connectivity, ODBC), java database connection (Java database connectivity, JDBC) object relational database management system (object-relational database MANAGEMENT SYSTEM, orbms) postgresql c language application interface Libpq, etc.
In order to solve the above-mentioned problems, an embodiment of the present application provides a data processing method, which is used for a retry process of an execution failure of a storage process, and is described below with reference to a specific embodiment.
Referring to fig. 3, an embodiment of a data processing method according to an embodiment of the application is shown.
301. Executing a storage process;
The database management system receives a request for a stored procedure sent by a client, the stored procedure can be created by the client, refer to fig. 4, and for an embodiment of creating a stored procedure, the stored procedure is created by sending a statement to create the stored procedure, and the statement is registered in the database, illustratively by the SQL client. Thereafter, the stored procedure may be invoked directly, e.g., by the statement "CALL STPTEST1 ()", a stored procedure request is sent to effect the invocation, and the stored procedure is executed in the database. The storage includes one or more ordered transactions.
302. Recording first position and intermediate state information of ending of a first transaction;
During execution of the stored procedure, when execution of a transaction ends, the database management system may record the tag information of the transaction, optionally, the database management system records location information of the statement that the transaction ended, or an identification of the transaction. Specifically, the database management system records the line number of the transaction end statement. The location at the end of the transaction, including the location of the COMMIT or the location of the ROLLBACK. For example, as shown in fig. 4, if the fourth line is in error, the position of the last COMMIT before the fourth line, i.e., the second line (line 2), is recorded;
in addition, the database management system also records intermediate state information of the stored procedures.
The intermediate state information includes various parameters or variables generated during the execution of the process, such as execution statement results, error prompt information generated during execution, temporary variables, global variables, input parameters, output parameters, branch execution states, and the like. It will be appreciated that intermediate state information is not satisfactory for data consistency if missing the intermediate state information during a retry, as is necessary for retries.
Optionally, the database management system caches the execution statement result or error prompt information generated in execution. In the storage process, if the intermediate result is needed to be returned before the COMMIT, the intermediate result is cached and is not sent to the user equipment until the COMMIT is performed, or the intermediate result is uniformly sent when the storage process is finished. By way of example, the term "track non-word 'hello world';" indicates that the screen print prompt message 'hello world' is sent directly before the end of the transaction, and is resent again when retried, so that the same message is printed on the screen multiple times, and therefore, the result of the execution statement is saved through data buffering, and is sent until the end of the execution of the storage process or the end of the transaction, so that repeated operations can be avoided.
Optionally, the database management system records values of variables of the stored procedure, including temporary variable execution results. For example, after the execution of the second row of COMMIT in FIG. 4, the value of variable i at that time is recorded.
Optionally, the database management system records input parameters and/or output parameters of the stored procedure, e.g., input parameters x, y of the stored procedure in fig. 4.
Optionally, the first transaction is located in a branch of the branch statement in the storing process, and the database management system records a branch execution state, where the branch execution state is used to identify the branch in which the first transaction is located. The branch execution state includes a conditional branch execution state or a case branch execution state, etc. Optionally, the branch execution state is a conditional determination result of the branch statement. Fig. 5 is a flow chart of a process of storing a process execution procedure according to an embodiment of the application. For example, as shown in fig. 5, if an if branch is included in the execution of the storage procedure, a branch determination result needs to be recorded, for example, for a branch determination if (select= =6), if select is equal to 6, a first branch is entered, if select is not equal to 6, a second branch is entered, the branch determination result is recorded to be equal to 6, and may be used to determine to execute the first branch, the branch determination result is recorded to be not equal to 6, and may be used to determine to execute the second branch, and when the storage procedure is retried, a branch determined according to the branch determination result may be directly entered without re-executing the if determination statement.
For a stored procedure that contains multiple transactions, there is corresponding intermediate state information for the end position of each transaction. For example, if the storage procedure includes 3 transactions, namely, an a transaction, a B transaction and a C transaction according to an execution sequence, when the a transaction is ended in the execution procedure of the storage procedure, corresponding first intermediate state information is recorded, when the B transaction is ended, corresponding second intermediate state information is recorded, at this time, optionally, the first intermediate state information may be released, if an error occurs in the C transaction in the execution procedure, when retrying, the already ended a transaction and B transaction may be skipped, and according to the second intermediate state information, retrying may be started after the position where the B transaction is ended.
Optionally, a request statement sent by the client to call the stored procedure, such as "CALL STPTEST1 ()", the request statement and the transaction end statement line number may be used together to determine the location of the retry start.
303. Retrying the stored procedure based on the first location and intermediate state information at which the first transaction ended;
based on the condition that the exception occurs in the process of executing the storage and the condition of retrying the storage process is met, retrying the transaction after the first transaction in the storage process according to the marking information of the first transaction.
Conditions for retrying the stored procedure include that the exception is a restorable exception. The types of exceptions that cause failure in the storage process execution are various, such as network exceptions, e.g., network flashovers, memory usage exceptions, e.g., memory application failures, input/output (I/O) exceptions, e.g., read exceptions due to busy hard disks, lock wait timeout exceptions, exceptions that occur during the two-phase transaction execution, including two-phase transaction commit exceptions. Specific reasons for failure of execution of the stored procedure are not limited herein. If the exception is a restorable exception that can be resolved by retry, a retry of the storage procedure will occur.
The database management system can realize exception capture, identify exception types, judge whether the exception types are restorable exceptions or not so as to determine whether retry is needed or not, and initiate retry logic at the server side if retry is needed. The specific process is not described here.
It should be noted that, the data processing aspect of the embodiment of the present application may be applicable to retry of execution failure of a storage process in different database usage scenarios, including application scenarios such as a stand-alone database, a cluster database, and a master/slave database.
The retry includes one retry or multiple retries, and the retry time may be a direct retry, a retry after waiting for a preset period of time, or a retry when a preset condition is satisfied, and the specific number of retries and the retry time are not limited herein.
Retrying the transaction following the first transaction in the storage according to the marking information of the first transaction. When the database management system retries, through the previously recorded execution intermediate state, the executed and submitted statement is skipped, the consistency state after the previous submission is restored, and the transaction after the first transaction is retried.
It will be appreciated that one or more transactions may be included in a storage process, and if there are multiple transactions in a storage process, when the execution of the storage process fails, a retry may be performed from the last transaction completed, e.g., for a storage process containing 3 transactions, if there is an error in the execution of the 3 rd transaction, a retry may be performed from the end of the 2 nd transaction.
For example, as shown in fig. 4, if the fourth line is in error, the sentence before the second line of COMMIT is skipped during retry, and the sentence after the second line of COMMIT is continuously executed according to the variable i recorded in step 302, the parameters x and y are input, and the intermediate result generated by execution is executed.
If the retry execution is successful, the correct execution result can be returned to the user.
The data processing method provided by the embodiment of the application realizes the execution failure retry of the storage process, supports the execution failure retry of the storage process containing the transaction, reduces the retry granularity to the transaction level because the finished transaction does not need to be re-executed, improves the execution efficiency of the retry, and avoids the error caused by repeatedly executing the submitted part during the retry.
Fig. 6 is a schematic diagram of a system architecture of a cluster scenario in an embodiment of the application.
In the database cluster scenario, including a coordinating node (coordinator) and a plurality of data nodes (data nodes), the data nodes shown in fig. 6 are deployed in a primary-backup mode, and when a primary data node fails, a backup data node takes over a query request. When a fault occurs, the coordination node decides whether to retry, retry strategy and the like according to the retry decision module. The embodiment of the application can process the retry of the primary and standby switching scenes and cope with various situations such as planned shutdown, unplanned shutdown and the like. Typically, the method comprises the steps of retrying the execution failure of the storage process and retrying the execution failure scene caused by the switching of the master node and the standby node. The following description will be made respectively:
1. In the cluster scenario, the storage process fails to execute a retry.
Referring to fig. 7, a diagram of one embodiment of a retry of a storage procedure in a clustered scenario is shown in an embodiment of the present application.
701. The business application sends a storage process request to the coordination node;
In a cluster database scenario, a business application sends a stored procedure request to a coordinating node.
702. The coordination node issues the request of the storage process;
And the coordination node executes the stored procedure request, and sends SQL to the data node according to the actual demand or preset rules, and the data node executes part of SQL.
The coordination node can acquire intermediate state information of a storage process and can also capture an abnormality in an execution process.
703. Synchronizing the master and the slave;
if the data nodes are configured in a master-slave mode, SQL received by the group data nodes is synchronized to the slave data nodes.
Step 703 is an optional step, and may or may not be performed, which is not limited herein.
704. The coordination node acquires failure information;
the storage process execution failure includes a coordination node execution failure or a data node execution failure.
Optionally, the types of exceptions that may cause failure in execution of the storage process include network exceptions such as network flashovers, memory usage exceptions such as memory application failures, input/output (I/O) exceptions such as read exceptions due to busy hard disk, lock wait timeout exceptions, and exceptions that occur during execution of the two-phase transaction. Specific reasons for failure of execution of the stored procedure are not limited herein. If the anomaly is a restorable anomaly, a retry of the stored procedure will be initiated.
Optionally, the data node executes SQL and returns the execution result to the coordination node, if the data node executes error, the coordination node can obtain failure information of the storage process.
The specific implementation process refers to step 302 in the corresponding embodiment of fig. 3, which is not described herein.
705. The coordination node checks the failure reason;
the coordination node identifies whether the reason of the failure of the execution of the data node is restorability abnormality which can be restored by retry, if so, the retry is initiated. Optionally, the coordination node decision module judges whether to retry or not and the time for delaying retry according to the current active/standby situation.
706. The storage process is retried.
The coordination node sends a retry request according to the recorded intermediate state information of the storage process, and the main data node or the standby data node retries according to the intermediate state information of the storage process, and the specific implementation process refers to step 303 in the embodiment corresponding to fig. 3, which is not described herein again.
2. In the cluster scene, the master-slave node is switched to cause the retry of the execution failure.
Referring to fig. 8, another embodiment of a retry of a storage procedure in a clustered scenario is shown in an embodiment of the present application.
801. The business application sends a storage process request to the coordination node;
802. The coordination node issues the request of the storage process;
803. synchronizing the master and the slave;
Step 803 is an optional step, and may or may not be performed, which is not limited herein.
Steps 801 to 803 are similar to steps 701 to 703 in the corresponding embodiment of fig. 7, and are not described here again.
804. When the data node fails to execute, the master-slave switching is performed;
When the data node performs the storing process, the coordinating node carries the record data with the intermediate state information of the storing process, and the specific implementation process refers to step 302 in the embodiment corresponding to fig. 3, which is not described herein again.
When the data node storage process fails to execute, an error is reported to the coordination node.
Optionally, there are various types of exceptions that cause the data node to store the process execution failure, and specific reasons for the store process execution failure are not limited herein.
If the main data node crashes and is down, the working state of the standby data node is normal, the background cluster management module senses the situation, and the coordination node indicates to perform main and standby switching. The switching of the master and slave nodes is generally controlled by a cluster management system, the cluster management system is mainly responsible for starting and stopping the whole cluster, and collects the running state of each node, and when the running abnormality is found, the master and slave switching is controlled by sending a command to the data node.
805. The storage process is retried.
The coordination node retries according to the recorded intermediate state information of the storage process, including sending the execution task to the backup data node for retrying, and the specific implementation process refers to step 303 in the embodiment corresponding to fig. 3, which is not described herein again.
Optionally, the coordination node may synchronize data with other coordination nodes in a synchronous manner, or save the variable in the execution result set before the execution failure of the storage process in a disk-drop manner, so that even if the coordination node crashes, the retry of the execution failure of the storage process can be realized.
Fig. 9 is a schematic diagram of a system architecture of a method for retrying a failed execution of a storage process according to an embodiment of the application.
The database management system stores the position of the executed transaction end statement and the intermediate state information when the transaction ends in the execution of the storage process, and after the database management system captures the abnormality, the database management system resends the storage process request to realize retry. Because the data processing method of the embodiment records the position when the transaction is ended and the corresponding intermediate state information, fine-granularity retry can be realized, and the customer experience when the software and the hardware are abnormal is improved.
The data processing method for implementing the retry of the execution failure of the storage process was described above, and the data processing apparatus implementing the method will be described below.
Referring to FIG. 10, a database management system according to an embodiment of the present application is shown.
The database management system provided by the embodiment of the application can be deployed in a stand-alone database system or nodes in various cluster database systems, and the nodes can run in physical machines or virtual machines, and are not limited in particular.
In practical applications, the data processing device for deploying the database management system may be a database server, or a dedicated data processing device, which is not limited herein.
The database management system 1000 provided in the embodiment of the present application includes:
an execution unit 1001 for executing a stored procedure, the stored procedure comprising one or more ordered transactions, in response to a stored procedure request;
A recording unit 1002 configured to record tag information of a first transaction when execution of the first transaction is ended among the one or more ordered transactions;
and a retry unit 1003, configured to retry a transaction after the first transaction in the storage process according to the tag information of the first transaction, based on that an exception occurs in the execution of the storage process and that a condition for retrying the storage process is satisfied.
The recording unit 1002 is further configured to record a request statement for invoking the stored procedure.
The tag information includes a line number of an end statement of the first transaction.
The apparatus further comprises:
A recording unit 1002, configured to record intermediate state information of the storage procedure, where the intermediate state information includes one or more of an execution statement result of the first transaction, error prompt information, an input parameter, an output parameter, a global variable, and a temporary variable.
The intermediate state information includes an execution statement result or error hint information of the first transaction, the apparatus further comprising:
The buffer unit 1004 is configured to buffer an execution statement result or error prompt information of the first transaction until the first transaction is ended, and then send the execution statement result or error prompt information to the user equipment.
The intermediate state information includes an execution statement result or error hint information of the first transaction, the apparatus further comprising:
the buffer unit 1004 is configured to buffer the execution statement result or the error prompt information of the first transaction until the storage process is finished, and then send the execution statement result or the error prompt information to the user equipment.
The first transaction is located at a branch of a branch statement in the storage process, the recording unit 1002 is further configured to:
recording intermediate state information of the stored procedure, the intermediate state information including a branch execution state, the branch execution state being used to identify a branch in which the first transaction is located;
the retry unit 1003 is specifically configured to execute, based on the branch execution status, a transaction subsequent to the first transaction in the branch where the first transaction is located.
The branch execution state includes a conditional determination of the branch statement in the execution of the store.
The condition for retrying the storage process is that the storage process exception is a restorability exception, including a network exception, a memory use exception, an input-output exception, a lock wait timeout exception, or a two-phase transaction commit exception.
The retry unit 1003 is specifically configured to retry a transaction after the first transaction in the storage process after waiting for a preset first duration.
The recording unit 1002 is further configured to record, when execution of a second transaction of the one or more ordered transactions ends, tag information of the second transaction, where the second transaction is a transaction executed after the first transaction;
the retry unit 1003 is specifically configured to retry a transaction after the second transaction in the storage procedure according to the tag information of the second transaction, where the second transaction is a last executed transaction in the executed transactions when the condition of retrying the storage procedure is satisfied.
Referring to FIG. 11, another embodiment of a database management system according to the present application is shown.
The database management system may include a plurality of software functional modules, including a database server framework (DB framework), a database compiler, a database executor, and a database exception handling module, where the newly added submodule implements retry logic. Wherein:
1. And the database server framework is used for analyzing the information sent by the user to determine what kind of information is, such as SQL information, PBE information and the like. The method is also used for storing original request sentences, preparing for retry, storing intermediate state information and the like.
2. And the database compiler is used for limiting the inspection of the application scene, and setting a non-retry mark if a non-support scene is met.
3. And the database executor is used for storing and storing the process variables and parameters, storing the positions of the transaction, including the positions of COMMIT and ROLLBACK, storing the intermediate state information of the transaction, including the first process parameters, the branch execution state, the execution results of the first variables and the like.
4. The storage process exception handling module is used for capturing exceptions, checking whether retries are needed or not, and can also be used for rollback transactions.
The database server can comprise a re-execution module used for starting a new transaction, skipping over a COMMIT statement according to the stored COMMIT position information when the database server is executed last time and recovering the current state comprising variables and parameters and result set information when the database server is executed last time according to information of a COMMIT point when the database server is executed last time.
It should be noted that, the preconditions include that the data is not sent to the client before the storage process is executed, and that the failure is a processable failure, and that the retry of the internal failure of the program is meaningless.
5. The abnormal retry module is used for restarting a new transaction, skipping over submitted sentences according to intermediate state information such as stored COMMIT position information, if branch judging results and the like in the last execution, and recovering the current state including variables, parameters and the like according to intermediate state information corresponding to the COMMIT points in the last execution.
Referring to FIG. 12, a schematic diagram of an embodiment of a data processing apparatus is shown.
The data processing device provided in this embodiment may be a server or a dedicated data processing device, for example, a database server, which is not limited in the embodiment of the present application.
The data processing apparatus 1200 may vary considerably in configuration or performance and may include one or more processors 1201 and memory 1202, with programs or data stored in the memory 1202.
Wherein the memory 1202 may be volatile storage or nonvolatile storage. Processor 1201 is optionally one or more central processing units (central processing unit, CPU), which may be a single-core CPU or a multi-core CPU, or a special purpose processor. The processor 1201 may be in communication with the memory 1202 to execute a series of instructions in the memory 1202 on the data processing apparatus 1200.
The data processing apparatus 1200 also includes one or more wired or wireless network interfaces 1203, such as an ethernet interface.
Optionally, although not shown in FIG. 12, the data processing device 1200 may also include one or more power supplies, one or more input/output interfaces, which may be used to connect a display, mouse, keyboard, touch screen device, or sensing device, etc., with or without optional components, and is not limited herein.
The flow executed by the processor 1201 in the data processing apparatus 1200 in this embodiment may refer to the method flow described in the foregoing method embodiment, and will not be described herein.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
While the application has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that the foregoing embodiments may be modified or equivalents may be substituted for some of the features thereof, and that the modifications or substitutions do not depart from the spirit and scope of the embodiments of the application.