CN110134704B

CN110134704B - A big data cluster transaction implementation method based on distributed cache

Info

Publication number: CN110134704B
Application number: CN201910467807.9A
Authority: CN
Inventors: 朱喜娜
Original assignee: Xiamen University Tan Kah Kee College
Current assignee: Xiamen University Tan Kah Kee College
Priority date: 2019-05-31
Filing date: 2019-05-31
Publication date: 2021-11-02
Anticipated expiration: 2039-05-31
Also published as: CN110134704A

Abstract

The invention relates to a big data cluster transaction implementation method based on distributed cache. The access proxy module interprets all terminal user access requests of big data grammatically, and intercepts transaction-related access. Transaction-related operation requests such as re-column data, such transactions will be intercepted by the transaction interception module and managed by the distributed cache transaction management cluster for transaction consistency, and lock the libraries and tables of the big data data warehouse with transaction operations. , so as to realize the distributed transaction consistency of the big data platform.

Description

Big data cluster transaction implementation method based on distributed cache

Technical Field

The invention relates to the technical field of big data, in particular to a big data cluster transaction implementation method based on distributed cache.

Background

The prior art framework of the big data platform has the difficulty that the transaction consistency cannot be realized, namely, the distributed transaction cannot be realized temporarily on the big data cluster platform. The prior art is mainly a single-engine Database Transaction (Database Transaction), which refers to a series of operations executed by a group of Database operation commands as a single logical work unit on a single Database, and the operations are either completely executed or not executed at all, but the above techniques can only implement transactions in a non-distributed environment.

Disclosure of Invention

In view of this, the present invention provides a method for implementing big data cluster transaction based on distributed cache, which can implement distributed database transaction on a big data platform.

The invention is realized by adopting the following scheme: a big data cluster transaction implementation method based on distributed cache specifically comprises the following steps:

step S1: providing an access agent module, intercepting and acting access requests of all terminal users;

step S2: providing a grammar parsing module, parsing the access request of the step S1, and entering a band containing the transaction operation into the next step;

step S3: providing a transaction interception module, transmitting the grammar analysis module to a table containing transaction operation, locking the transaction of the table, and forbidding the query and operation of the table in the time period of the transaction operation;

step S4: the transaction attributes of the table are defined through the transaction interception module and stored in the distributed cache cluster, so that the access requests of all nodes of the big data distributed cluster can directly acquire the transaction attributes of the database table of the distributed big data in the distributed cache, and distributed transactions are realized. The distributed cache transaction management cluster can be realized on different access agent modules or other access interfaces to realize transaction consistency and avoid the problem of transaction conflict.

Further, in step S1, the access agent module is a unique necessary interface for the external end user to access the big data, so that all the access related to the transaction can be effectively intercepted and processed, and the transaction consistency of all the users can be realized.

Preferably, the syntax parsing module defines a unified SQL standard and syntax, and needs to perform unified parsing on the SQL syntax, so as to transform an input 'character string' into a 'structure body' describing the character string, so that the computer can more easily understand what the user inputs the character string. This stage includes three processes, lexical analysis, syntactic analysis, and output of abstract syntax trees.

1) Lexical analysis: is a Deterministic Finite Automata (DFA) that can convert an input character set into 'words' according to a defined lexical method.

2) And (3) syntax analysis: after lexical analysis, the next process is grammatical analysis, the result of the lexical analysis can be used as the input of the grammatical analysis, the grammatical analysis judges whether a word input by a user accords with a sentence containing a transaction operation on the basis of the lexical analysis, for example, an INSERT INTO table valuES is a sentence conforming to the transaction operation, a corresponding operated table is handed to a transaction interception module for processing, the grammatical interpretation can carry out the transaction analysis of a table partition with finer granularity, so that the transaction interception of the table partition is realized, other partitions can be normally accessed and cannot be locked, and the concurrency performance of the transaction is improved.

Further, in step S3, the prohibiting the table modification and query operation during the time period of the transaction operation further includes: if the table has partitions, the partitions are only locked at the transaction level, and the consistency of the transactions is realized.

Further, in step S3, the transaction intercepting module filters the access requests of all terminals, and performs transaction consistency determination for the query and operation related to the transaction, and if the corresponding table is performing the transaction operation, the access requests are made to enter an access blocking state until the related access request operation is executed after the transaction is finished.

Particularly, all terminal user access requests of the big data are subjected to syntax interpretation through the access agent module, transaction related access is intercepted, for transaction related operation requests containing row adding, row deleting, column changing data and the like, the transactions are intercepted by the transaction intercepting module, transaction consistency management is carried out through the distributed cache transaction management cluster, and libraries and tables of a data warehouse of the big data with transaction operation are locked, so that distributed transaction consistency of the big data platform is achieved.

Compared with the prior art, the invention has the following beneficial effects:

1. the invention realizes distributed transaction management and solves the problem of inconsistent transactions of the big data distributed data warehouse.

2. The invention can realize the distributed transaction consistency in the distributed environment of big data and realize the transaction consistency of distributed application programs through the transaction attribute of the cluster distributed cache.

3. The transaction granularity in the invention is table partitioning: the transaction locking granularity is finer, the transaction locking can be performed on the table partition, other table partitions are not affected, better transaction concurrency performance can be achieved, and the access and transaction operation of other partitions are not affected.

Drawings

FIG. 1 is a schematic block diagram of an embodiment of the present invention.

Detailed Description

The invention is further explained below with reference to the drawings and the embodiments.

It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.

As shown in fig. 1, this embodiment provides a method for implementing a big data cluster transaction based on a distributed cache, which specifically includes the following steps:

step S4: the transaction attribute of the table is defined by the transaction interception module and is stored in the cluster based on the distributed cache (memory), so that the access requests of all nodes of the big data distributed cluster can directly acquire the transaction attribute of the database table of the distributed big data in the distributed cache, and the distributed transaction is realized. The distributed cache transaction management cluster can be realized on different access agent modules or other access interfaces to realize transaction consistency and avoid the problem of transaction conflict.

In this embodiment, in step S1, the access proxy module is the only necessary interface for the external end user to access the big data, so that all the access related to the transaction can be effectively intercepted and processed, and the transaction consistency of all the users can be realized.

Preferably, in this embodiment, the syntax parsing module defines a unified SQL standard and syntax, and needs to perform unified parsing on the SQL syntax, so as to transform an input 'character string' into a 'structure body' describing the character string, so that the computer can more easily understand what the user inputs the character string. This stage includes three processes, lexical analysis, syntactic analysis, and output of abstract syntax trees.

In this embodiment, in step S3, the prohibiting the table modification and query operation during the time period of the transaction operation further includes: if the table has partitions, the partitions are only locked at the transaction level, and the consistency of the transactions is realized.

In this embodiment, in step S3, the transaction intercepting module filters the access requests of all terminals, performs transaction consistency determination for query and operation related to the transaction, and if the corresponding table is performing the transaction operation, makes the access requests enter an access blocking state until the related access request operation is executed after the transaction is finished.

Particularly, in this embodiment, the access agent module performs syntax interpretation on all terminal user access requests of the big data, intercepts transaction-related access, intercepts such transactions by the transaction interception module and performs transaction consistency management by the distributed cache transaction management cluster for transaction-related operation requests including row addition, row deletion, column change data and the like, and locks libraries and tables of a data warehouse of the big data with transaction operation, thereby implementing distributed transaction consistency of the big data platform.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The foregoing is directed to preferred embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow. However, any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the present invention are within the protection scope of the technical solution of the present invention.

Claims

1. a big data cluster transaction realization method based on distributed cache, is characterized in that, comprises the following steps:

Step S1: provide an access proxy module to intercept and proxy all access requests of end users;

Step S2: provide a grammar parsing module, perform grammar parsing on the access request of Step S1, and bring the table containing the transaction operation to the next step;

Step S3: provide a transaction interception module, pass the grammar parsing module to the table containing the transaction operation, perform transaction locking of the table, and prohibit the modification and query operation of the table within the time period of the transaction operation;

Step S4: Define the transaction attribute of the table through the transaction interception module, and store it in the distributed cache-based cluster, so that the access requests of all nodes of the big data distributed cluster can directly obtain the distributed big data database in the distributed cache The transaction attribute of the table, so as to realize the distributed transaction;

Wherein, in step S3, the prohibition of modifying and querying the table within the time period of the transaction operation further includes: if the table has a partition, only perform a transaction-level partition lock on the partition to achieve transaction consistency .

2 . The method for realizing a big data cluster transaction based on a distributed cache according to claim 1 , wherein, in step S1 , the access proxy module is the only necessary interface for external end users to access big data. 3 .

3. a kind of big data cluster transaction realization method based on distributed cache according to claim 1 is characterized in that, in step S3, described transaction interception module filters the access requests of all terminals, and for transaction-related access requests. The modification and query operations are used for transaction consistency judgment. If the corresponding table is undergoing transaction operations, these access requests will enter the access blocking state, and the related access request operations will not be executed until the transaction ends.