CN112818018A - Data processing method, device and system for distributed database - Google Patents
Data processing method, device and system for distributed database Download PDFInfo
- Publication number
- CN112818018A CN112818018A CN202110109567.2A CN202110109567A CN112818018A CN 112818018 A CN112818018 A CN 112818018A CN 202110109567 A CN202110109567 A CN 202110109567A CN 112818018 A CN112818018 A CN 112818018A
- Authority
- CN
- China
- Prior art keywords
- data
- log
- log information
- distributed database
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/466—Transaction processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明提供一种分布式数据库的数据处理方法、装置及系统，该方法包括：存储节点接收协调节点发送的日志信息，并在内存中构造一个队列用于存储所述日志信息，形成日志队列；从所述日志队列中按照先进先出的次序依次读出存储的所述日志信息，将读出的所述日志信息写入硬盘；根据日志序列号排序所述日志信息，并按事务ID区分所述日志信息；解析所述日志信息中的数据改动操作，产生新的数据页；根据所述新的数据页，写入及备份数据。本发明只传输和写入日志，降低了网络及IO开销，提高了性能。
The invention provides a data processing method, device and system for a distributed database. The method includes: a storage node receives log information sent by a coordination node, and constructs a queue in a memory for storing the log information to form a log queue; The stored log information is sequentially read from the log queue according to the first-in, first-out order, and the read log information is written into the hard disk; the log information is sorted according to the log sequence number, and the log information is sorted by transaction ID. The log information is analyzed; data modification operations in the log information are parsed to generate a new data page; data is written and backed up according to the new data page. The present invention only transmits and writes logs, reduces network and IO overhead, and improves performance.
Description
Technical Field
The present invention relates to the field of distributed database technologies, and in particular, to a method, an apparatus, and a system for processing data in a distributed database.
Background
PGXL is a distributed database implemented based on PostgreSQL, separating coordinating nodes from data nodes, thereby implementing horizontal spreading. Currently, when a coordinating node writes data into a data node, the transmitted content includes data pages, metadata, logs, and the like. The current PGXL coordination node writes data pages, metadata, logs and other files into the data node, so that huge network and IO input/output bottlenecks are caused.
Disclosure of Invention
In view of this, the present invention provides a data processing method, device and system for a distributed database, so as to reduce the overhead of data processing of the distributed database, and further improve the data processing performance of the distributed database.
In one aspect, the present invention provides a data processing method for a distributed database, where the distributed database includes: the data processing method of the distributed database comprises the following steps:
the storage node receives the log information sent by the coordination node, and constructs a queue in a memory for storing the log information to form a log queue;
sequentially reading the stored log information from the log queue according to the first-in first-out order, and writing the read log information into a hard disk;
sorting the log information according to a log serial number, and distinguishing the log information according to a transaction ID;
analyzing data change operation in the log information to generate a new data page;
and writing and backing up data according to the new data page.
Further, the step of writing and backing up data according to the new data page comprises: the outdated data is periodically purged to free up space.
Further, the step of analyzing the data change operation in the log information and generating a new data page includes: judging whether data fragmentation is needed or not;
and when the data fragmentation is determined to be needed, finding the fragmentation position, and executing the steps of writing and backing up data according to the new data page according to the fragmentation position.
Further, the distributed database is a Postgres-XL distributed database.
In another aspect, the present invention further provides a data processing system for a distributed database, including:
the log receiving module is used for receiving the log information sent by the coordination node, and constructing a queue in a memory for storing the log information to form a log queue;
the log updating module is used for sequentially reading the stored log information from the log queue according to a first-in first-out order and writing the read log information into a hard disk;
the log sorting and sorting module is used for sorting the log information according to the log serial number and distinguishing the log information according to the transaction ID;
the log analysis module is used for analyzing the data change operation in the log information to generate a new data page;
and the data writing module is used for writing and backing up data according to the new data page.
Further, the data processing apparatus of the distributed database further includes: and the data cleaning module is used for periodically cleaning the expired data to release the space.
Further, the log parsing module is further configured to: and judging whether the data fragmentation is needed, finding the fragmentation position when the data fragmentation is needed, and writing and backing up the data according to the fragmentation position and the new data page.
Further, the distributed database is a Postgres-XL distributed database.
In another aspect, the present invention further provides a data processing system for a distributed database, including: the memory is used for receiving the log information sent by the coordination node and constructing a queue for storing the log information sent by the coordination node to form a log queue;
the hard disk is used for sequentially reading the stored log information from the log queue according to the first-in first-out order and storing the read log information and the data page;
the CPU is used for sequencing the log information according to the log serial number and distinguishing the log information according to the transaction ID; analyzing data change operation in the log information to generate a new data page; and writing and backing up data according to the new data page.
Further, the CPU is further configured to: and judging whether the data fragmentation is needed, finding the fragmentation position when the data fragmentation is needed, and writing and backing up data according to the fragmentation position and the new data page.
According to the data processing method, device and system of the distributed database, only the redo log is constructed when the coordination node needs to write data into the data node; the data node puts the log sent by the coordinating node into a memory queue, then removes the log from the queue and makes the log persistent to the hard disk; and the data nodes merge the log records to generate the data of the latest version, and the data is converted into data blocks (namely data pages) required by the database. Because only logs are written between the coordination nodes of Postgres-XL (PGXL) and the data nodes, files such as data pages, metadata and the like are not written, and data construction work is handed to the data nodes, the writing efficiency and the overall performance of the PGXL distributed database are greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method of a distributed database according to an exemplary first embodiment of the present invention.
Fig. 2 is a flowchart of a data processing method of a distributed database according to an exemplary second embodiment of the present invention.
Fig. 3 is a block diagram of a data processing apparatus of a distributed database according to an exemplary third embodiment of the present invention.
Fig. 4 is a block diagram of a data processing system of a distributed database according to an exemplary third embodiment of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
It should be noted that, in the case of no conflict, the features in the following embodiments and examples may be combined with each other; moreover, all other embodiments that can be derived by one of ordinary skill in the art from the embodiments disclosed herein without making any creative effort fall within the scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the appended claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the disclosure, one skilled in the art should appreciate that one aspect described herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. Additionally, such an apparatus may be implemented and/or such a method may be practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
Fig. 1 is a flowchart of a data processing method for a distributed database according to an exemplary first embodiment of the present invention, and as shown in fig. 1, the data processing method for a distributed database according to the present invention includes:
step 101: the storage node receives the log sent by the coordination node, and in order to avoid the problem caused by the transmission speed being higher than the processing speed, a queue is constructed in the memory to store the received log record (i.e. the log information), so as to form a log queue.
Step 102: and sequentially reading log records from the log queue according to the first-in first-out order, and persistently writing the log records into the hard disk. The change of the user to the data is persisted to the hard disk, and the persistence of the transaction is realized.
Step 103: the log records are sorted according to the log serial number and are sorted according to the transaction ID so as to distinguish the operation of each transaction, and the isolation of the transactions is ensured.
Step 104: and analyzing the data change operation in the log so as to generate a new data page.
Step 105: and writing and backing up data according to the new data page.
Preferably, because PGXL is a distributed database, if data fragmentation is needed, step 105 can first find the corresponding location of the data fragmentation, write the data, and copy the data to the backup data fragment. This process ensures transaction consistency using the quorum protocol.
Further preferably, after the data is successfully written, the data node sends a confirmation message to the coordinating node to inform the coordinating node that the data persistence process is completed. If the transaction fails, all data operations in the transaction roll back, so that the atomicity of the transaction is realized.
Further preferably, step 105 may be followed by: periodically, expired data, such as identified deleted data, is cleared, freeing space.
In the embodiment, when the coordination node needs to write data into the data node, only the redo log is constructed; the data node puts the log sent by the coordinating node into a memory queue, then removes the log from the queue and makes the log persistent to the hard disk; and the data nodes merge the log records to generate the data of the latest version, and the data is converted into data blocks (namely data pages) required by the database. Because only logs are written between the coordination nodes of Postgres-XL (PGXL) and the data nodes, files such as data pages, metadata and the like are not written, and data construction work is handed to the data nodes, the writing efficiency and the overall performance of the PGXL distributed database are greatly improved.
Fig. 2 is a flowchart of a data processing method of a distributed database according to an exemplary second embodiment of the present invention. Fig. 2 is a preferred embodiment of the process shown in fig. 1. As shown in fig. 2, the data processing method of the distributed database includes:
1. the storage node receives the log sent by the coordination node, and in order to avoid the problem caused by the fact that the transmission speed is higher than the processing speed, a queue is constructed in the memory of the storage node to store the received log record.
2. And sequentially reading the log records from the log receiving queue in a first-in first-out mode, and persistently writing the log records into the hard disk. This process is extremely simple, with no additional operations, and therefore fast.
3. The log records are sorted by log sequence number and sorted by transaction ID grouping. Therefore, the operation of each transaction can be distinguished, and the isolation of the transactions is ensured.
4. Analyzing the data change operation in the log so as to generate a new data page, which is the same as the recovery process by using the WAL log when the system fails: and analyzing the WAL log, acquiring the modified snapshot of the data page stored in the WAL log, and restoring the modified snapshot to a data buffer area similar to a traditional database.
5. Because PGXL is a distributed database, does the decision need to split the data?
6. If necessary, find the corresponding position of the data fragment.
7. And writing data and persisting the data to the hard disk.
8. And copying data into the backup data slice, and ensuring the consistency of transactions by using a quorom protocol.
9. Is it determined whether writing of copied data succeeded?
10. If the transaction fails, all data operations in the transaction roll back, so that the atomicity of the transaction is realized.
11. If the data persistence process is successful, the data node sends a confirmation message to the coordination node to tell the coordination node that the data persistence process is completed. The change of the user to the data is persisted to the hard disk, and the persistence of the transaction is realized.
12. Periodically, the hard disk space is released by using the expired data of the Vacuum process, such as the identified deletion data.
According to the embodiment, only the log is transmitted and written, so that the network overhead and the IO overhead are reduced, and the performance is improved; the log already contains the relevant operations of all users, and the data operation in the transaction is ensured to be either completely successful or completely failed by only analyzing the log, so that the atomicity of the transaction is realized; the log is sorted and sorted into data pages at the storage node, and the consistency of the transaction is ensured by utilizing the self quorum protocol of the PGXL; the logs are sorted and sorted at the storage nodes to distinguish the operation of each transaction, so that the isolation of the transactions is ensured; and the change of the user to the data is persisted to the hard disk by only analyzing the log, so that the persistence of the transaction is realized.
Fig. 3 is a block diagram of a data processing apparatus of a distributed database according to an exemplary third embodiment of the present invention. The embodiments shown in fig. 1 and fig. 2 can be applied to this embodiment, specifically, as shown in fig. 3, the data processing apparatus of the distributed database includes:
the log receiving module 301 is configured to receive a log sent by the coordinating node, store the log in a queue data structure of a memory of the storage node, and send a log record to the log updating module 302.
And the log updating module 302 is used for receiving log records from the log receiving module 301, persistently writing the log records into the hard disk, and sending the log records to the log sorting and sorting module 303.
And the log sorting and sorting module 303 is used for sorting the log records, sorting the log records according to the transaction ID groups, and sending the sorted log records to the log analysis module 304.
The log parsing module 304 is used to parse the data change operation in the log, construct a new data page, and send the new data page to the data writing module 305.
The data writing module 305 is used for persistently writing a new data page to the hard disk, copying the data to the data page copying node, and receiving a cleaning request sent by a data cleaning module (not shown in the figure). The data cleaning module is used to clean out expired data, release hard disk space, and send a cleaning command to the write data writing module 305 to complete cleaning.
The log of the embodiment already contains the relevant operations of all users, and the data operation in the transaction is ensured to be either completely successful or completely failed by only analyzing the log, so that the atomicity of the transaction is realized; the logs are sorted and sorted into data pages at the storage nodes by only analyzing the logs, and the consistency of transactions is ensured by utilizing the self quorum protocol of the PGXL; the log is sorted and sorted at the storage node by only analyzing the log to distinguish the operation of each transaction, so that the isolation of the transaction is ensured; and the change of the user to the data is persisted to the hard disk by only analyzing the log, so that the persistence of the transaction is realized.
Fig. 4 is a block diagram of a data processing system of a distributed database according to an exemplary fourth embodiment of the present invention. The embodiments shown in fig. 1 to fig. 3 can be applied to this embodiment, and specifically, as shown in fig. 4, the data processing system of the distributed database includes:
the memory is used for receiving the logs sent by the coordination node and storing the logs in a queue data structure of the memory of the storage node; specifically, the method is used for receiving the log information sent by the coordinating node, and constructing a queue for storing the log information sent by the coordinating node to form a log queue.
Hard disk, which is used to persist log record and data page; specifically, the method and the device are used for sequentially reading the stored log information from the log queue according to a first-in first-out order, and storing the read log information and the data page.
The CPU is used for processing the logs, including sorting, arranging and analyzing the logs, searching for overdue data and the like; specifically, the log sorting device is used for sorting the log information according to a log serial number and distinguishing the log information according to a transaction ID; analyzing data change operation in the log information to generate a new data page; and writing and backing up data according to the new data page.
Preferably, the CPU is further configured to: and judging whether the data fragmentation is needed, finding the fragmentation position when the data fragmentation is needed, and writing and backing up data according to the fragmentation position and the new data page.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A data processing method of a distributed database, the distributed database comprising: the distributed database data processing method comprises the following steps:
the storage node receives the log information sent by the coordination node, and constructs a queue in a memory for storing the log information to form a log queue;
sequentially reading the stored log information from the log queue according to the first-in first-out order, and writing the read log information into a hard disk;
sorting the log information according to a log serial number, and distinguishing the log information according to a transaction ID;
analyzing data change operation in the log information to generate a new data page;
and writing and backing up data according to the new data page.
2. The method of claim 1, wherein the step of writing and backing up data according to the new data page is followed by:
the outdated data is periodically purged to free up space.
3. The data processing method of the distributed database according to claim 1 or 2, wherein the step of parsing the data change operation in the log information and generating a new data page is followed by:
judging whether data fragmentation is needed or not;
and when the data fragmentation is determined to be needed, finding the fragmentation position, and executing the steps of writing and backing up data according to the new data page according to the fragmentation position.
4. The method for processing the data in the distributed database according to claim 3, wherein the distributed database is a Postgres-XL distributed database.
5. A data processing apparatus for a distributed database, comprising:
the log receiving module is used for receiving the log information sent by the coordination node, and constructing a queue in a memory for storing the log information to form a log queue;
the log updating module is used for sequentially reading the stored log information from the log queue according to a first-in first-out order and writing the read log information into a hard disk;
the log sorting and sorting module is used for sorting the log information according to the log serial number and distinguishing the log information according to the transaction ID;
the log analysis module is used for analyzing the data change operation in the log information to generate a new data page;
and the data writing module is used for writing and backing up data according to the new data page.
6. The data processing apparatus of a distributed database according to claim 5, further comprising:
and the data cleaning module is used for periodically cleaning the expired data to release the space.
7. The data processing apparatus of a distributed database according to claim 5 or 6, wherein the log parsing module is further configured to: and judging whether the data fragmentation is needed, finding the fragmentation position when the data fragmentation is needed, and writing and backing up the data according to the fragmentation position and the new data page.
8. The distributed database data processing apparatus as claimed in claim 7, wherein the distributed database is a Postgres-XL distributed database.
9. A data processing system for a distributed database, comprising:
the memory is used for receiving the log information sent by the coordination node and constructing a queue for storing the log information sent by the coordination node to form a log queue;
the hard disk is used for sequentially reading the stored log information from the log queue according to the first-in first-out order and storing the read log information and the data page;
the CPU is used for sequencing the log information according to the log serial number and distinguishing the log information according to the transaction ID; analyzing data change operation in the log information to generate a new data page; and writing and backing up data according to the new data page.
10. The data processing system of the distributed database of claim 9, wherein the CPU is further configured to: and judging whether the data fragmentation is needed, finding the fragmentation position when the data fragmentation is needed, and writing and backing up data according to the fragmentation position and the new data page.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110109567.2A CN112818018A (en) | 2021-01-26 | 2021-01-26 | Data processing method, device and system for distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110109567.2A CN112818018A (en) | 2021-01-26 | 2021-01-26 | Data processing method, device and system for distributed database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112818018A true CN112818018A (en) | 2021-05-18 |
Family
ID=75859687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110109567.2A Pending CN112818018A (en) | 2021-01-26 | 2021-01-26 | Data processing method, device and system for distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112818018A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101395602A (en) * | 2005-12-29 | 2009-03-25 | 亚马逊科技公司 | Method and apparatus for distributed file storage and indexing service |
| CN106503020A (en) * | 2015-09-08 | 2017-03-15 | 阿里巴巴集团控股有限公司 | Daily record data processing method and processing device |
| CN111930886A (en) * | 2020-07-06 | 2020-11-13 | 国网江西省电力有限公司电力科学研究院 | Log processing method, system, storage medium and computer equipment |
-
2021
- 2021-01-26 CN CN202110109567.2A patent/CN112818018A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101395602A (en) * | 2005-12-29 | 2009-03-25 | 亚马逊科技公司 | Method and apparatus for distributed file storage and indexing service |
| CN106503020A (en) * | 2015-09-08 | 2017-03-15 | 阿里巴巴集团控股有限公司 | Daily record data processing method and processing device |
| CN111930886A (en) * | 2020-07-06 | 2020-11-13 | 国网江西省电力有限公司电力科学研究院 | Log processing method, system, storage medium and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7271670B2 (en) | Data replication method, device, computer equipment and computer program | |
| JP7090744B2 (en) | Distributed database cluster system and data synchronization method | |
| US8250033B1 (en) | Replication of a data set using differential snapshots | |
| US6397351B1 (en) | Method and apparatus for rapid data restoration including on-demand output of sorted logged changes | |
| CN1752939B (en) | Method and system for synthetic backup and restore | |
| CN100375048C (en) | Method for restoring a repository to a previous state | |
| CN100562858C (en) | Method, device and system for remote disaster recovery of memory database | |
| CN108255647B (en) | High-speed data backup method under samba server cluster | |
| CN103761161B (en) | Recover the method for data, server and system | |
| US20150213100A1 (en) | Data synchronization method and system | |
| CN105138691B (en) | Analyze the method and system of subscriber traffic | |
| US9229970B2 (en) | Methods to minimize communication in a cluster database system | |
| CN105824744A (en) | Real-time log collection and analysis method on basis of B2B (Business to Business) platform | |
| US20090070343A1 (en) | Method for managing a database system | |
| US8762347B1 (en) | Method and apparatus for processing transactional file system operations to enable point in time consistent file data recreation | |
| US20070271431A1 (en) | Snapshot mechanism and method thereof | |
| JP2007183930A (en) | Maintaining consistency when mirroring data using different copy technologies | |
| CN114968966B (en) | Distributed metadata remote asynchronous replication method, device and equipment | |
| CN106407224A (en) | Method and device for file compaction in KV (Key-Value)-Store system | |
| CN115114370B (en) | Master-slave database synchronization method and device, electronic equipment and storage medium | |
| US9087115B1 (en) | Mirror resynchnronization of fixed page length tables for better repair time to high availability in databases | |
| CN112068992B (en) | Remote data replication method, storage device and storage system | |
| CN115380268A (en) | Moving window data deduplication in distributed storage | |
| CN109947730B (en) | Metadata recovery method, device, distributed file system and readable storage medium | |
| US7337194B2 (en) | Asynchronous remote mirroring techniques for databases |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210518 |