CN115221131A - High-speed data reading and writing method and device for time sequence database - Google Patents

Abstract

The invention relates to the technical field of distributed databases, in particular to a high-speed data reading and writing method of a time sequence database, which comprises the following steps: s1, a memory management cache block; s2, managing a pre-written log WAL; s3, managing metadata; and S4, indexing information. Compared with the prior art, the method and the system reduce the hardware configuration requirement of the side-end system of the time sequence database, reduce the hardware cost and improve the product competitiveness. The memory management structure is simplified, and the use of the time sequence data to the CPU and the disk of the system resource is more balanced. The time sequence database realizes the additional writing of the index file and has high processing efficiency. When the tables partitioned by the historical time are inquired and analyzed, the table indexes can be quickly positioned without loading data of irrelevant tables.

Description

High-speed data reading and writing method and device for time series database

technical field

The invention relates to the technical field of distributed databases, and in particular provides a high-speed data reading and writing method and device for a time series database.

Background technique

In order to give full play to the ability of the smart microgrid to manage distributed power sources, storage devices and related loads, and effectively improve the security, stability and economic operation level of the microgrid, the smart microgrid energy management system (Microgrid Energy Management System, MEMS) has become indispensable. effective means. Different from the energy management system of the traditional power system, MEMS needs to fully combine the characteristics of the microgrid itself, through the real-time monitoring of the internal data of the smart microgrid and the timely interaction of external information, to formulate a reasonable economic operation plan, and carry out the intelligent microgrid equipment. Effective management and control.

MEMS needs to comprehensively consider various operating modes of the smart microgrid, equipment operating conditions and external related information to coordinate optimal control and management of distributed power sources, energy storage devices, and loads. MEMS provides functions such as data acquisition, analysis, system monitoring and alarming, energy control scheduling, and cloud-side collaboration with energy routers as the core at the edge.

The amount of data generated by smart microgrids is thousands or even tens of thousands of times more than traditional relational data application scenarios (such as banks and transactions), and is collected in real time, with high frequency and high density, and the data model may change at any time. It is difficult to store, query and analyze these data with traditional databases. Time series database is a specialized database for storing and managing time series data. It is good at dealing with scenarios such as more writes and fewer reads, no transactions, separation of hot and cold, high concurrent writes, continuous writing of massive data, and aggregation analysis based on time intervals.

The data writing process of the current mainstream time series database (such as TDEngine, which has been open sourced) is basically shown in Figure 5. In order to ensure data security, write ahead log (Write Ahead Log, WAL) first. Then write the data of a group of tables into a large cache (Cache). When the data in the cache reaches a certain size or the cache creation exceeds a certain time, the system will pull up the disk drop thread to drop the data to the disk, clear the cache, build an index, and delete the WAL. In order to improve the compression ratio and speed up the calculation and analysis, the data files are generally stored in the column storage format. Each column storage block contains multiple rows of records. In order to prevent the number of rows in the column storage block from being too small, a merge thread will merge the small column storage block into a large column storage block that meets the requirements of the number of rows.

On low-profile board hardware in energy routers, the main problems with this architecture are:

Every time a large cache (default 32MB) is filled, it will be dropped. The disk is free when data is written to the cache. When the disk is placed, due to the large cache, data sorting, compression, column storage format construction, etc., it is also easy to cause a CPU computing spike. This is not conducive to the balanced use of the entire hardware resources.

Since the cache contains data from multiple tables, when the cache is full, the number of written records in some tables does not meet the minimum number of rows required for a column storage block. At this time, many small column storage blocks will be generated when the disk is dropped. Small column storage blocks are not conducive to data compression, analysis and management. In TDEngine, the column storage blocks that do not meet the number of rows are written into a separate last file, the cache is merged with the last file each time the disk is placed, and the blocks that meet the number of rows are written into another data data file. Although this reduces the number of column storage blocks, one more merge thread will be used, and each merge will cause the last file to be rewritten, resulting in system performance consumption.

Although the data data file is written additionally, its index needs to be rebuilt every time the disk is placed, and the index file needs to be overwritten and rewritten every time the disk is placed.

When reading data in historical time partitions, if you only read the index of one table, you need to load the indexes of all tables into memory. Generate redundant memory footprint and performance waste.

SUMMARY OF THE INVENTION

The invention provides a high-speed data reading and writing method for a time series database with strong practicability, aiming at the shortcomings of the above-mentioned prior art.

A further technical task of the present invention is to provide a high-speed data reading and writing device for time series database with reasonable design, safety and application.

The technical scheme adopted by the present invention to solve its technical problems is:

A high-speed data reading and writing method for a time series database, comprising the following steps:

S1, memory management cache block;

S2, write-ahead log WAL management;

S3, metadata management;

S4. Index information.

Further, in step S1, two buffer blocks are allocated in advance when the system is started or when the table is created, one buffer block is called Mutable Buffer, and the other buffer block is called Immutable Buffer, and the Mutable Buffer is used to receive writes. Enter new data, and copy the append directly to the Mutable Buffer in the form of row storage.

Further, if the Mutable Buffer is full, it will be converted into the Immutable Buffer and placed in the waiting queue table, waiting for subsequent pre-computation, compression, disk placement and update, among which, the pre-computation and compression of each table Do not interfere with each other.

Further, the size of each Mutable Buffer is determined according to the number of attributes and data types of the table, and there are two principles for setting the size of each Mutable Buffer, the first principle is that the first principle is that each The initial size of the Mutable Buffer must exceed a certain threshold; the second principle is that each Mutable Buffer can hold a certain amount of data.

Further, in step S2, the generation time of the Mutable Buffer of each table is recorded, and if the minimum number of bars to be dropped has not been reached beyond the threshold, it is necessary to forcibly drop the disc;

When the system restarts to restore data from the WAL, the index records the column storage block information of each table, and reads the time point when the table was last dropped from the column storage block information, and writes after the time point when restoring the WAL. Only the data needs to be restored to the memory, and the previous data has been placed on the disk and can be discarded.

Further, in step S3, the metadata will maintain the unique identifier UUID of each table, which is stored in order in the metadata inside the storage engine, the storage order is fixed, and the order corresponding to each table is defined. It is id, the data type is an integer, and the id is fixed for each table. In the subsequent index file storage, the index storage order of the table is consistent with the order of the id, which is convenient for quickly locating the index of the table being queried in the query. Location.

Further, in step S4, the index information includes index information in the memory and in the file, wherein the index information in the memory maintains all index information in the current file for each table in the memory.

Further, the index information in the file includes the index format of the current time partition and the index file format of the historical time partition. The index format of the current time partition is updated by appending at the end of the file, and the system does not need to read during normal operation. For the current index file, the entire index file should be read into the memory only when the system fails and needs to be restarted;

Each index record is as follows id, start, end and data offset. id is the unique identifier of the table in the storage, numbered in sequence; start is the minimum timestamp in the data block; end is the maximum timestamp in the data block; data offset is the offset of the data block in the current raw data file.

Further, in the historical time partition index file format, the current file is no longer written with data. When it is converted to a historical partition file, the index file should be cleared, and then the index information in the memory will be written into the index file in sequence according to the table. The index information of a table is stored sequentially.

A high-speed data reading and writing device for time series database, comprising: at least one memory and at least one processor;

the at least one memory for storing a machine-readable program;

The at least one processor is configured to invoke the machine-readable program to execute a high-speed data reading and writing method for a time series database.

Compared with the prior art, the high-speed data reading and writing method and device of a time series database of the present invention have the following outstanding beneficial effects:

The invention reduces the hardware configuration requirements of the time series database on the side-end system, reduces the hardware cost, and improves the product competitiveness. Simplify the memory management structure to make time series data use more balanced CPU and disk usage of system resources. The time series database implements additional write index files for efficient processing. When querying and analyzing historical time-partitioned tables, you can quickly locate table indexes without loading data from unrelated tables.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic diagram of a table cache block structure in a high-speed data reading and writing method of a time series database;

Accompanying drawing 2 is a kind of schematic diagram of metadata format in time series database high-speed data reading and writing method;

3 is a schematic diagram of the structure of the current partition index file in a time series database high-speed data reading and writing method;

Accompanying drawing 4 is a kind of schematic drawing of historical partition index file structure in a time series database high-speed data reading and writing method;

FIG. 5 is a schematic diagram of a flow chart of data writing in a time series database in the prior art in a high-speed data reading and writing method for a time series database.

Detailed ways

In order to make those skilled in the art better understand the solution of the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

A preferred embodiment is given below:

As shown in Figure 1-4, a high-speed data reading and writing method for a time series database in this embodiment includes the following steps:

S1, memory management cache block;

Each table maintains its own two cache blocks, pre-allocated at system startup, or allocated when the table is created. One of the cache blocks is called Mutable Buffer (mutable cache block), and the other is called Immutable Buffer (immutable cache block). Mutable Buffer is used to receive and write new data, and copy the appending directly to MutableBuffer in the form of row storage.

If the Mutable Buffer is full, it will be converted to Immutable Buffer and placed in the waiting queue table, waiting for subsequent operations such as pre-computation, compression, disk placement, and index update. Therefore, multi-threading can be used for concurrent processing according to resource conditions.

The size of each Mutable Buffer is determined according to the number of attributes of the table and its data type. The size parameter of each Mutable Buffer can be configured, such as 64K. There are two principles for setting:

(1) In order to prevent frequent memory application, the initialization size of each Mutable Buffer must exceed a certain threshold;

(2) Each block must be able to hold at least a certain amount of data.

Since the buffer of each table reaches a certain number of rows before it can be placed on the disk, it will not generate small column storage blocks like TDEngine, and it does not need to be adjusted by the merge thread.

S2, write-ahead log WAL management;

Since the memory is managed according to the Buffer of the table, WAL appends records according to the writing time. Suppose the buffer of table A is full and can be dropped, but the writing frequency of table B is slow and has not reached the condition of dropping. At this time, the data of table A and table B are stored in WAL, and WAL cannot be deleted directly. Because the data of table B is still in the memory, if it is deleted, the data in the memory of table B will be lost during abnormal restart. So when to clean up the WAL is something to consider.

The solution is to record the generation time of the Mutable Buffer of each table. If the threshold (default 3 hours) is exceeded and the minimum number of records has not been reached, the disk must be forced to be placed. In this way, although a small column storage block will be generated, because the threshold interval is relatively long, there will not be many such column storage blocks. In addition, the maximum retention time of the WAL is also limited to not exceed the threshold setting time to prevent the WAL from increasing indefinitely.

When the system restarts to restore data from the WAL, the index records the column storage block information of each table, and can read the time point when the table was last placed on the disk from the column storage block information, and write after the time point when restoring the WAL. Only the entered data needs to be restored to the memory, and the previous data has been placed on the disk and can be discarded.

S3, metadata management;

The metadata will maintain the unique identifier UUID of each table, which is stored in order in the metadata inside the storage engine, and the storage order is fixed. The order corresponding to each table is defined as id, and its data type is Integer, which is 0, 1, 2, ..., and this id is fixed for each table. In the subsequent index file storage, the index storage order of the table is consistent with the order of the id. It is convenient to quickly locate the index position of the table being queried in a query. The storage format of each table is shown in Figure 2:

allocated, uuid, attributes

Among them, allocated is bool type, indicating whether the id has been allocated, true is allocated, false is unallocated/deleted. Deleted ids can be recycled and can be assigned to deleted ids when a new table is created.

S4, index information;

(1) Update of index information, including index information in memory and in files.

memory index

Each table maintains all index information in the current file in memory. The main information is as follows:

index count

start time, end time, data offset 1

start time, end time, data offset 2

start time, end time, data offset 3

......

Where data offset points to the offset of the corresponding data block in the original data file.

(2) The index file format of the current time partition

In order to improve the update efficiency of the index file, it will also be updated by appending at the end of the file. Since a copy of the index information is still stored in the memory, the current index file does not need to be read during the normal operation of the system. Only when the system fails When restarting is required, the entire index file must be read into memory.

Each index record is as follows: id, start, end, data offset

id is the unique identifier of the table in the storage, numbered in sequence; start is the minimum timestamp in the data block; end is the maximum timestamp in the data block; data offset is the offset of the data block in the current original data file. As shown in Figure 3, the current index file is appended sequentially.

(3) Index file format of historical time partition

The current file no longer writes data. When it is converted to a historical partition file, the index file should be cleared, and then the index information in the memory will be written into the index file in sequence according to the table, so that the index information of the same table is stored sequentially:

Header

Index header 0 (id=0)

Index header 1 (id=1)

Index header 2 (id=2)

......

index block 0 (id=0)

index block 1 (id=1)

Index fast 2 (id=2)

......

Among them, the index header is roughly defined as follows: index count, index offset

The size of each index header is fixed, and the offset of its index header in the index file can be calculated according to the id of the table. The index count in the index header records several pieces of index information in the index block corresponding to the table, and the indexoffset is the offset of the index block in the index file.

The size of each index block is not fixed and depends on how many pieces of index information are in the index block, that is, the index count in the index header. An index block is roughly defined as follows:

index info 0

Index information 1

Index information 2

......

Among them, each index information is roughly defined as follows: start, end, data offset, length

start and end are the start time and end time of the corresponding data block, respectively, and data offset is the offset of the data block in the original data file.

The mapping format of the historical index file and the original data file is shown in Figure 4.

Based on the above method, a high-speed data reading and writing device for a time series database in this embodiment includes: at least one memory and at least one processor;

the at least one memory for storing a machine-readable program;

The at least one processor is configured to invoke the machine-readable program to execute a high-speed data reading and writing method for a time series database.

The above-mentioned specific embodiments are only specific cases of the present invention, and the scope of patent protection of the present invention includes but is not limited to the above-mentioned specific embodiments, any high-speed data reading and writing method and device for a time series database in accordance with the present invention. Any appropriate changes or substitutions made by those of ordinary skill in the technical field shall fall into the scope of patent protection of the present invention.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, and substitutions can be made in these embodiments without departing from the principle and spirit of the invention and modifications, the scope of the present invention is defined by the appended claims and their equivalents.

Claims (10)

1. A high-speed data reading and writing method for a time sequence database is characterized by comprising the following steps:

s1, a memory management cache block;

s2, managing a pre-written log WAL;

s3, managing metadata;

and S4, indexing information.

2. The high-speed data reading and writing method for the time series database according to claim 1, wherein in step S1, two cache blocks are pre-allocated at system startup or allocated at table creation time, one cache block is called a executable Buffer, the other cache block is called an Immutable Buffer, and the executable Buffer is used for receiving and writing new data and directly copying an additional Buffer into the executable Buffer in a line memory manner.

3. The high-speed data reading and writing method for the time series database according to claim 2, characterized in that if the table Buffer is full, the table Buffer is converted into the Imutable Buffer and is placed into a waiting queue list to wait for subsequent pre-calculation, compression, disk dropping and updating, wherein the pre-calculation and compression of each table are not interfered with each other.

4. The high-speed data reading and writing method for the time sequence database according to claim 3, wherein the size of each of the table buffers is determined according to the attribute number and the data type of the table, two principles are set for the size of each of the table buffers, and the first principle is that the initialized size of each of the table buffers exceeds a certain threshold; the second principle is that each volatile Buffer can hold a certain amount of data.

5. The high-speed data reading and writing method for the time series database according to claim 4, characterized in that in step S2, the generating time of the table Buffer of each table is recorded, and if the minimum number of the dropped disks is not reached after exceeding a threshold, the dropped disks need to be forced;

when the system restarts to recover data from the WAL, the column block information of each table falling off the disk is recorded in the index, the time point of the last falling off of the table is read from the column block information, the data written after the time point when the WAL is recovered needs to be recovered to the memory, and the previous data falling off the disk can be discarded.

6. The method as claimed in claim 4, wherein in step S3, the metadata maintains a unique identifier UUID of each table, the UUID is stored in the metadata in the storage engine in sequence, the storage sequence is fixed, the sequence corresponding to each table is defined as id, the data type is an integer, the id is fixed for each table, and in the following index file storage, the index storage sequence of the table is consistent with the sequence of the id, so as to facilitate fast locating the index position of the table being queried in the query.

7. The method according to claim 6, wherein in step S4, the index information includes index information in the memory and index information in the file, and wherein the index information in the memory maintains all index information in the current file in the memory for each table.

8. The high-speed data reading-writing method for the time sequence database according to claim 6, wherein the index information in the file comprises an index format of a current time partition and an index file format of a historical time partition, the index format of the current time partition is updated in a way of adding the tail of the file, the system does not need to read the current index file in the normal operation process, and the whole index file is read into the memory only when the system is in failure and needs to be restarted;

each index records the following id, start, end and data offset, wherein the id is a unique identifier of the table in the storage and is numbered in sequence; start is the minimum timestamp in the data block; end is the maximum timestamp in the data block; the data offset is the offset of the data block in the current original data file.

9. The method as claimed in claim 8, wherein in the historical time partition index file format, when the current file is not written with data any more and is converted into the historical partition file, the index file is emptied, and then the index information in the memory is sequentially written into the index file according to the tables, and the index information of the same table is sequentially stored.

10. A high-speed data read-write device of a time sequence database is characterized by comprising: at least one memory and at least one processor;

the at least one memory to store a machine readable program;

the at least one processor, configured to invoke the machine readable program to perform the method of any of claims 1 to 9.

Images