CN118394803B - Caching method, device, equipment and medium for separate database of calculation - Google Patents
Caching method, device, equipment and medium for separate database of calculation Download PDFInfo
- Publication number
- CN118394803B CN118394803B CN202410851471.7A CN202410851471A CN118394803B CN 118394803 B CN118394803 B CN 118394803B CN 202410851471 A CN202410851471 A CN 202410851471A CN 118394803 B CN118394803 B CN 118394803B
- Authority
- CN
- China
- Prior art keywords
- metadata
- page
- record
- key value
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a caching method, a caching device, caching equipment and caching media for a storage and calculation separation database, which relate to the technical field of databases and comprise the following steps: splicing the database ID and the metadata table ID into a first character string, taking the first character string as a prefix of each piece of data in the metadata table, splicing the record number of each piece of data behind the prefix, taking the record number as a key value of the piece of metadata, taking the content of the piece of metadata as a corresponding value, and obtaining a key value pair of the metadata table; extracting the value of the index field corresponding to each record in each index table of the metadata table, splicing the values into a second character string, splicing the character string behind the prefix to serve as a key value of the record, and taking the line number of the record as a corresponding value to obtain a key value pair of each index table of the metadata table. The invention adopts the key value to carry out metadata caching, thereby realizing data consistency high-efficiency caching under a memory calculation separation architecture and effectively improving the access efficiency and the system performance of metadata.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a caching method, a caching device, caching equipment and caching media for a storage and calculation separation database.
Background
With the rapid development of big data, cloud computing and micro-service architecture, traditional database systems face challenges in terms of performance, scalability, flexibility, and the like. A separate memory-store architecture (computer-Storage Separation Architecture) has developed that enables independent expansion and dynamic scheduling of resources by decoupling computing resources from memory resources. The architecture is widely applied to the fields of cloud database services, distributed database systems and the like.
Metadata (category) includes information such as tables, indexes, views, etc. in the database, and is a key component for the normal operation and query optimization of the database system. Conventional database systems typically store metadata in the same instance as data. Under the existence separation architecture, metadata is stored on a separate metadata server, and the computing node needs to access the metadata through Remote Procedure (RPC) or the like. Its management and access efficiency is critical to the overall performance of the system.
Some related technologies exist, such as: (1) Amazon Aurora: amazon Aurora is a relational database service of AWS, which implements a separate architecture for memory computation. Aurora ensures high availability and expandability of data by separating the storage nodes from the compute nodes and utilizing multiple copies and shared storage mechanisms. (2) Google Cloud Spanner: google Cloud Spanner is a Google's distributed database service with global consistency and high availability. Spanner realizes high-performance data management in a large-scale distributed environment through a multi-level storage and calculation architecture. (3) Memcached and Redis: these two are widely used distributed cache systems that increase data access speed and system scalability by caching data on multiple nodes. (4) MySQL InnoDB: innoDB is a MySQL storage engine, and the data access speed is improved by caching the common data page and index page through a Buffer Pool (Buffer Pool).
However, in the existing method, the metadata management mechanism of Aurora is realized by a closed source, and no clear metadata caching strategy exists; panner manage metadata using a specialized directory service and ensure consistency of metadata through a distributed protocol, which is not optimized enough for caching of metadata; memcached and Redis are typically used for application layer caching, but are not specific to database metadata. InnoDB's caching mechanism is tightly coupled to the database instance and is difficult to work efficiently with a memory-separate architecture.
Disclosure of Invention
In view of this, the embodiment of the application provides a caching method, a device, equipment and a medium for a separate database for calculation, and the embodiment of the application adopts a Key-Value Pair (KV Pair for short) mode to cache metadata, thereby realizing data consistency high-efficiency cache under a separate architecture for calculation and effectively improving the access efficiency and system performance of metadata.
The embodiment of the application provides the following technical scheme: a caching method of a storage separation database, comprising:
Splicing the database ID and the metadata table ID into a first character string, taking the first character string as a prefix of each piece of data content in the metadata table, splicing a record number of each piece of data content in the metadata table behind the prefix, taking the whole spliced first character string and record number as a key value of the piece of metadata, taking the data content of the piece of metadata as a value corresponding to the key value, and obtaining a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
Extracting the value of the index field corresponding to each record in each index table of the metadata table, splicing all the extracted values of the index fields of the record into a second character string according to a set sequence, splicing the second character string behind the prefix, taking the whole spliced first character string and second character string as key values of the record, taking the line number of the record as the value corresponding to the key values of the record, and obtaining the key value pair of each index table of the metadata table.
According to one embodiment of the application, the method further comprises:
Dividing all data contents in the metadata table into different pages according to different line numbers, and storing the data contents corresponding to key values of the line metadata in the pages;
Intercepting a set range of all records in the index table according to a starting record key value given by inquiry as a page, wherein a plurality of records are stored in the page;
Ordering the initial Key value of the cache page of each page in the metadata table and the index table to form an ordering index, and determining the corresponding page in the ordering index through a given Key value; wherein the page is used to characterize a base unit in which the cached data is stored.
According to one embodiment of the application, the method further comprises:
and constructing a lookup table by the physical file numbers of the metadata table and the index table, and indexing the cache data of the metadata table and the index table through the lookup table.
According to one embodiment of the present application, determining the corresponding page in the ordering index by a given Key value further comprises:
Sequentially scanning in the corresponding pages to inquire target data; if the target data is not queried, initiating an RPC request to a metadata server through a computing node, requesting metadata of a corresponding page, applying for a shared page, placing the obtained metadata of the corresponding page into the shared page, and inserting a starting key value of the shared page into the corresponding ordering index.
According to one embodiment of the application, a binary search is employed in the ordering index to determine the corresponding page by a given Key value.
According to one embodiment of the application, the method further comprises:
the metadata and the shared data of the database are put into a shared memory to be globally cached, so that all processes on the computing node share the cached data;
the metadata to be modified is locally cached before submission to isolate the modification of the metadata.
According to one embodiment of the application, the method further comprises:
When the computing node modifies the metadata, a metadata invalidation message is sent to a metadata server on a transaction boundary and a command boundary, and other computing nodes in the metadata server clear the corresponding metadata in the cache according to the metadata invalidation message, and acquire the latest metadata from the metadata server again.
The application also provides a caching device for storing and calculating the separation database, which comprises the following steps:
The first key value pair generation module is used for splicing the database ID and the metadata table ID into a first character string, taking the first character string as a prefix of each piece of data content in the metadata table, splicing the record number of each piece of data content in the metadata table behind the prefix, taking the whole spliced first character string and record number as a key value of the piece of metadata, taking the data content of the piece of metadata as a value corresponding to the key value, and obtaining a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
And the second key value pair generating module is used for extracting the values of the index fields corresponding to each record in each index table of the metadata table, splicing all the extracted values of the index fields of the record into a second character string according to a set sequence, splicing the second character string behind the prefix, taking the whole spliced first character string and second character string as key values of the record, and taking the line number of the record as value values corresponding to the key values of the record to obtain key value pairs of each index table of the metadata table.
The application also provides a computer device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the caching method for the separated database when executing the computer program.
The present application also provides a computer-readable storage medium storing a computer program for executing the above-described caching method for storing and computing a split database.
Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least: the embodiment of the invention adopts the key value to carry out metadata caching, thereby realizing data consistency high-efficiency caching under a memory-calculation separation architecture and effectively improving the access efficiency and the system performance of metadata. In addition, the embodiment of the invention keeps effective metadata in the shared memory as much as possible, adds the cache inquiry through a multi-level indexing mechanism, and ensures the consistency of the transaction through the invalid message. The metadata remote storage under the storage separation architecture is possible, and the access efficiency is ensured. The metadata is stored in a centralized way at a remote place, which is the basis for further separating data from metadata, so that the metadata can be shared by a plurality of computing systems, and the flexibility of the system is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a caching method of a separate database according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a cache organization of a separate database according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a caching apparatus for storing separate databases according to an embodiment of the present invention;
fig. 4 is a schematic structural view of the computer device of the present invention.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present application will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application with reference to specific examples. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
As shown in fig. 1, an embodiment of the present invention provides a caching method for storing and computing a separate database, including:
S101, splicing a database ID and a metadata table ID into a first character string, taking the first character string as a prefix of each piece of data content in the metadata table, splicing a record number of each piece of data content in the metadata table behind the prefix, taking the whole spliced first character string and record number as a key value of the piece of metadata, and taking the data content of the piece of metadata as a value corresponding to the key value to obtain a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
s102, extracting values of index fields corresponding to each record in each index table of the metadata table, splicing the values of all index fields of the extracted record into a second character string according to a set sequence, splicing the second character string behind the prefix, taking the whole of the spliced first character string and second character string as key values of the record, taking a line number of the record as a value corresponding to the key values of the record, and obtaining key value pairs of each index table of the metadata table.
The database architecture of Storage-Compute Separation is a design model that separates the Storage and computation functions of data to improve the scalability and flexibility of the database. Taking the conventional database PostgreSQL as an example, metadata (commonly referred to as category) of the database is also stored in the database in the form of a data Table (Table), and the metadata of each database may be different. The embodiment of the invention provides a mode of mapping different metadata tables into a unified KV space, namely a caching mechanism of KV pair to realize indiscriminate data storage, thereby improving the access speed of metadata and the expansibility of a system.
The embodiment of the invention definitely discloses an independently deployed metadata server and a caching mechanism based on KV pair, and the method is as follows when in specific implementation:
(1) The database ID and the metadata table ID are spliced into a character string which is used as the prefix of the data content of the metadata table. To ensure the legitimacy of the key, coding techniques are used to convert the mathematical and special characters involved into readable strings.
(2) For the metadata table, the record number of the data is spelled behind the prefix as the key value of the piece of metadata. The recorded content is stored as a value as the original database.
(3) For each index table of metadata, the values of all index fields of the corresponding record are spelled into a character string, and the character string is attached to the back of the prefix to be used as a key value. Stored in the value is the location of the corresponding record, i.e., the line number.
Since the index record is repeatedly modified, the key value of the same index is remapped to a new location. Thus, an index record may have multiple versions, with a number representing different modification opportunities for querying. By adopting the method, each piece of metadata is mapped to a global unified address space in a unique mode, and corresponding contents can be accessed by specifying key values.
In a specific implementation, the record number of the data content is used for characterizing a unique identifier of each piece of data content in the metadata table, where the unique identifier may be: (1) Main Key (PRIMARY KEY): in a relational database, each table will have a primary key, which is a field or combination of fields, that uniquely identifies each row of data in the table. (2) record ID: in some systems, there may be an automatically generated record ID field that tracks the insertion order of each record or as a unique identification for each record. (3) Sequence Number: in some database designs, a serial number may be used to identify a record, which is typically automatically incremented in the order in which the records are inserted. (4) line Number (Row Number): in some queries or reports, each record may be assigned a line number to facilitate referencing or ordering. (5) unique identifier (Unique Identifier, UID): without a conventional primary key, a special field, such as UUID (Universally Unique Identifier), may be used as the unique identifier for the record.
In one embodiment, the method further comprises:
Dividing all data contents in the metadata table into different pages according to different line numbers, and storing the data contents corresponding to key values of the line metadata in the pages;
Intercepting a set range of all records in the index table according to a starting record key value given by inquiry as a page, wherein a plurality of records are stored in the page;
Ordering the initial Key value of the cache page of each page in the metadata table and the index table to form an ordering index, and determining the corresponding page in the ordering index through a given Key value; wherein the page is used to characterize a base unit in which the cached data is stored.
In the specific implementation, as shown in fig. 2. Page is the basic unit for storing cached data, storing a range of key-specific records. For the original table of the metadata table, it is split into different pages according to the line number. Pages of the index table are formed by intercepting a range based on querying a given starting Key. An index-cached page contains several records that record the locations in the original table and the modified sequence numbers. For data in the original table, there may also be multiple versions, which differ in line number.
For each table in the cache, tableIndex (index table) maintains one ordering of all cache page start Key values. Given a key value, a binary search may be used by TableIndex to determine if a page has been dropped and then scanned sequentially within the page to finally determine if a hit.
In one embodiment, the method further comprises:
and constructing a lookup table by the physical file numbers of the metadata table and the index table, and indexing the cache data of the metadata table and the index table through the lookup table.
In practice, the cached data of a table is indexed by its physical file number (relnode), whether it be the original table or the index table. Taking the database PostgreSQL as an example, the database number plus the number of the table space plus relnode data uniquely identifying a table, and taking the combination of the database number and the table space as key values, tableIndex of the corresponding table can be found so as to complete all queries.
In one embodiment, determining the corresponding page in the ordering index by a given Key value further comprises: sequentially scanning in the corresponding pages to inquire target data; if the target data is not queried, initiating an RPC request to a metadata server through a computing node, requesting metadata of a corresponding page, applying for a shared page, placing the obtained metadata of the corresponding page into the shared page, and inserting a starting key value of the shared page into an index table of the corresponding ordering index.
In one embodiment, the method further comprises: the metadata and the shared data of the database are put into a shared memory to be globally cached, so that all processes on the computing node share the cached data; the metadata to be modified is locally cached before submission to isolate the modification of the metadata.
In practice, by default, the computing node uses global caches. The globally cached data is placed in a shared memory, which is shared by all processes on the node.
The database will modify the metadata during execution of the transaction, which uncommitted metadata is not allowed to be seen by other processes. Therefore, the data cannot directly modify the data in the metadata in the global cache, but an internal memory cache structure which is the same as the global cache structure is additionally used, which is equivalent to maintaining the metadata changed in the local cache.
Wherein which cache is used operates this granularity at the table level. For example, a transaction may access metadata table A and modify metadata table B. The local cache is used when B is accessed and the global cache is still used when a is accessed. The method keeps the effectiveness of the original cache data to the greatest extent, and improves the utilization rate of the cache.
In particular implementations, commit operations may occur when a transaction completes and prepares to permanently save changes to the database. At this point, modifications in the local cache will be incorporated into the global cache and database. If the transaction fails or needs to be withdrawn, a rollback operation may be performed, withdrawing all modifications in the transaction, and returning to the state before the transaction began.
In implementations, the uncommitted data may be written locally only, and written to the metadata server at commit time. In addition, when the memory space is insufficient, part of the cache data can be written to the disk.
In one embodiment, the method further comprises: when the computing node modifies the metadata, a metadata invalidation message is sent to a metadata server on a transaction boundary and a command boundary, and other computing nodes in the metadata server clear the corresponding metadata in the cache according to the metadata invalidation message, and acquire the latest metadata from the metadata server again.
In database operations, transaction boundaries define the beginning and ending of a transaction. When a compute node needs to modify metadata, the modification must be done in the context of a transaction. Command boundaries refer to the beginning and end of a single database command.
In particular, when the metadata is modified and is no longer valid, the cached metadata is emptied, so that data errors caused by using the expired data are avoided.
The separate database is an architecture that shares metadata across processes, requiring invalidation messages to be sent to the metadata server on transaction boundaries and command (in the PostgreSQL database, commands are operations that split the visibility of locally modified metadata within a transaction, if a command sequence number is not added, modified metadata in the present command is also not visible to the present process) when a compute node issues a modify operation on metadata. Other processes receive these invalidation messages, thereby eliminating the cached data. When the data needs to be read next time, the required metadata is re-read from metaserver.
For simplicity, locally modified metadata will be written directly to the metadata server. And when the corresponding metadata invalidation message is processed locally, the local metadata cache is cleared, and the latest metadata is acquired from the metadata server again.
Taking the database PostgreSQL as an example, for most metadata tables, metadata modifications will result in a new record, and the computing node identifies whether to accept the record by virtue of the transaction id and command id carried on the record. This approach is simple and straightforward and can be further optimized.
With such a cache invalidation mechanism, the separate database architecture can maintain cache consistency across multiple compute nodes even in cases where metadata is frequently modified.
The embodiment of the invention mainly comprises the following steps: (1) Mapping a traditional database, such as a catalog of PostgreSQL, to kv storage; (2) caching pages and indexes in memory; 3) Adopting global and local two-layer caches to respectively process the reading and writing of data; 4) Invalidation of the cached data is accomplished using an invalidation messaging mechanism. The method retains effective metadata in the shared memory as much as possible, adds the cache inquiry through a multi-level indexing mechanism, and ensures the consistency of the transaction through the invalid message. The metadata can be stored remotely under the architecture of memory separation, and the access efficiency is ensured. The metadata is stored in a centralized way at a remote place, which is the basis for further separating data from metadata, so that the metadata can be shared by a plurality of computing systems, and the flexibility of the system is improved.
Although PosgreSQL is described as an example in this method, the method is not limited to PosgreSQL database, but is also not limited to relational databases. The method is a mechanism for realizing concurrency control, but the method is not limited to multi-version concurrency control.
As shown in fig. 3, the present application further provides a caching apparatus 200 for storing a separate database, including:
a first key value pair generating module 201, configured to splice a database ID and a metadata table ID into a first string, respectively splice a record number of each piece of data content in the metadata table behind the prefix with the first string and the record number after being spliced as a key value of the piece of metadata, and use the data content of the piece of metadata as a value corresponding to the key value to obtain a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
And a second key value pair generating module 202, configured to extract values of index fields corresponding to each record in each index table of the metadata table, splice values of all index fields of the extracted record into a second string according to a set order, splice the second string to the back of the prefix, use the spliced first string and second string as key values of the record as a whole, and use a line number of the record as a value corresponding to the key value of the record to obtain a key value pair of each index table of the metadata table.
When the device is implemented, the device also comprises a page storage module, wherein the page storage module is used for respectively dividing all data contents in the metadata table into different pages according to different row numbers, and the page stores the data contents corresponding to key values of the row metadata; intercepting a set range of all records in the index table according to a starting record key value given by inquiry as a page, wherein a plurality of records are stored in the page; ordering the initial Key value of the cache page of each page in the metadata table and the index table to form an ordering index, and determining the corresponding page in the ordering index through a given Key value; wherein the page is used to characterize a base unit in which the cached data is stored.
The page storage module is further configured to form a lookup table from the physical file numbers of the metadata table and the index table, and index the cache data of the metadata table and the index table through the lookup table.
The page storage module is also used for sequentially scanning in the corresponding pages and inquiring target data; if the target data is not queried, initiating an RPC request to a metadata server through a computing node, requesting metadata of a corresponding page, applying for a shared page, placing the obtained metadata of the corresponding page into the shared page, and inserting a starting key value of the shared page into the corresponding ordering index.
When the device is implemented, the device also comprises a two-level cache module, wherein the two-level cache module is used for putting metadata and shared data of the database into a shared memory for global cache so that all processes on the computing node share the cache data; the metadata to be modified is locally cached before submission to isolate the modification of the metadata.
When the device is implemented, the device also comprises a cache invalidation control module, wherein the cache invalidation control module is used for sending a metadata invalidation message to a metadata server on a transaction boundary and a command boundary when the metadata is modified by a computing node, and other computing nodes in the metadata server clear the corresponding metadata in the cache according to the metadata invalidation message and acquire the latest metadata from the metadata server again.
The device is suitable for a storage and calculation separation architecture mode, and can locally cache the metadata stored in a centralized way on the side of a computing cluster. The method can efficiently realize memory caching of all metadata and correctly realize consistency of the metadata.
In one embodiment, a computer device is provided, as shown in fig. 4, including a memory 301, a processor 302, and a computer program stored on the memory and capable of running on the processor, where the processor implements the above-mentioned caching method for storing and separating databases when executing the computer program.
In particular, the computer device may be a computer terminal, a server or similar computing means.
In the present embodiment, a computer-readable storage medium storing a computer program for executing the above-described caching method of storing a split database is provided.
In particular, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable storage media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present application should be included in the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (8)
1. A method for caching a separate database, comprising:
Splicing the database ID and the metadata table ID into a first character string, taking the first character string as a prefix of each piece of data content in the metadata table, splicing a record number of each piece of data content in the metadata table behind the prefix, taking the whole spliced first character string and record number as a key value of the piece of metadata, taking the data content of the piece of metadata as a value corresponding to the key value, and obtaining a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
Extracting the value of an index field corresponding to each record in each index table of the metadata table, splicing all the extracted values of the index fields of the record into a second character string according to a set sequence, splicing the second character string behind the prefix, taking the spliced first character string and second character string as key values of the record as a whole, and taking the line number of the record as a value corresponding to the key values of the record to obtain key value pairs of each index table of the metadata table;
The method further comprises the steps of:
Dividing all data contents in the metadata table into different pages according to different line numbers, and storing the data contents corresponding to key values of the line metadata in the pages;
Intercepting a set range of all records in the index table according to a starting record key value given by inquiry as a page, wherein a plurality of records are stored in the page;
Ordering the initial Key value of the cache page of each page in the metadata table and the index table to form an ordering index, and determining the corresponding page in the ordering index through a given Key value; wherein the page is used for representing a basic unit for storing cache data;
determining, by a given Key value, a corresponding page in the ordering index, further comprising:
Sequentially scanning in the corresponding pages to inquire target data; if the target data is not queried, initiating an RPC request to a metadata server through a computing node, requesting metadata of a corresponding page, applying for a shared page, placing the obtained metadata of the corresponding page into the shared page, and inserting a starting key value of the shared page into the corresponding ordering index.
2. The method of caching a split database according to claim 1, further comprising:
and constructing a lookup table by the physical file numbers of the metadata table and the index table, and indexing the cache data of the metadata table and the index table through the lookup table.
3. The method of claim 1, wherein given Key values, binary search is used to determine corresponding pages in the ordering index.
4. The method of caching a split database according to claim 1, further comprising:
the metadata and the shared data of the database are put into a shared memory to be globally cached, so that all processes on the computing node share the cached data;
the metadata to be modified is locally cached before submission to isolate the modification of the metadata.
5. The method of caching a split database according to claim 1, further comprising:
When the computing node modifies the metadata, a metadata invalidation message is sent to a metadata server on a transaction boundary and a command boundary, and other computing nodes in the metadata server clear the corresponding metadata in the cache according to the metadata invalidation message, and acquire the latest metadata from the metadata server again.
6. A caching apparatus for storing a split database, comprising:
The first key value pair generation module is used for splicing the database ID and the metadata table ID into a first character string, taking the first character string as a prefix of each piece of data content in the metadata table, splicing the record number of each piece of data content in the metadata table behind the prefix, taking the whole spliced first character string and record number as a key value of the piece of metadata, taking the data content of the piece of metadata as a value corresponding to the key value, and obtaining a key value pair of the metadata table; wherein the record number of the data content is used for representing the unique identification of each piece of data content in the metadata table;
The second key value pair generating module is used for extracting the values of the index fields corresponding to each record in each index table of the metadata table, splicing all the extracted values of the index fields of the record into a second character string according to a set sequence, splicing the second character string behind the prefix, taking the whole spliced first character string and second character string as key values of the record, taking the line number of the record as value values corresponding to the key values of the record, and obtaining key value pairs of each index table of the metadata table;
The page storage module is used for respectively dividing all data contents in the metadata table into different pages according to different row numbers, and storing the data contents corresponding to key values of the row metadata in the pages; intercepting a set range of all records in the index table according to a starting record key value given by inquiry as a page, wherein a plurality of records are stored in the page; ordering the initial Key value of the cache page of each page in the metadata table and the index table to form an ordering index, and determining the corresponding page in the ordering index through a given Key value; wherein the page is used for representing a basic unit for storing cache data;
The page storage module is also used for sequentially scanning in the corresponding pages and inquiring target data; if the target data is not queried, initiating an RPC request to a metadata server through a computing node, requesting metadata of a corresponding page, applying for a shared page, placing the obtained metadata of the corresponding page into the shared page, and inserting a starting key value of the shared page into the corresponding ordering index.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the caching method of the split-store database of any one of claims 1 to 5 when the computer program is executed by the processor.
8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that performs the caching method of storing the split database according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410851471.7A CN118394803B (en) | 2024-06-28 | 2024-06-28 | Caching method, device, equipment and medium for separate database of calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410851471.7A CN118394803B (en) | 2024-06-28 | 2024-06-28 | Caching method, device, equipment and medium for separate database of calculation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118394803A CN118394803A (en) | 2024-07-26 |
| CN118394803B true CN118394803B (en) | 2024-09-17 |
Family
ID=91989366
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410851471.7A Active CN118394803B (en) | 2024-06-28 | 2024-06-28 | Caching method, device, equipment and medium for separate database of calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118394803B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114547031A (en) * | 2022-02-21 | 2022-05-27 | 中国银联股份有限公司 | Key value pair database operation method and device and computer readable storage medium |
| CN115114374A (en) * | 2022-06-27 | 2022-09-27 | 腾讯科技（深圳）有限公司 | Transaction execution method and device, computing equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6457021B1 (en) * | 1998-08-18 | 2002-09-24 | Microsoft Corporation | In-memory database system |
| CN118035255A (en) * | 2024-01-23 | 2024-05-14 | 天津大学 | Non-intrusive log push-down method for storage and computing separation database |
-
2024
- 2024-06-28 CN CN202410851471.7A patent/CN118394803B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114547031A (en) * | 2022-02-21 | 2022-05-27 | 中国银联股份有限公司 | Key value pair database operation method and device and computer readable storage medium |
| CN115114374A (en) * | 2022-06-27 | 2022-09-27 | 腾讯科技（深圳）有限公司 | Transaction execution method and device, computing equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118394803A (en) | 2024-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US5519855A (en) | Summary catalogs | |
| US8386494B2 (en) | Providing data structures for determining whether keys of an index are present in a storage system | |
| US9501550B2 (en) | OLAP query processing method oriented to database and HADOOP hybrid platform | |
| US9256607B2 (en) | Efficient file access in a large repository using a two-level cache | |
| CN105677826A (en) | Resource management method for massive unstructured data | |
| WO2020125630A1 (en) | File reading | |
| CN109213760B (en) | High-load business storage and retrieval method for non-relational data storage | |
| CN113448964A (en) | Hybrid storage method and device based on graph-KV | |
| CN116049306B (en) | Data synchronization methods, devices, electronic devices, and readable storage media | |
| CN111309266A (en) | Distributed storage metadata system log optimization system and method based on ceph | |
| CN111611246B (en) | Method and system for optimizing index performance of B+ tree based on persistent memory | |
| US20180276267A1 (en) | Methods and system for efficiently performing eventual and transactional edits on distributed metadata in an object storage system | |
| US20080250017A1 (en) | System and method for aiding file searching and file serving by indexing historical filenames and locations | |
| CN120215825A (en) | A B+ tree index construction method for dynamic placement of hot and cold data based on ZNS SSD | |
| CN102169497B (en) | Method and device for managing metadata through bitmaps | |
| CN114691307B (en) | Transaction processing method and computer system | |
| CN118394803B (en) | Caching method, device, equipment and medium for separate database of calculation | |
| CN115203211A (en) | A method and system for generating a unique hash sequence number | |
| WO2025153909A2 (en) | Verification structure updating method, data verification method, device, medium and product | |
| CN116756253B (en) | Data storage and query methods, devices, equipment and media of relational database | |
| Cheng et al. | Optimizing small file storage process of the HDFS which based on the indexing mechanism | |
| CN118519998B (en) | Distributed database local temporary table management method, device, equipment and medium | |
| CN111858516B (en) | Data processing method and device | |
| CN114218277A (en) | Efficient query method and device for relational database | |
| CN1492363A (en) | Data storage and searching method of embedded system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |