CN119806409A - Data storage method, device, terminal equipment and computer readable storage medium - Google Patents
Data storage method, device, terminal equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN119806409A CN119806409A CN202411874704.1A CN202411874704A CN119806409A CN 119806409 A CN119806409 A CN 119806409A CN 202411874704 A CN202411874704 A CN 202411874704A CN 119806409 A CN119806409 A CN 119806409A
- Authority
- CN
- China
- Prior art keywords
- data
- state data
- storage
- units
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请适用于区块链的数据存储技术领域，提供了一种数据存储方法、装置、终端设备及计算机可读存储介质，包括：获取区块链中多个第一单元的第一状态数据；其中，每个第一单元为区块链中发生数据更新的区块单元；将多个第一状态数据进行去重处理，得到多个第二状态数据；将多个第二状态数据批量存储至存储区域中对应的存储位置。上述方法可以减少存储空间的浪费，提高存储资源的利用率。
The present application is applicable to the data storage technology field of blockchain, and provides a data storage method, device, terminal device and computer-readable storage medium, including: obtaining first state data of multiple first units in the blockchain; wherein each first unit is a block unit where data update occurs in the blockchain; performing deduplication processing on the multiple first state data to obtain multiple second state data; and storing the multiple second state data in batches to corresponding storage locations in the storage area. The above method can reduce the waste of storage space and improve the utilization rate of storage resources.
Description
Technical Field
The present application relates to the field of blockchain data storage technologies, and in particular, to a data storage method, a device, a terminal device, and a computer readable storage medium.
Background
With the wide application of blockchain technology in the fields of finance, the internet of things, supply chains and the like, the data volume in the blockchain has shown explosive growth. Each participating node needs to store all transaction data and status updates starting from the blockchain creation block, which results in a dramatic increase in blockchain storage requirements.
In a blockchain related scene, in order to improve the storage performance of data, a copy-on-write strategy is adopted in the related technology, old data is not directly covered when a new version is generated, but historical data is reserved by creating a new node, so that the consistency of the version data is ensured, but because the node is stored in units of pages (pages), when a small amount of data in the pages needs to be modified, the whole Page must be copied and rewritten, and an excessive data copying phenomenon exists, so that the storage resource is inefficiently utilized.
Disclosure of Invention
The embodiment of the application provides a data storage method, a data storage device and a terminal equipment computer readable storage medium, which can avoid repeated storage of unchanged data, reduce the waste of storage space and improve the utilization rate of storage resources.
In a first aspect, an embodiment of the present application provides a blockchain data storage method, including:
acquiring first state data of a plurality of first units in a block chain, wherein each first unit is a block unit in which data update occurs in the block unit;
Performing de-duplication processing on the plurality of first state data to obtain a plurality of second state data;
And storing the second state data in batches to corresponding storage positions in the storage area.
In the embodiment of the application, the information of all the blocks with data updating on the block chain is collected, then the collected information of the blocks is subjected to the deduplication processing, namely, the repeatedly recorded data is filtered, and finally, all the deduplicated data are written into the storage area in batches, which is equivalent to the fact that the data are written into the storage area after the deduplication processing through accumulating a plurality of writing operations, so that the unchanged data are prevented from being repeatedly stored, the waste of the storage space is reduced, and the utilization rate of the storage medium is improved.
In a possible implementation manner of the first aspect, the performing deduplication processing on the plurality of first state data to obtain a plurality of second state data includes:
Acquiring coded data for carrying out data identification on the first state data from the first state data of each first unit;
And filtering repeated coded data and first state data from the plurality of coded data and the plurality of first state data to obtain a plurality of second state data.
In the embodiment of the application, the stored data is subjected to the deduplication processing, and only the incremental data is stored, so that the waste of the storage space can be reduced, and the utilization rate of the storage medium can be improved.
In one embodiment, filtering the repeated encoded data and the repeated first state data from the plurality of encoded data and the plurality of first state data to obtain a plurality of second state data includes:
traversing a plurality of coded data, and filtering repeated coded data;
traversing the first state data corresponding to the plurality of coded data, and filtering out the first state data which are repeated in the first state data corresponding to the coded data so as to obtain a plurality of second state data.
In the embodiment of the application, the writing amplification effect can be reduced and the writing performance of the data can be improved by carrying out the de-duplication processing on the stored data.
In one embodiment, the storing the second status data in batches to the corresponding storage locations in the storage area includes:
Sorting the plurality of first units according to the identifier corresponding to each first unit;
Classifying the plurality of second state data according to the ordered first units to obtain classified second state data;
And storing the classified second state data in batches to the corresponding storage positions in the storage area.
In the embodiment of the application, the plurality of state data can be classified and arranged according to the block identification for the block height, and the specific block can be quickly positioned through the block height, so that the processing performance of the state data is improved.
In one embodiment, classifying the plurality of second state data according to the ordered first units to obtain classified second state data includes:
sequencing the plurality of second state data according to the coded data corresponding to each second state data to obtain sequenced second state data;
and classifying the sorted second state data into each first unit after sorting according to the identifiers corresponding to the first units.
In the embodiment of the application, the second state data in each block identifier (height) are ordered according to codes (hash values), so that the subsequent batch processing and query performance can be optimized.
In one embodiment, storing the classified second state data in batches to corresponding storage locations in the storage area includes:
Data filtering is carried out on a plurality of second state data in each first unit every preset time, and repeated second state data in each first unit are filtered;
And storing the second state data in the plurality of first units after the repeated data are filtered to the corresponding storage positions in the storage area.
In one embodiment, after the plurality of second state data are stored in batches in the storage locations corresponding to the storage areas, and when third state data of a second unit updated in the blockchain exist, performing data deduplication processing on the third state data, and storing the third state data subjected to the deduplication processing in the new storage locations in the storage areas, wherein the second unit is a blockunit in the blockchain, in which data update occurs after the first unit, and the third state data is updated state data corresponding to the second unit.
In the embodiment of the application, after each data writer operation, the system generates a new node, namely a new storage position or area, for the status data writing operation of subsequent updating, and the dynamic updating mechanism can ensure the stability and the data ordering of the system.
In a second aspect, an embodiment of the present application provides a data storage device, including:
The system comprises a data acquisition module, a data updating module and a data updating module, wherein the data acquisition module is used for acquiring first state data of a plurality of first units in a block chain, and each first unit is a block unit in which data updating occurs in the block chain;
The data deduplication module is used for performing deduplication processing on the plurality of first state data to obtain a plurality of second state data;
and the data storage module is used for storing the second state data in batches to the corresponding storage area.
In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the data storage method according to any one of the first aspects when the computer program is executed by the processor.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which when executed by a processor implements a data storage method as in any of the first aspects above.
In a fifth aspect, an embodiment of the application provides a computer program product for, when run on a terminal device, causing the terminal device to perform the data storage method of any of the first aspects described above.
It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data storage method according to an embodiment of the present application;
FIG. 2 is a flowchart of acquiring second status data according to an embodiment of the present application;
FIG. 3 is a schematic diagram of the Batch structure data according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a flow chart of status data batch storage according to an embodiment of the present application;
FIG. 5 is a block cell ordering diagram using BlockQueue structures according to one embodiment of the present application;
FIG. 6 is a schematic diagram of status data ordering using BlockQueue according to one embodiment of the present application;
FIG. 7 is a schematic diagram of a BlockQueue data structure provided by an embodiment of the present application;
FIG. 8 is a second schematic flow chart of status data batch storage according to an embodiment of the present application;
FIG. 9 is a schematic diagram of a data storage device according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the term "comprises/comprising" when used in this specification and the appended claims is taken to specify the presence of stated features, integers, steps, operations, elements, and/or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in the present description and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or but not all embodiments" unless expressly specified otherwise.
With the wide application of blockchain technology in the fields of finance, the internet of things, supply chains and the like, the data volume in the blockchain has shown explosive growth. Each participating node needs to store all transaction data and status updates starting from the blockchain creation block, which results in a dramatic increase in blockchain storage requirements.
In order to improve the storage performance of data in a blockchain scenario, a Tree structure (Time-Structured Multi-version MERKLE TREE, TSMM-Tree) for Multi-version storage according to a Time structure is adopted in the related art, and a copy-on-write strategy is combined, when a new version is generated, old data is not directly covered, but historical data is reserved through creating new nodes, so that the consistency of the version data is ensured, but because the nodes are stored in units of pages, when a small amount of data in the pages needs to be modified, the whole pages must be copied and rewritten, and when leaf nodes are changed in the related art, the change is propagated upwards along the hierarchy of the Tree, and a parent node is updated until a root node in turn. Such a hierarchical propagation mechanism ensures the integrity of the tree structure and can reflect the most current state changes at all levels. The problems caused by the above are as follows:
① Excessive data replication
Since nodes are stored in pages, when a small amount of data in a Page needs to be modified, the entire Page must be copied and rewritten. Even though the actual modified data only occupies a small portion of the Page, the entire 4KB Page needs to be rewritten, resulting in redundancy caused by the large amount of data replication.
② Redundancy overhead of copy-on-write-the system must replicate the leaf nodes of the previous version each time a new version is generated, although the actual modified data represents only a small fraction of the nodes. This manner of operation increases unnecessary disk I/O and storage overhead.
③ The effect of write amplification by hierarchical propagation is that a leaf node's change affects not only itself, but also recursively updates its parent node until the root node. Each update causes nodes on the entire path to be replicated and rewritten, further increasing the write overhead. Such recursive updating can result in each modification involving overwriting of nodes, magnifying the write cost.
In order to solve the above related technical problems, the present application provides a data storage method, which combines the actual application scenario of the blockchain, and when the data storage is performed by using the tree structure, by introducing Batch (Batch), blocking queue (BlockQueue), check point (Checkpoint) mechanism and additional log (Vlog) storage strategy, unnecessary Page copy and node update operations are reduced, the write amplification effect is effectively alleviated, and the write performance and overall efficiency of the system are improved.
The Batch mechanism is a mechanism for processing a plurality of tasks or data items as a whole instead of processing the tasks or data items one by one, the blocking queue is a special queue, when the queue is empty, a task execution thread waits until data is available in the queue, when the queue is full, a task generation thread waits until a free space exists in the queue, the Checkpoint mechanism is used for storing a snapshot of the state of the system at a specific time point and triggering an event when the system stores certain data at a specific time point, and the additional log storage strategy is used for writing new log records (such as operation logs, transaction logs and the like) into a log file in an additional mode instead of covering or modifying existing data. By utilizing the mechanism, when the data updated in the data are stored in the block chain scene, the unchanged data do not need to be repeatedly stored, the waste of storage space is reduced, and the utilization rate of storage resources is improved.
Referring to fig. 1, which is a schematic flow chart of a data storage method according to an embodiment of the present application, by way of example, and not limitation, the method may include the following steps:
S101, acquiring first state data of a plurality of first units in a block chain, wherein each first unit is a block unit in which data update occurs in the block chain.
In an embodiment of the present application, state data for a plurality of block units (referred to as first units) in which data update occurs is obtained from a blockchain. "state data" generally refers to current information stored on the blockchain, such as account balance, asset ownership, state of the smart contract, etc., and "first state data" refers to state data after a transaction has occurred.
In particular, a blockchain browser or communication with a blockchain node may be used to retrieve a complete record of the transaction, analyze the transaction record to determine the state of the blockchain prior to the transaction, extract relevant state data from the transaction record, which may include, but is not limited to, account balances, address owned tokens, storage variables of smart contracts, and the like.
S102, performing de-duplication processing on the plurality of first state data to obtain a plurality of second state data.
In the embodiment of the application, when the first state data with data update is stored, in order to improve the storage space utilization rate of the system, data cleaning, namely data deduplication, is needed to be performed on the updated first state data, so that some repeated data can be cleaned, and the writing operation of the repeated data is reduced.
Specifically, a Batch mechanism may be used when performing deduplication processing on the first state data, where the Batch mechanism is a data structure table implemented by using a skip table. A skip list is a data structure that increases the efficiency of searching by adding multiple levels of indexes to a linked list, where it is used to implement a "Batch," i.e., a Batch-processed data set. The Batch is equivalent to a container for collecting data of transactions in a blockchain, the internal use of the skip list structure is realized, and the skip list is used for improving the efficiency of data writing and searching, and can be rapidly positioned to the position of the data through multi-level indexes, so that the searching time is shortened, and the deduplication operation can be realized.
After a plurality of updated data (first state data) updated in the blockchain are obtained, all transaction data of each unit are required to be written into a Batch structure data table preferentially, after the Batch structure data table is written, batch de-duplication processing is carried out on all the transaction data by utilizing a skip table structure in the Batch structure data table, so as to obtain second state data after de-duplication processing, and the process of de-duplication processing on the first state data by utilizing a Batch mechanism is detailed in steps S201-S202.
In one embodiment, as shown in fig. 2, a flowchart of acquiring second status data provided in an embodiment of the present application, step S102 includes:
S201, obtaining coded data for carrying out data identification on the first state data from the first state data of each first unit.
In embodiments of the present application, the encoded data generally refers to a particular field in the transaction record, such as a Key field, which is used to uniquely identify a transaction, and this Key field may be the transaction ID itself, which is the unique identifier of the transaction record. In many systems, the transaction ID is generated by a hash algorithm, such as SHA-256, ensuring its uniqueness and non-tamper-ability.
In one implementation, the Key field may be a reduced representation of a transaction hash value, one for each transaction in the blockchain, which is a digest of the transaction's content. For convenience of reference and storage, sometimes a substring or a specific length of a hash Value is used, as shown in fig. 3, which is a schematic diagram of a Batch structure provided by an embodiment of the present application, and includes a plurality of Key fields, and Value values (state data) corresponding to each field.
In another implementation, this KEY field may be the account address of the participating transaction if the transaction involves a transfer of funds between accounts. The account address is public and is typically of a fixed length, e.g. the address in bitcoin is 26 to 35 characters.
S202, repeated coded data and first state data are filtered out from the coded data and the first state data, and a plurality of second state data are obtained.
In one implementation, step S202 includes:
Traversing the first state data corresponding to the plurality of coded data, filtering the first state data with repetition in the first state number corresponding to the coded data, and obtaining a plurality of second state data.
In the embodiment of the application, by traversing all state data in the Batch, some identical data is removed based on the first encoded data, for example, ken1= A, key1 =b, etc., the identical ken 1 (encoded data) can be filtered, and one ken 1 code is reserved. Meanwhile, the first state data a or B, etc., having the data value (value) corresponding to the same key field is removed, and the data after the duplication removal is regarded as "second state data".
In the method, the first state data is subjected to preliminary deduplication through the Batch structure, so that the complexity of subsequent processing and repeated operation are reduced.
S103, storing the second state data in batches to corresponding storage positions in the storage area.
In one embodiment, as shown in fig. 4, a flow chart of batch storage of status data provided in the embodiment of the present application is shown in fig. 1, and step S103 includes:
S301, sorting the plurality of first units according to the identifiers corresponding to the first units.
In the embodiment of the present application, in order to reduce frequent duplication of pages during writing operations, to reduce waste of storage space, blockQueue may be used to store the second state data in batches, and BlockQueue is a queue structure for storing Batch data. One Batch of data stores more transaction data of one block unit, and BlockQueue can store more Batch of data.
BlockQueue are organized using block heights (identifiers) of block units, where a block height refers to the position of a block in a blockchain on the chain, typically expressed in terms of an integer. Specifically, the Block height is 0 from the start of the generated Block (Genesis Block) of the Block chain, the height of each subsequent Block is sequentially incremented by 1, wherein, tile height 0, 1. The sorting is done in order from small to large according to identifiers at BlockQueue. Fig. 5 shows first units after BlockQueue is a schematic diagram of block unit ordering by using a BlockQueue structure according to an embodiment of the present application, as shown in fig. 5, seq0... Seqn, which are respectively block heights corresponding to the first units in which data update occurs in a block chain.
S302, classifying the plurality of second state data according to the ordered first units to obtain classified second state data.
In one embodiment, step S302 includes:
Sorting the plurality of second state data according to the coded data corresponding to each second state data to obtain the sorted second state data, and respectively classifying the sorted second state data into each first unit after sorting according to the identifiers corresponding to the first units.
In the embodiment of the present application, after the Batch writing of the transaction data of one block is completed, the second state data (the data after the duplication removal process) in the Batch is ordered according to the code corresponding to each state data (i.e. the hash value may be a key field), as shown in fig. 6, which is a schematic diagram of the state data ordering by using BlockQueue provided in the embodiment of the present application, as shown in fig. 6, is ordering data of one Batch, the code data of the ordered second state data is key1, key2, keyN, valueA, value a, keyZ is the second state data corresponding to each code data.
The data of one block unit is written into one Batch data, after the Batch data of a plurality of block units are all ordered according to the above, the Batch data are temporarily stored in a BlockQueue data structure, data classification is carried out according to the block height, the classified second state data are temporarily stored in the corresponding position of BlockQueue, and the sorted third state data are shown in fig. 7 and are a BlockQueue data structure schematic diagram provided by the implementation of the application.
As shown in fig. 7, seq in BlockQueue is the current block height, key is the transaction Key field in the current block, len is the total number of keys in the current block, vtr is the storage position pointer information of Value data corresponding to the current Key on Vlog, in one BlockQueue, all data are continuously arranged in a growing manner according to Seq, in one Seq, all keys are arranged according to the data sequence after Hash, and the purpose is that when querying data, the efficiency of query can be controlled to store according to BlockQueue data in the subsequent writing process.
In the above method, the system sorts the data according to the height of the block according to the hash value, and stores the sorted data in BlockQueue, blockQueue is used for temporarily storing the data, so as to avoid frequent writing operations.
S303, storing the classified second state data in batches to corresponding storage positions in the storage area.
In the embodiment of the present application, after BlockQueue caches the data of the Batch structure, the data of BlockQueue may be written into the corresponding storage area.
In the method, a plurality of block data are accumulated through BlockQueue and organized according to the block height (Seq), and are ordered according to the hash value of Key, so that batch processing is facilitated. Through BlockQueue, a plurality of small batches of data can be uniformly written into the storage area when one Checkpoint is triggered, so that the frequency of data copying and the effect of write amplification are effectively reduced.
In an embodiment, referring to fig. 8, a second flow chart of batch storage of status data provided in the embodiment of the present application is shown, and step S303 includes:
s401, data filtering is carried out on a plurality of second state data in each first unit every preset time, and repeated second state data in each first unit are filtered.
In an embodiment of the present application, in order to avoid frequent write operations, the present application provides an optimization mechanism for Checkpoint batch writing, where a trigger condition of Checkpoint may be preset, for example, triggered based on time (e.g., every 1 minute) or data amount (e.g., blockQueue data amounts stored in reach a certain threshold or every n block heights).
When the Checkpoint is triggered, the second state data is subjected to deduplication processing again before being written into the storage medium, so that repeated data writing is avoided, the Page replication times of single writing are remarkably reduced through batch writing operation, the writing amplification effect is reduced, when a plurality of deduplicated second state data are written into the storage position, the writing is performed by adopting an append type Vlog mechanism, and the append type writing is performed on the data in Vlog according to the time sequence, so that random writing operation is not involved.
And S402, storing the second state data in the plurality of first units after repeated data are filtered to corresponding storage positions in a storage area.
In the embodiment of the present application, when the Checkpoint is triggered and after performing the deduplication processing on the second state data, the system performs batch processing on the data (the second state data) in the first units in BlockQueue, and after the Checkpoint period is finished, the system writes the data in BlockQueue into the storage area in batch.
When data is stored, the tree structure is utilized to store the data, the generation block is generated by initializing a block chain, the reference tree is generated in the state tree based on the generation block, and the updated state data is used for writing the current reference tree. Wherein the state tree is a tree structure comprising a plurality of nodes, each node representing a state data, the state tree being operable to represent a state change of the system.
In the storage process, the reference tree performs data writing according to the Key-Value pair of BlockQueue, the Value part is stored in an independent Vlog, and Vlog adopts an additional writing mechanism to write the first storage position corresponding to the storage area, so that performance loss caused by random writing can be effectively avoided, wherein the storage area can be a magnetic disk. Vlog are written additionally in time sequence, and random writing operation is not involved. This approach can greatly improve write performance and avoid memory fragmentation problems caused by write operations. Each Key will record its corresponding Value's location (Vtr pointer) in Vlog in the tree structure, thereby ensuring that the data can be located quickly. During query operation, the system can quickly find Value data through the Vtr pointer, so that query efficiency is optimized.
In the above method, a Checkpoint mechanism is introduced, and batch writing is performed after the completion of the Checkpoint period by delaying the writing operation of data. This bulk write strategy significantly reduces the node replication and update frequency, reducing the write amplification effect of the system. The delay writing not only reduces the copying times of the Page, but also reduces unnecessary disk I/O operation, and improves the writing performance of the system.
In one embodiment, the method further comprises:
And after the plurality of second state data are stored in batches to the corresponding storage positions in the storage area, and when the third state data of the updated second unit exist in the block chain, carrying out data deduplication processing on the third state data, and storing the third state data subjected to the deduplication processing to the new storage positions in the storage area, wherein the second unit is a block unit in the block chain, wherein the data update occurs after the first unit, and the third state data is updated state data corresponding to the second unit.
In the embodiment of the application, after each Checkpoint operation is finished, the tree structure is updated to generate a new reference tree as a new storage position in the storage area where data is written, because in one Checkpoint period, the initial reference tree already stores enough state data of block units, and when the state data of a new block unit, namely the second unit is updated, the reference tree which needs to be regenerated into information is used for writing operation of subsequent new updated data (third single-state data), and the reference tree updating strategy ensures the stability of the tree structure and avoids frequent node updating and data out-of-order problems.
It should be noted that, when the state data stored in the reference tree in the memory is sufficiently large, some historical data may be transferred to some other device to reduce the memory pressure in order to reduce the memory pressure.
It should be noted that, after each Checkpoint operation, the latest state tree generated by the system is used as a new reference number, i.e. the second node, for the next round of data writing operation. The dynamic updating of the reference tree ensures the structural stability and data ordering of the tree structure.
The application aims at solving the problem that a large amount of repeated data are required to be copied in the data storage process by utilizing a tree structure, and provides an innovative optimization solution by combining the actual application scene of a block chain, and unnecessary Page copying and node updating operations are reduced by introducing Batch, blockQueue, checkpoint mechanisms and adding Vlog storage strategies, so that the write-in amplification effect is effectively relieved, and the write-in performance and the overall efficiency of the system are improved.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
In one embodiment, a data storage device is provided, which corresponds to the data storage method in the above embodiment one by one. Fig. 9 is a schematic diagram of a data storage device according to an embodiment of the present application, including:
A data obtaining module 91, configured to obtain first status data of a plurality of first units in a blockchain, where each first unit is a blockunit in the blockchain where data update occurs;
the data deduplication module 92 is configured to perform deduplication processing on the plurality of first state data to obtain a plurality of second state data;
The data storage module 93 is configured to store the plurality of second state data in a storage location corresponding to the storage area in batch.
Optionally, the data deduplication module 92 is further configured to:
Acquiring coded data for carrying out data identification on the first state data from the first state data of each first unit;
and filtering repeated coded data and repeated first state data from the plurality of coded data and the plurality of first state data to obtain a plurality of second state data.
Optionally, the data deduplication module 92 is further configured to:
Traversing a plurality of coded data, and filtering repeated coded data;
Traversing the first state data corresponding to the plurality of coded data, and filtering out the first state data which are repeated in the first state number corresponding to the coded data so as to obtain a plurality of second state data.
Optionally, the data storage module 93 is further configured to:
Sorting the plurality of first units according to the identifier corresponding to each first unit;
classifying the plurality of second state data according to the ordered first units to obtain classified second state data;
And storing the classified second state data in batches to the corresponding storage positions in the storage area.
Optionally, the data storage module 93 is further configured to:
sequencing the plurality of second state data according to the coded data corresponding to each second state data to obtain sequenced second state data;
and classifying the sorted second state data into each first unit after sorting according to the identifiers corresponding to the first units.
Optionally, the data storage module 93 is further configured to:
Data filtering is carried out on a plurality of second state data in each first unit every preset time, and repeated second state data in each first unit are filtered;
And storing the second state data in the plurality of first units after repeated data are filtered to corresponding storage positions in the storage area.
Optionally, the data storage module 93 is further configured to:
And after the plurality of second state data are stored in batches to the first storage position in the storage area, and when third state data of the updated second unit exist in the block chain, carrying out data deduplication processing on the third state data, and storing the third state data subjected to the deduplication processing to a new storage position in the storage area, wherein the second unit is a block unit in the block chain, wherein the data update occurs after the first unit, and the third state data is updated state data corresponding to the second unit.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Fig. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application. As shown in fig. 10, the terminal device 10 of this embodiment includes at least one processor 100 (only one is shown in fig. 10), a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100, the processor 100 implementing the steps in any of the various data storage method embodiments described above when executing the computer program 102.
The terminal device may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal device may include, but is not limited to, a processor, a memory. It will be appreciated by those skilled in the art that fig. 10 is merely an example of the terminal device 10 and is not intended to limit the terminal device 10, and may include more or fewer components than shown, or may combine certain components, or may include different components, such as input-output devices, network access devices, etc.
The Processor 100 may be a central processing unit (Central Processing Unit, CPU), and the Processor 100 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Memory 101 may in some embodiments be an internal storage unit of terminal device 10, such as a hard disk or memory of terminal device 10. The memory 101 may also be an external storage device of the terminal device 10 in other embodiments, such as a plug-in hard disk provided on the terminal device 10, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like. Further, the memory 101 may also include both an internal storage unit and an external storage device of the terminal device 10. The memory 101 is used to store an operating system, application programs, boot Loader (Boot Loader), data, and other programs and the like, such as program codes of computer programs and the like. The memory 101 may also be used to temporarily store data that has been output or is to be output.
The embodiments of the present application also provide a computer readable storage medium storing a computer program, which when executed by a processor, implements the steps of the above-described method embodiments.
Embodiments of the present application provide a computer program product enabling a terminal device to carry out the steps of the method embodiments described above when the computer program product is run on the terminal device.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above-described embodiments, and may be implemented by a computer program to instruct related hardware, and the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, executable files or in some intermediate form, etc. The computer readable medium can include at least any entity or device capable of carrying computer program code to an apparatus/terminal device, a recording medium, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of modules or elements is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., elements or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over network elements. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The foregoing embodiments are merely for illustrating the technical solution of the present application, but not for limiting the same, and although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solution described in the foregoing embodiments may be modified or substituted for some of the technical features thereof, and that these modifications or substitutions should not depart from the spirit and scope of the technical solution of the embodiments of the present application and should be included in the protection scope of the present application.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411874704.1A CN119806409A (en) | 2024-12-18 | 2024-12-18 | Data storage method, device, terminal equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411874704.1A CN119806409A (en) | 2024-12-18 | 2024-12-18 | Data storage method, device, terminal equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN119806409A true CN119806409A (en) | 2025-04-11 |
Family
ID=95277280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411874704.1A Pending CN119806409A (en) | 2024-12-18 | 2024-12-18 | Data storage method, device, terminal equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN119806409A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902686A (en) * | 2014-03-25 | 2014-07-02 | 华为技术有限公司 | Data duplicate removing method and data duplicate removing device |
| CN110609872A (en) * | 2019-09-20 | 2019-12-24 | 北京海益同展信息科技有限公司 | Method and apparatus for synchronizing node data |
| KR20210074980A (en) * | 2019-12-12 | 2021-06-22 | (주)포뎁스 | Electronic terminal device capable of update processing for application based on blockchain and operating method thereof |
| CN116662440A (en) * | 2022-02-17 | 2023-08-29 | 腾讯科技（深圳）有限公司 | Block chain storage method, device, electronic equipment, storage medium and program product |
-
2024
- 2024-12-18 CN CN202411874704.1A patent/CN119806409A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902686A (en) * | 2014-03-25 | 2014-07-02 | 华为技术有限公司 | Data duplicate removing method and data duplicate removing device |
| CN110609872A (en) * | 2019-09-20 | 2019-12-24 | 北京海益同展信息科技有限公司 | Method and apparatus for synchronizing node data |
| KR20210074980A (en) * | 2019-12-12 | 2021-06-22 | (주)포뎁스 | Electronic terminal device capable of update processing for application based on blockchain and operating method thereof |
| CN116662440A (en) * | 2022-02-17 | 2023-08-29 | 腾讯科技（深圳）有限公司 | Block chain storage method, device, electronic equipment, storage medium and program product |
Non-Patent Citations (1)
| Title |
|---|
| 王千阁等: "区块链系统的数据存储与查询技术综述", 计算机科学, vol. 45, no. 12, 15 December 2018 (2018-12-15), pages 12 - 18 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11416350B2 (en) | Recovery of in-memory databases from log records | |
| JP7395227B2 (en) | Data backup methods, devices, servers and computer programs | |
| US7366859B2 (en) | Fast incremental backup method and system | |
| US8849767B1 (en) | Method and apparatus for identifying and eliminating duplicate data blocks and sharing data blocks in a storage system | |
| US7958091B2 (en) | Method for fast bulk loading data into a database while bypassing exit routines | |
| US8560500B2 (en) | Method and system for removing rows from directory tables | |
| EP3316150B1 (en) | Method and apparatus for file compaction in key-value storage system | |
| CN102436408B (en) | Data storage cloudification and cloud backup method based on Map/Dedup | |
| US10866865B1 (en) | Storage system journal entry redaction | |
| US20110035359A1 (en) | Database Backup and Restore with Integrated Index Reorganization | |
| US10866968B1 (en) | Compact snapshots of journal-based storage systems | |
| CN111522791B (en) | Distributed file repeated data deleting system and method | |
| CN107665219B (en) | Log management method and device | |
| US12099486B2 (en) | Schema management for journal-based storage systems | |
| US20230083104A1 (en) | Efficiently Deleting Snapshots in a Log-Structured File System (LFS)-Based Storage System | |
| CN112965939A (en) | File merging method, device and equipment | |
| US10380141B1 (en) | Fast incremental backup method and system | |
| US10452496B2 (en) | System and method for managing storage transaction requests | |
| US20180011897A1 (en) | Data processing method having structure of cache index specified to transaction in mobile environment dbms | |
| US11599520B1 (en) | Consistency management using query restrictions in journal-based storage systems | |
| CN114691307A (en) | Transaction processing method and computer system | |
| CN110515897B (en) | Method and system for optimizing reading performance of LSM storage system | |
| CN119806409A (en) | Data storage method, device, terminal equipment and computer readable storage medium | |
| CN112380174A (en) | XFS file system analysis method containing deleted files, terminal equipment and storage medium | |
| CN115878563B (en) | Method for realizing directory-level snapshot of distributed file system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |