[go: up one dir, main page]

HK40081512A - Data update method and apparatus - Google Patents

Data update method and apparatus Download PDF

Info

Publication number
HK40081512A
HK40081512A HK42023070464.5A HK42023070464A HK40081512A HK 40081512 A HK40081512 A HK 40081512A HK 42023070464 A HK42023070464 A HK 42023070464A HK 40081512 A HK40081512 A HK 40081512A
Authority
HK
Hong Kong
Prior art keywords
data
operation information
storage block
change operation
storage
Prior art date
Application number
HK42023070464.5A
Other languages
Chinese (zh)
Inventor
何超
Original Assignee
腾讯科技(深圳)有限公司
Filing date
Publication date
Application filed by 腾讯科技(深圳)有限公司 filed Critical 腾讯科技(深圳)有限公司
Publication of HK40081512A publication Critical patent/HK40081512A/en

Links

Description

Data updating method and device
Technical Field
The present application relates to the field of data processing, and in particular, to a data updating method and apparatus.
Background
With the development of internet technologies (such as big data, artificial intelligence, etc.), the data volume shows explosive growth, and the data storage demand also increases sharply. In the storage system, as the data storage capacity increases, the data read-write performance is reduced, and the stability of the storage system is seriously influenced.
Disclosure of Invention
In order to solve the above technical problem, embodiments of the present application provide a data updating method and apparatus, an electronic device, and a computer-readable storage medium.
According to an aspect of the embodiments of the present application, there is provided a data updating method applied to a data storage system, where the system includes a variable storage block and a data storage block, the method including:
acquiring data change operation information;
writing the acquired data change operation information into the variable storage block, and reading the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first storage threshold value;
and updating the data stored in the data storage block based on the read data change operation information.
According to an aspect of the embodiments of the present application, there is provided a data updating apparatus applied to a data storage system, the system including a variable storage block and a data storage block, the apparatus including:
the acquisition module is configured to acquire data change operation information;
the reading module is configured to write the acquired data change operation information into the variable storage block, and read the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first threshold;
and the updating module is configured to update the data stored in the data storage block based on the read data change operation information.
According to an aspect of an embodiment of the present application, there is provided an electronic device including:
a memory storing computer readable instructions;
and the processor reads the computer readable instructions stored in the memory to execute the data updating method of any one of the above.
According to an aspect of embodiments of the present application, there is provided a computer-readable storage medium having stored thereon computer-readable instructions, which, when executed by a processor of a computer, cause the computer to execute the data updating method of any one of the above.
According to an aspect of embodiments herein, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data updating method provided in the above-mentioned various optional embodiments.
In the technical scheme provided by the embodiment of the application, the data storage system comprises a variable storage block and a data storage block, the acquired data change operation information is written into the variable storage block by acquiring the data change operation information, when the occupied storage space of the variable storage block reaches a preset first storage threshold value, the data change operation information stored in the occupied storage space is read from the variable storage block, and the data stored in the data storage block is updated based on the read data change operation information, so that on one hand, the writing of the data change operation information is sorted and separated from the data in the data storage block, and the reading and writing performance of the data change operation information is improved; on the other hand, when the occupied storage space in the variable storage block reaches the preset first storage threshold, the data change operation information stored in the occupied storage space is read from the variable storage block, and the data is updated based on the data change operation information, so that even if more data change operation information is acquired in a short time, the data change operation information can be temporarily stored in the variable storage block, and when the occupied storage space of the variable storage block reaches a certain degree, the data in the data storage block is updated based on the data change operation information, so that buffer time is provided for updating the data, the reduction of the read-write performance of the storage system and the probability of the breakdown of the storage system caused by the sharp increase of the data change operation information are reduced, and the stability of the storage system is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application. It is obvious that the drawings in the following description are only some embodiments of the application, and that for a person skilled in the art, other drawings can be derived from them without inventive effort. In the drawings:
FIG. 1 is a schematic illustration of an implementation environment shown in an exemplary embodiment of the present application;
FIG. 2 is a schematic illustration of an implementation environment shown in another exemplary embodiment of the present application;
FIG. 3 is a flow chart illustrating a data update method according to an exemplary embodiment of the present application;
FIG. 4 is a flow chart of step S120 in the embodiment shown in FIG. 3 in an exemplary embodiment;
FIG. 5-1 is a schematic diagram of a alterable memory block shown in an exemplary embodiment of the present application;
FIG. 5-2 is a schematic diagram illustrating an update to the alterable memory block shown in FIG. 5-1 in accordance with an exemplary embodiment of the present application;
FIG. 6 is a flow chart of a data update method shown in another exemplary embodiment of the present application;
FIG. 7 is a schematic diagram of a data storage block shown in an exemplary embodiment of the present application;
FIG. 8 is a flow chart diagram of a data update method shown in yet another exemplary embodiment of the present application;
FIG. 9 is a flowchart of step S240 in the embodiment shown in FIG. 8 in an exemplary embodiment;
FIG. 10 is a flowchart of step S130 in the embodiment shown in FIG. 3 in an exemplary embodiment;
FIG. 11 is a diagram illustrating a metadata storage block in accordance with an exemplary embodiment of the present application;
FIG. 12-1 is a schematic diagram of a memory block shown in an exemplary embodiment of the present application;
FIG. 12-2 is a schematic structural view of the common block header shown in FIG. 12-1;
FIG. 13 is a flow diagram in an exemplary embodiment based on the application scenario shown in FIG. 2;
FIG. 14 is a block diagram of a data update apparatus shown in an exemplary embodiment of the present application;
fig. 15 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It should also be noted that: reference to "a plurality" in this application means two or more. "and/or" describe the association relationship of the associated objects, meaning that there may be three relationships, e.g., A and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
Before the technical solutions of the embodiments of the present application are described, terms and expressions referred to in the embodiments of the present application are explained, and the terms and expressions referred to in the embodiments of the present application are applied to the following explanations.
Cloud technology refers to a hosting technology for unifying serial resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.
A distributed cloud storage system (hereinafter, referred to as a storage system) refers to a storage system that integrates a large number of storage devices (storage devices are also referred to as storage nodes) of different types in a network through application software or application interfaces to cooperatively work by using functions such as cluster application, grid technology, and a distributed storage file system, and provides a data storage function and a service access function to the outside.
At present, a storage method of a storage system is as follows: logical volumes are created, and when created, each logical volume is allocated physical storage space, which may be the disk composition of a certain storage device or of several storage devices. The client stores data on a certain logical volume, that is, the data is stored on a file system, the file system divides the data into a plurality of parts, each part is an object, the object not only contains the data but also contains additional information such as data identification (ID, ID entry), the file system writes each object into a physical storage space of the logical volume, and the file system records storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
The process of allocating physical storage space for the logical volume by the storage system specifically includes: physical storage space is divided in advance into stripes according to a group of capacity measures of objects stored in a logical volume (the measures often have a large margin with respect to the capacity of the actual objects to be stored) and Redundant Array of Independent Disks (RAID), and one logical volume can be understood as one stripe, thereby allocating physical storage space to the logical volume.
In the related art, as internet technologies (e.g., big data, artificial intelligence, etc.) are developed, the amount of data is increased explosively, and the data storage demand is also increased sharply. In the storage system, along with the increase of data storage capacity, the data read-write performance is reduced, and the stability of the storage system is seriously influenced. Based on this, the embodiment of the application provides a data updating method and device, which can improve the read-write performance and stability of a storage system.
Referring to fig. 1, fig. 1 is a schematic diagram illustrating an implementation environment according to an exemplary embodiment of the present application. The implementation environment includes a server 10, a variable storage block 11, and a data storage block 12.
The server 10 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a CDN (Content Delivery Network), a big data and artificial intelligence platform, and the like, which is not limited herein.
The variable storage block 11 and the data storage block 12 are both storage resources capable of storing data. Wherein, the variable storage block 11 may be an independent storage device, and the data storage block 12 may be an independent storage device; alternatively, the variable storage block 11 and the data storage block 12 may be storage resources in a cloud storage system built based on cloud storage technology.
The data storage block 12 stores data, and the variable storage block 11 stores data change operation information for changing data stored in the data storage block 12.
The server 10 may acquire data change operation information from a user, write the acquired data change operation information into the variable storage block 11, read the data change operation information stored in the occupied storage space from the variable storage block 11 when the occupied storage space of the variable storage block 11 reaches a first storage threshold, and update the data stored in the data storage block 12 based on the read data change operation information. Therefore, on one hand, the writing of the data change operation information is separated from the data arrangement in the data storage block 12, so that the reading and writing performance of the data change operation information is improved; on the other hand, when the occupied storage space in the variable storage block 11 reaches the preset first storage threshold, the data change operation information stored in the occupied storage space is read from the variable storage block 11, and the data is updated based on the data change operation information, so that even if more data change operation information is acquired in a short time, the data change operation information can be temporarily stored in the variable storage block 11, and when the occupied storage space of the variable storage block 11 reaches a certain degree, the data in the data storage block 12 is updated based on the data change operation information, thereby providing buffer time for updating the data, reducing the probability of storage system breakdown and the probability of storage system read-write performance reduction caused by sharp increase of the data change operation information, and improving the stability of the storage system.
Referring to fig. 2, fig. 2 is a schematic diagram of an implementation environment according to another exemplary embodiment of the present application, where the implementation environment further includes a metadata storage block 13 on the basis of the implementation environment shown in fig. 1. The metadata storage block 13 may be a separate storage device; or, the storage resources may be storage resources in a cloud storage system constructed based on cloud storage technology.
The metadata storage block 13 is used to store the address of the variable storage block 11 and the address of the data storage block 12.
In this embodiment, the server 10 may access the metadata storage block 13 to acquire the address of the variable storage block 11 from the metadata storage block 13, and store the data change operation information to the variable storage block 11 or read the data change operation information from the variable storage block 11 based on the address of the variable storage block 11. The server 10 may also acquire the address of the data storage block 12 from the metadata storage block 13 and update the data stored in the address of the data storage block 12 based on the read data change operation information.
It should be noted that the numbers of the variable storage blocks 11, the data storage blocks 12, and the metadata storage blocks 13 in fig. 1 and 2 are merely exemplary. There may be any number of variable storage blocks 11, data storage blocks 12 and metadata storage blocks 13, as may be practical. For example, one metadata storage block 13 may be provided.
If the number of the variable storage blocks 11 is multiple, one of the variable storage blocks 11 is an active variable storage block, the server 10 may write the acquired data change operation information into the active variable storage block, and when an occupied storage space of the active variable storage block reaches a preset second storage threshold, switch the active variable storage block to a frozen variable storage block (where data writing in the frozen variable storage block is prohibited); selecting a variable storage block which does not store data from a plurality of variable storage blocks as a new active variable storage block, and skipping to the step of writing the acquired data change operation information into the active variable storage block; and when the number of the frozen variable storage blocks is monitored to reach a preset number threshold, reading data change operation information from all the frozen variable storage blocks. In this way, at the same time, only one variable storage block 11 may store data change operation information, and when the occupied storage space in the variable storage block 11 reaches the preset second storage threshold, data storage in the variable storage block 11 is prohibited, and the variable storage block 11 is switched to a frozen variable storage block, so that the data change operation information stored in each frozen variable storage block is guaranteed to have a chronological order in the acquisition time (time for acquiring the data change operation information), and therefore, in the subsequent reading process, the data change operation information in the frozen variable storage block 11 at the front of the acquisition time can be read first, the data in the data storage block 12 is updated based on the read data change operation information, and then the data change operation information in the frozen variable storage block 11 at the back of the acquisition time is read, and the data in the data storage block 12 is updated based on the read data change operation information, thereby guaranteeing that the updating based on the data change operation information is performed based on the chronological order.
If the number of the data storage blocks 12 is plural, the addresses of the data storage blocks 12 in the metadata storage block 13 may be stored in order based on the index identification number range. In this way, the server 10 may obtain the order among the plurality of data storage blocks 12 and the index identification number range corresponding to each data storage block 12 from the metadata storage block 13, and sequentially compare the index identification number range of each data storage block 12 with the index identification number included in the read data change operation information according to the order among the plurality of data storage blocks 12, so as to find out the first target data storage block from the plurality of data storage blocks 12; the address of the first target data storage block is acquired from the metadata storage block 13, and the data stored in the address of the first target data storage block is updated based on the read data change operation information, wherein the range of the index identification number of the first target data storage block matches the index identification number contained in the read data change operation information. Thereby increasing the seek speed.
FIG. 3 is a flow diagram illustrating a data update method according to an example embodiment. The method is applied to a data storage system which comprises a variable storage block 11 and a data storage block 12 shown in fig. 1 or fig. 2 and is specifically executed by a server 10 in the environment of the embodiment shown in fig. 1 or fig. 2. Wherein the variable storage block 11 may be a stand-alone storage device; or, the storage resources in the cloud storage system constructed based on the cloud storage technology; the data storage block 12 may be a stand-alone storage device; or, the storage resources in the cloud storage system constructed based on the cloud storage technology can be used. The number and capacity of the variable storage blocks 11 can be flexibly set according to actual needs, the number and capacity of the data storage blocks 12 can be flexibly set according to actual needs, and the capacities of different variable storage blocks 11 can be the same or different; the different data storage blocks 12 may be of the same or different capacities. It should be understood that the capacity of the variable storage block 11 is the size of the maximum available storage space of the variable storage block, and the capacity of the data storage block 12 is the size of the maximum available storage space of the data storage block. Wherein, the variable storage block 11 and the data storage block 12 can be deployed in different storage nodes, thereby realizing distributed storage.
As shown in fig. 3, in an exemplary embodiment, the data updating method may include steps S110 to S130, which are described in detail as follows:
in step S110, data change operation information is acquired.
It should be noted that the data storage block in this embodiment is used for storing data, the data in the data storage block may be changed (for example, new, replaced, deleted, etc.), and the data change operation information is used for instructing the server how to change the data in the data storage block, and is information corresponding to the data change operation. The data change operation includes, but is not limited to, at least one of a data add operation, a data replace operation, a data delete operation, and the like.
In some embodiments, the data change operation may be performed on a platform provided by a server, and the server obtains data change operation information. The platform provided by the server can be a website or application software.
And S120, writing the acquired data change operation information into the variable storage block, and reading the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first storage threshold value.
The variable storage block is used to store data change operation information. The preset first storage threshold is preset, and a specific value of the preset first storage threshold may be flexibly set according to actual needs, for example, may be 100% of the capacity of the variable storage block, or may be 90% of the capacity of the variable storage block, or may be 80% of the capacity of the variable storage block.
In this embodiment, after the data change operation information is acquired, the acquired data change operation information is written into the variable storage block, and when the stored space of the variable storage block reaches the preset first storage threshold, the data change operation information is read from the stored space.
In some embodiments, the data change operation information may be stored in order in the variable storage block, such that when the data change operation information is read from the variable storage block, the data change operation information may be read therefrom in sequence according to the order. The order may be based on the acquisition time of the data change operation information, for example, the later the acquisition time is, the earlier the order is; or the later the acquisition time is, the later the sequence is; in this way, the data change operation information with the earlier acquisition time is read from the variable storage block and the data change operation information with the later acquisition time is read from the variable storage block according to the sequence, so that the update based on the data change operation information is performed based on the time sequence. Alternatively, the order may be sorted based on the index identification number of the data to be changed included in the data change operation information, for example, the smaller the index identification number is, the earlier the order is; the smaller the index identification number, the later the order. It should be noted that, in order to improve the data retrieval speed, an index is usually created for the data, where the index is a separate storage structure that physically sorts values in one or more columns in the data table; the index identification number is used to distinguish different indexes. In order to increase the processing speed of the data to be processed, the data change operation information may include an index identification number of the data to be processed, and may be stored in order based on the index identification number of the data to be processed.
And S130, updating the data stored in the data storage block based on the read data change operation information.
After the data change operation information is read, the data stored in the data storage block is updated based on the read data change operation information.
In this embodiment, on one hand, the writing of the data change operation information is sorted and separated from the data in the data storage block, so as to improve the read-write performance of the data change operation information; on the other hand, when the occupied storage space in the variable storage block reaches the preset first storage threshold, the data change operation information stored in the occupied storage space is read from the variable storage block, and the data is updated based on the data change operation information, so that even if the data change operation information acquired in a short time is more, the data change operation information can be temporarily stored in the variable storage block, and when the occupied storage space of the variable storage block reaches a certain degree, the data in the data storage block is updated based on the data change operation information, so that buffer time is provided for updating the data, the reduction of the read-write performance of the storage system caused by the sharp increase of the data change operation information and the probability of the breakdown of the storage system are reduced, and the stability of the storage system is improved.
Referring to fig. 4, fig. 4 is a flowchart illustrating a step of writing the acquired data change operation information into the variable storage block in step S120 shown in fig. 3 in an exemplary embodiment. As shown in fig. 4, the process of writing the obtained data change operation information into the variable storage block may include steps S121 to S122, which are described in detail as follows:
in step S121, the storage order of the acquired data change operation information in the variable storage block is determined based on the index identification number included in the acquired data change operation information.
It should be noted that, in order to increase the data retrieval speed, in this embodiment, an index is created for the data, and the index is a separate and physical storage structure that sorts the values of one or more columns in the data table. The index identification number is an index id and is used for distinguishing indexes. In order to quickly process the data to be processed, the data change operation information may include an index identification number of the data to be processed.
The data change operation information in the variable storage block is stored in order based on the index identification numbers contained in the data change operation information, wherein the data change operation information can be stored in the order of the index identification numbers from large to small, and can also be stored in the order of the index identification numbers from small to large.
In this embodiment, after the data change operation information is acquired, the storage order of the data change operation information in the variable storage block may be determined based on the index identification number included in the data change operation information.
In step S122, the acquired data change operation information is stored in the variable storage block based on the determined storage order.
After the storage order of the data change operation information is determined, the data change operation information may be stored into the variable storage block based on the determined storage order. For example, assuming that the data change operation information in the variable storage block is stored in the order of increasing index identification numbers, the data change operation information and the index identification number of the data change operation information stored in the variable storage block are as shown in fig. 5-1, and if the data change operation information 4 is acquired and the index identification number included in the data change operation information 4 is 6, the storage order of the data change operation information 4 is determined to be 3 based on the index identification number 6, and therefore, as shown in fig. 5-2, the data change operation information 4 is stored in the 3 rd bit, and the data change operation information 3 originally stored in the 3 rd bit is stored in the 4 th bit.
In this embodiment, the data change operation information in the variable storage block is stored in order based on the index identification number, so that a plurality of pieces of data change operation information with the same index identification number can be stored in one block, and when the data in the data storage block is updated based on the data change operation information subsequently, the data in the data storage block can be updated based on the plurality of pieces of data change operation information by only searching the position of the index identification number once, thereby improving the searching speed and the data updating speed.
Referring to fig. 6, fig. 6 is a flowchart illustrating a data update method according to an exemplary embodiment under the condition that data change operation information in a variable storage block is stored in order based on an index identification number contained in the data change operation information.
As shown in fig. 6, in an exemplary embodiment, the data updating method may further include steps S140 to S160, which are described in detail as follows:
in step S140, after the first data change operation information is read from the variable storage block and the data stored in the data storage block is updated based on the first data change operation information, the second data change operation information is read from the variable storage block.
The first data change operation information is any data change operation information stored in the variable storage block, and the second data change operation information is also data change operation information stored in the variable storage block. The storage location of the second data change operation information is located after the storage location of the first data change operation information, and the storage location of the second data change operation information is adjacent to the storage location of the first data change operation information, that is, the first data change operation information and the second data change operation information are data change operation information having a pair of adjacent storage locations stored in the variable storage block, where the first data change operation information is data change operation information having a storage location that is earlier and the second data change operation information is data change operation information having a storage location that is later.
In this embodiment, the data change operation information in the variable storage block is stored in order based on the index identification number included in the data change operation information, and therefore, according to this order, the first data change operation information can be read from the variable storage block, the data stored in the data storage block can be updated based on the first data change operation information, and the second data change operation information can be read from the variable storage block.
Step S150, using the first position as a starting point, searching the index identification number included in the second data change operation information from the data storage block to determine a second position of the index identification number included in the second data change operation information in the data storage block.
The first position is the position of the index identification number contained in the first data change operation information in the data storage block. The second position is a position of the index identification number contained in the second data change operation information in the data storage block.
In the process of updating the data stored in the data storage block based on the first data change operation information, a first position is determined, and after the second data change operation information is read, the index identification number contained in the second data change operation information can be searched in the data storage block by taking the first position as a starting point, so that the second position is obtained. It should be noted that, when storing data and the index identification numbers of the data, the data and the index identification numbers are usually stored in order based on the index identification numbers, and based on the relationship between the storage order and the index identification numbers and the relationship between the index identification numbers included in the first data change operation information and the index identification numbers included in the second data change operation information, it is possible to determine whether the first position is located before or after the second position, so that the second position can be determined quickly with the first position as a starting point. If the data change operation information in the variable storage block is stored according to the sequence of the index identification numbers from small to large, the index identification number contained in the second data change operation information can be searched from the data storage block by taking the first position as a starting point and according to the direction of the index identification numbers from small to large so as to obtain a second position; if the data change operation information in the variable storage block is stored in the order of the index identification numbers from large to small, the index identification number contained in the second data change operation information can be searched from the data storage block by taking the first position as a starting point and according to the direction of the index identification numbers from large to small so as to determine the second position. For example, as shown in fig. 7, if data and index identification numbers of the data are stored in the data storage block in the order of the index identification numbers from small to large (it should be noted that, in other examples, the index identification numbers of the data and the data may also be stored in the data storage block in the order of the index identification numbers from large to small), as shown in fig. 5-1, the data change operation information in the variable storage block is stored in the order of the index identification numbers from small to large, assuming that the first data change operation information is the data change operation information 2, the second data change operation information is the data change operation information 3, and in the process of updating the data stored in the data storage block based on the first data change operation information, the first position (i.e., line 4 in fig. 7) of the index identification number 3 contained in the data change operation information 2 in the data storage block is found, and after the data change operation information 3 is read, the first position (i.e., line 4 in fig. 7) is found, and the data is stored in the order of the index identification numbers from small to large (i.e., from upper to lower in the order of the data storage block, and the second position is determined from upper position in fig. from the data storage block 8.
Step S160, updating the data stored in the second location based on the second data change operation information.
After determining the second location, updating the data stored in the second location based on the second data change operation information.
In the present embodiment, after reading the first data change operation information from the variable storage block and updating the data stored in the data storage block based on the first data change operation information, the second data change operation information is read from the variable storage block; the storage position of the second data change operation information is positioned behind the first data change operation information, and the storage position of the second data change operation information is adjacent to the storage position of the first data change operation information; searching the index identification number contained in the second data change operation information from the data storage block by taking the first position as a starting point so as to determine a second position of the index identification number contained in the second data change operation information in the data storage block; the first position is the position of an index identification number contained in the first data change operation information in the data storage block; updating the data stored in the second location based on the second data change operation information; therefore, the searching speed can be increased, and the updating efficiency is improved.
Referring to fig. 8, fig. 8 is a flowchart of step S120 in the embodiment shown in fig. 3 in an exemplary embodiment under the condition that the number of the variable storage blocks is multiple (i.e., at least two), and one of the variable storage blocks is an active variable storage block. In the plurality of variable memory blocks, the capacity of each variable memory block may be the same or different.
As shown in fig. 8, the process of writing the obtained data change operation information into the variable storage block, and reading the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches the preset first storage threshold may include steps S210 to S240, and the following is described in detail:
and step S210, writing the acquired data change operation information into an active variable storage block.
And writing the data change operation information into the active variable storage block when the data change operation information is acquired. For a specific writing process, reference may be made to the foregoing description, which is not described herein again.
Step S220, when the occupied storage space of the active variable storage block reaches a preset second storage threshold, switching the active variable storage block to a frozen variable storage block.
It should be noted that, the preset second storage threshold is preset, and a specific value thereof may be flexibly set according to actual needs, for example, may be 100% of the capacity of the variable storage block, or may be 950% of the capacity of the variable storage block, or may be 70% of the capacity of the variable storage block.
Freezing the alterable memory block prohibits writing data.
And when the occupied storage space of the active variable storage block reaches a preset second storage threshold value, switching the active variable storage block into a frozen variable storage block.
Step S230, selecting one variable storage block not storing data from the plurality of variable storage blocks as a new active variable storage block, and skipping to the step of writing the acquired data change operation information into the active variable storage block.
After the active variable memory block is switched to the frozen variable memory block, it is necessary to select one variable memory block which does not store data from the plurality of variable memory blocks as a new active variable memory block, and jump to step S210.
Step S240, when it is monitored that the number of the frozen variable storage blocks reaches the preset number threshold, reading data change operation information from all the frozen variable storage blocks.
The preset number threshold may be flexibly set according to actual needs, and for example, may be set to 50% of the total number of the variable memory blocks, 90% of the total number of the variable memory blocks, or may be set to 1, 2, 5, or the like.
And when the number of the frozen variable storage blocks is monitored to reach a preset number threshold, reading data change operation information from all the frozen variable storage blocks.
Based on the method provided by this embodiment, the time sequence of the acquisition time of the data change operation information in each frozen variable storage block can be made (that is, the acquisition time of each piece of data change operation information in one frozen variable storage block is earlier than that of each piece of data change operation information in another frozen variable storage block, or the acquisition time of each piece of data change operation information in one frozen variable storage block is later than that of each piece of data change operation information in another frozen variable storage block), so that the update based on the data change operation information is performed in the time sequence, and confusion is avoided.
Referring to fig. 9, fig. 9 is a flow chart of step S240 in the embodiment shown in fig. 8 in an exemplary embodiment, provided that the system further includes a metadata storage block.
As shown in fig. 9, the process of reading the data change operation information from all the frozen variable storage blocks may include steps S241 to S242, which are described in detail as follows:
in step S241, the order between the frozen variable storage blocks is obtained from the metadata storage blocks.
It should be noted that the order between the respective frozen variable memory blocks is sorted based on the time of the respective frozen variable memory blocks, wherein the later the time is, the later the sorting is. The time to freeze the variable storage block may be an acquisition time of data change operation information first written in the frozen variable storage block.
The metadata storage block stores the addresses of a plurality of variable storage blocks included in the system, and the corresponding variable storage blocks can be found based on the addresses. The addresses of the frozen variable storage blocks can be stored in order according to the time sequence based on the time of each frozen variable storage block, so that the sequence among the frozen variable storage blocks can be acquired from the metadata storage block.
In this embodiment, when it is monitored that the number of the frozen variable storage blocks reaches the preset number threshold, the metadata storage blocks are accessed to obtain the sequence among the frozen variable storage blocks from the metadata storage blocks.
In step S242, the data change operation information is sequentially read from the respective frozen variable memory blocks according to the order between the respective frozen variable memory blocks.
According to the sequence among the variable storage blocks, the data change operation information in the frozen variable storage block with the earlier time is read firstly, and then the data change operation information in the frozen variable storage block with the later time is read, so that the data change operation information is guaranteed to be updated to the data storage blocks according to the time sequence.
Referring to fig. 10, fig. 10 is a flowchart of step S130 in the embodiment shown in fig. 3 in an exemplary embodiment, under the condition that the system further includes a metadata storage block, and the number of the data storage blocks is plural. The capacities of the plurality of data storage blocks may be the same or different.
As shown in fig. 10, the process of updating the data stored in the data storage block based on the read data change operation information may include steps S131 to S133, which are described in detail as follows:
step S131, obtaining the order among the multiple data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block.
The sequence among the data storage blocks is sorted based on the index identification number range of each data storage block, wherein the data storage blocks can be sorted according to the sequence from large index identification number range to small index identification number range, or sorted according to the sequence from small index identification number range to large index identification number range.
It should be noted that the data storage block stores data and index identification numbers of the data, and the range of the index identification numbers corresponding to the data storage block is the range of the index identification numbers stored in the data storage block. In order to avoid the problems of slow retrieval speed and the like caused by the fact that data with the same index identification number are stored in different data storage blocks, the ranges of the index identification numbers corresponding to the data storage blocks do not overlap.
The metadata storage block stores the address of each data storage block in the plurality of data storage blocks and the index identification number range corresponding to each data storage block, and the corresponding metadata storage block can be found based on the address. The addresses and the index identification number ranges of the data storage blocks can be stored in the metadata storage block in order based on the index identification number ranges, so that the order among the data storage blocks and the index identification number ranges corresponding to the data storage blocks can be obtained from the metadata storage block. For example, referring to fig. 11, the addresses of the data storage blocks and the corresponding index identification number ranges are stored in the order of the index identification number ranges from small to large.
In this embodiment, after the data change operation information is read, the order among the plurality of data storage blocks and the index identification number range corresponding to each data storage block are obtained from the metadata storage block, so as to facilitate searching for the data storage block corresponding to the read data change operation information.
Step S132, sequentially comparing the index identification number range of each data storage block with the index identification number included in the read data change operation information according to the sequence among the plurality of data storage blocks, so as to find out the first target data storage block from the plurality of data storage blocks.
The range of the index identification numbers of the first target data storage block is matched with the index identification numbers contained in the read data change operation information, namely the index identification numbers contained in the data change operation information are in the range of the index identification numbers of the first target data storage block. For example, if the index identification number included in the read data change operation information is 16, and there is a data storage block whose corresponding index identification number range is 10-20, the data storage block is the first target data storage block.
According to the sequence among the data storage blocks, firstly comparing the index identification number range corresponding to the data storage block with the front sequence with the index identification number contained in the read data change operation information, and then comparing the index identification number range corresponding to the data storage block with the back sequence with the index identification number contained in the read data change operation information, so as to determine a first target data storage block from the data storage blocks.
Step S133, acquiring the address of the first target data storage block from the metadata storage block, and updating the data stored in the address of the first target data storage block based on the read data change operation information.
And after the first target data storage block is determined, acquiring the address of the first target data storage block from the metadata storage block, and searching the first target data storage block based on the address, so as to update the data stored in the address of the first target data storage block based on the read data change operation information.
In this embodiment, data is searched based on the sequence among the data storage blocks and the range of the index identification number corresponding to each data storage block, so that the search speed and the data update speed can be increased.
In an exemplary embodiment, on the condition that the data change operation information includes at least one of data addition operation information, data replacement operation information, and data deletion operation information, step S133 shown in fig. 10 may include steps S310 to S330, which are described in detail as follows:
step S310, if the data adding operation information is read, storing the data to be added and the index identification number of the data to be added contained in the data adding operation information into a storage area corresponding to the address of the first target data storage block.
And if the data adding operation information is read, the data to be added and the index identification number of the data to be added are stored into a storage area corresponding to the address of the first target data storage block.
Under the condition that the data in the data storage block and the index identification numbers corresponding to the data are stored in order based on the index identification numbers, the position in the first target data storage block can be determined based on the index identification numbers of the data to be newly added, so that the data to be newly added and the index identification numbers of the data to be newly added are stored based on the determined position.
Step S320, if the data replacement operation information is read, searching the index identification number included in the data replacement operation information from the storage area corresponding to the address of the first target data storage block, and replacing the data corresponding to the searched index identification number with the to-be-replaced data included in the data replacement operation information.
And if the data replacement operation information is read, searching the index identification number of the data to be replaced from the storage area corresponding to the address of the first target data storage block, and replacing the data of the index identification number with the data to be replaced.
Step S330, if the data deletion operation information is read, searching the index identification number included in the data deletion operation information from the storage area corresponding to the address of the first target data storage block, and deleting the data corresponding to the searched index identification number.
And the data deleting operation information contains the index identification number of the data to be deleted, if the data deleting operation information is read, the index identification number of the data to be deleted is searched from the storage area corresponding to the address of the first target data storage block, and the data corresponding to the index identification number is deleted.
In an exemplary embodiment, the data updating method may further include steps S410-S450, which are described in detail as follows:
and step S410, receiving a data searching request sent by the terminal.
The data searching request is used for requesting to search data and comprises an index identification number of the data to be searched.
The server may have a data query function and may receive a data query request transmitted by the terminal.
In step S420, the order among the data storage blocks and the index identification number range corresponding to each data storage block are obtained from the metadata storage block.
The sequence among the data storage blocks is sorted based on the index identification number range of each data storage block, wherein the data storage blocks can be sorted according to the sequence from large index identification number range to small index identification number range, or sorted according to the sequence from small index identification number range to large index identification number range.
The metadata storage block stores the address of each data storage block in the plurality of data storage blocks and the index identification number range corresponding to each data storage block, and the corresponding metadata storage block can be found based on the address. The addresses and the index identification number ranges of the data storage blocks can be stored in the metadata storage block in order based on the index identification number ranges, so that the order among the data storage blocks and the index identification number ranges corresponding to the data storage blocks can be obtained from the metadata storage block.
In this embodiment, after receiving a data search request, the metadata storage blocks are accessed to obtain the order among the plurality of data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage blocks.
Step S430, sequentially comparing the index identification number range corresponding to each data storage block with the index identification number of the data to be searched according to the sequence among the plurality of data storage blocks, so as to find out a second target data storage block from the plurality of data storage blocks.
And matching the index identification number range of the second target data storage block with the index identification number of the data to be searched, wherein the matching of the index identification number range and the index identification number means that the index identification number is in the index identification number range.
According to the sequence among the data storage blocks, the index identification number range corresponding to the data storage block with the front sequence is compared with the index identification number of the data to be searched, and then the index identification number range corresponding to the data storage block with the back sequence is compared with the index identification number of the data to be searched, so that the second target data storage block is searched.
Step S440, sequentially comparing the index identification number in the second target data storage block with the index identification number of the data to be searched, so as to search the data to be searched from the second target storage block.
In this embodiment, the data and the index identification numbers corresponding to the data are stored in the data storage block, and the data and the index identification numbers corresponding to the data are stored in order according to the index identification numbers, so that the search speed can be further increased. The index identification numbers can be stored in descending order or in ascending order.
And sequentially comparing the index identification numbers in the second target data storage block with the index identification numbers of the data to be searched based on the sequence of the index identification numbers in the data storage blocks so as to search the data to be searched from the second target storage block.
And step S450, sending the searched data to the terminal.
And sending the searched data to the terminal, so that the terminal processes the searched data.
In the embodiment, the searching is supported through the range of the index identification number, so that the searching speed is improved.
In an exemplary embodiment, for each of the variable storage blocks, the data storage blocks, and the metadata storage blocks, which are each composed of a block header and a block body, as shown in fig. 12-1, the block header is a common block header, and direct storage is performed using a byte stream. The block is used for storing data, for example, for a variable storage block, the block is used for storing data change operation information; for a data storage block, the block body of the data storage block is used for storing data and an index identification number of the data; for a metadata storage chunk, its chunk is used to store the address of the variable storage chunk, the address of the data storage chunk, and the index number range of the data storage chunk. The data in the block is stored using the encoding algorithm and compression algorithm specified in the block header.
Referring to fig. 12-2, the common block header includes:
magic number: recording a fixed number, mainly used for basic verification of the block, and verifying whether the storage block is a variable storage block, a data storage block or a metadata storage block. If the dirty data is read, the dirty data can be checked out through the magic number, so that the dirty data can be skipped over.
Protocol version: the protocol version of the memory block structure is recorded. For a memory block structure, whether a common block header or a block, an upgrade is possible, so this field is used to record the protocol version to which the current memory block structure belongs.
Head length: and recording the byte length of the public block head, so that after data is read, the public block head can be conveniently and quickly intercepted for analysis.
Total length: the total length of the storage block is recorded, so that the block body can be conveniently intercepted for subsequent processing.
Block type: and recording storage block types including variable storage blocks, metadata storage blocks and data storage blocks.
Data version number: the version number of the data in the block is recorded. In the block of the memory block, the data version number is changed every time data is changed.
And (3) a compression algorithm: and recording the compression algorithm of the block body, and conveniently compressing or decompressing by adopting a corresponding compression algorithm after reading the field.
And (3) an encoding algorithm: and recording the encoding algorithm of the block body, so that the field can be conveniently read and then encoded or decoded by adopting the corresponding encoding algorithm.
The number of data pieces: the total number of pieces of data stored in the block is recorded.
A reserved field: for subsequent possible expanded field sparing.
Referring to fig. 13, fig. 13 is an interactive flowchart of a data update method in an exemplary embodiment based on the application scenario shown in fig. 2. The data updating method in the embodiment is applied to a data updating system, and the system comprises a variable storage block, a data storage block and a metadata storage block.
The data storage block is used for storing data and index identification numbers corresponding to the data, the storage sequence of the data storage block is sorted from small to large according to the index identification numbers, and the smaller the index identification number is, the earlier the storage sequence is; in order to avoid the problems of slow retrieval speed and the like caused by the fact that data with the same index identification number are stored in different data storage blocks, the ranges of the index identification numbers corresponding to the data storage blocks can be not overlapped.
The variable storage block is used for storing data change operation information, the data change operation information contains an index identification number of data to be processed, the storage sequence of the data change operation information in the variable storage block is sorted from small to large according to the index identification number, the smaller the index identification number is, the earlier the storage sequence is, and the data change operation information is used for changing the data in the data storage block.
The number of the variable storage blocks is multiple, wherein one variable storage block is an active variable storage block; the number of data storage blocks is also plural.
The metadata storage block is used for storing the address of each variable storage block in the plurality of variable storage blocks; the metadata storage block is further used for storing the address and the index identification number range of each data storage block in the plurality of data storage blocks, and the storage sequence of the address and the index identification number range of each data storage block is as follows: the smaller the index identification number range, the earlier the storage order.
As shown in fig. 13, in an exemplary embodiment, the data updating method may include steps S510 to S519, which are described in detail as follows:
in step S510, the server obtains data change operation information.
The user can perform data change operation on a platform provided by the server, and the server acquires data change operation information. The platform provided by the server may be a website or application software.
In step S511, the server determines the storage order of the data change operation information in the active variable storage block based on the index identification number included in the data change operation information.
It should be noted that the larger the index identification number is, the later the storage order is. Therefore, a plurality of pieces of data change operation information with the same index identification number acquired within a period of time can be stored in one block, and when data in the data storage block is updated based on the data change operation information subsequently, the position of the index identification number only needs to be searched once, so that the data in the data storage block can be updated based on the plurality of pieces of data change operation information, and the searching speed and the data updating speed are improved.
In step S512, the server stores the acquired data change operation information to the active variable storage block based on the determined storage order.
And storing the acquired data change operation information into the active variable storage block based on the determined storage sequence, so that the data change operation information is stored in the variable storage block according to the sequence of the index identification numbers from small to large.
In step S513, when the occupied storage space of the active variable storage block reaches the preset second storage threshold, the server switches the active variable storage block to the frozen variable storage block.
Wherein freezing the alterable memory block inhibits writing data.
The preset second storage threshold value can be flexibly set according to actual needs.
In step S514, the server selects a variable storage block that does not store data from the plurality of variable storage blocks as a new active variable storage block, and jumps to step S511.
Step S515, when it is monitored that the number of the frozen variable storage blocks reaches the preset number threshold, the server acquires the sequence between the frozen variable storage blocks from the metadata storage block.
The sequence among the frozen variable storage blocks is sorted based on the time of each frozen variable storage block, and the later the time is, the later the sorting is, the time of the frozen variable storage block is the acquisition time of the data change operation information written in the frozen variable storage block at first.
The preset number threshold value can be flexibly set according to actual needs.
In step S516, the server sequentially reads the data change operation information from each of the frozen variable storage blocks according to the order between the frozen variable storage blocks.
The server reads the data change operation information stored in the frozen variable storage block which is ranked earlier, and then reads the data change operation information stored in the frozen variable storage block which is ranked later.
In step S517, the server obtains the order among the data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block.
In step S518, the server compares the index identification number range of each data storage block with the index identification number included in the read data change operation information in sequence according to the sequence among the plurality of data storage blocks, so as to find out the first target data storage block from the plurality of data storage blocks.
The range of the index identification numbers of the first target data storage block is matched with the index identification numbers contained in the read data change operation information.
The index identification number range of the data storage block in the front of the sequence is compared with the index identification number contained in the read data change operation information, and then the index identification number range of the data storage block in the back of the sequence is compared with the index identification number contained in the read data change operation information, so that the first target data storage block is found out.
In step S519, the server acquires an address of the first target data storage block from the metadata storage block, and updates data stored in the address of the first target data storage block based on the read data change operation information.
The method comprises the steps of obtaining an address of a first target data storage block from a metadata storage block, determining the first target data storage block based on the address of the first target data storage block, searching an index identification number contained in data change operation information from the first target data storage block, and updating data corresponding to the searched index identification number based on the read data change operation information.
It should be noted that the detailed processes of steps S510 to S519 are all described in the foregoing embodiments, and therefore, the detailed description is not repeated here.
Fig. 14 is a block diagram of a data update apparatus according to an exemplary embodiment of the present application, which is applied to a data storage system, where the system includes a variable storage block and a data storage, and the variable storage block and the data storage block may be deployed in different storage nodes, and the apparatus includes:
an obtaining module 601 configured to obtain data change operation information; the reading module 602 is configured to write the acquired data change operation information into the variable storage block, and read the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first threshold; an updating module 603 configured to update the data stored in the data storage block based on the read data change operation information.
In another exemplary embodiment, the obtaining module 601 includes:
and the determining module is configured to determine the storage sequence of the acquired data change operation information in the variable storage block based on the index identification number contained in the acquired data change operation information.
And the storage module is configured to store the acquired data change operation information to the variable storage block based on the determined storage sequence.
In another exemplary embodiment, the apparatus further comprises:
an operation information reading module configured to read second data change operation information from the variable storage block after reading the first data change operation information from the variable storage block and updating the data stored in the data storage block based on the first data change operation information; the storage location of the second data change operation information is located after the first data change operation information, and the storage location of the second data change operation information is adjacent to the storage location of the first data change operation information.
The position determining module is configured to search the index identification number contained in the second data change operation information from the data storage block by taking the first position as a starting point so as to determine a second position of the index identification number contained in the second data change operation information in the data storage block; the first position is a position of the index identification number contained in the first data change operation information in the data storage block.
And the data updating module is configured to update the data stored in the second position based on the second data change operation information.
In another exemplary embodiment, on the condition that the number of the variable storage blocks is plural, wherein one of the variable storage blocks is an active variable storage block, the reading module 602 includes:
and the writing submodule is configured to write the acquired data change operation information into the active variable storage block.
The switching module is configured to switch the active variable storage block into the frozen variable storage block when the occupied storage space of the active variable storage block reaches a preset second storage threshold; wherein freezing the alterable memory block inhibits writing data.
And the skipping module is configured to select one variable storage block which does not store data from the plurality of variable storage blocks as a new active variable storage block, and skip to the step of writing the acquired data change operation information into the active variable storage block.
And the reading sub-module is configured to read data change operation information from all the frozen variable storage blocks when the number of the frozen variable storage blocks is monitored to reach a preset number threshold.
In another exemplary embodiment, the reading module 602 includes, on condition that the system further includes a metadata storage block:
the order acquisition module is configured to acquire the order among the frozen variable storage blocks from the metadata storage blocks; the sequence among the frozen variable storage blocks is sorted based on the time of each frozen variable storage block, and the later the time is, the later the sorting is, the time for freezing the variable storage blocks is the acquisition time of the data change operation information which is written into the frozen variable storage blocks firstly.
And the sequence reading module is configured to read the data change operation information from the frozen variable storage blocks in sequence according to the sequence among the frozen variable storage blocks.
In another exemplary embodiment, under the condition that the system further includes a metadata storage block, and the number of the data storage blocks is plural, the updating module 603 further includes:
the first data acquisition module is configured to acquire the sequence among the plurality of data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block; the sequence among the data storage blocks is sorted based on the index identification number range of each data storage block.
The first searching module is configured to sequentially compare the index identification number range of each data storage block with the read index identification numbers contained in the data change operation information according to the sequence among the plurality of data storage blocks so as to search out a first target data storage block from the plurality of data storage blocks; the range of the index identification numbers of the first target data storage block is matched with the index identification numbers contained in the read data change operation information.
And the updating submodule is configured to acquire the address of the first target data storage block from the metadata storage block and update the data stored in the address of the first target data storage block based on the read data change operation information.
In another exemplary embodiment, the updating sub module includes, on a condition that the data change operation information includes at least one of data addition operation information, data replacement operation information, and data deletion operation information:
and the data adding module is configured to store the data to be added and the index identification number of the data to be added contained in the data adding operation information into a storage area corresponding to the address of the first target data storage block if the data adding operation information is read.
And the data replacement module is configured to search the index identification number contained in the data replacement operation information from the storage area corresponding to the address of the first target data storage block if the data replacement operation information is read, and replace the data corresponding to the searched index identification number with the data to be replaced contained in the data replacement operation information.
And the data deleting module is configured to search the index identification number contained in the data deleting operation information from the storage area corresponding to the address of the first target data storage block if the data deleting operation information is read, and delete the data corresponding to the searched index identification number.
In another exemplary embodiment, under the condition that the data in the data storage block is stored in order according to the index identification number, the apparatus further comprises:
the receiving module is configured to receive a data searching request sent by a terminal; the data searching request comprises an index identification number of the data to be searched;
and the second data acquisition module is configured to acquire the sequence among the data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block.
The second searching module is configured to sequentially compare the index identification number range corresponding to each data storage block with the index identification number of the data to be searched according to the sequence among the plurality of data storage blocks so as to search a second target data storage block from the plurality of data storage blocks; the index identification number range of the second target data storage block is matched with the index identification number of the data to be searched.
And the third searching module is configured to sequentially compare the index identification number in the second target data storage block with the index identification number of the data to be searched so as to search the data to be searched from the second target storage block.
And the sending module is configured to send the searched data to the terminal.
Embodiments of the present application further provide an electronic device, which includes a processor and a memory, where the memory has stored thereon computer readable instructions, and when executed by the processor, the computer readable instructions implement the data updating method as described above.
Fig. 15 is a schematic structural diagram of an electronic device shown in accordance with an exemplary embodiment.
It should be noted that the electronic device is only an example adapted to the application and should not be considered as providing any limitation to the scope of the application. The electronic device is also not to be construed as requiring reliance on, or necessity of, one or more components of the exemplary electronic device illustrated in fig. 15.
As shown in fig. 15, in an exemplary embodiment, the electronic device includes a processing component 801, a memory 802, a power component 803, a multimedia component 804, an audio component 805, a sensor component 807, and a communication component 808. The above components are not all necessary, and the electronic device may add other components or reduce some components according to its own functional requirements, which is not limited in this embodiment.
The processing component 801 generally controls overall operation of the electronic device, such as operations associated with display, data communication, and log data processing. The processing component 801 may include one or more processors 809 to execute instructions to perform all or portions of the above-described operations. Further, the processing component 801 may include one or more modules that facilitate interaction between the processing component 801 and other components. For example, the processing component 801 may include a multimedia module to facilitate interaction between the multimedia component 804 and the processing component 801.
The memory 802 is configured to store various types of data to support operation at the electronic device, examples of which include instructions for any application or method operating on the electronic device. The memory 802 stores one or more modules configured to be executed by the one or more processors 809 to perform all or part of the steps of the methods described in the embodiments above.
The power supply component 803 provides power to the various components of the electronic device. The power components 803 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for an electronic device.
The multimedia component 804 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a TP (Touch Panel) and an LCD (Liquid Crystal Display). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.
The audio component 805 is configured to output and/or input audio signals. For example, the audio component 805 includes a microphone configured to receive external audio signals when the electronic device is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. In some embodiments, the audio component 805 also includes a speaker for outputting audio signals.
The sensor assembly 807 includes one or more sensors for providing various aspects of state assessment for the electronic device. For example, the sensor assembly 807 may detect an open/closed state of the electronic device, and may also detect a temperature change of the electronic device.
The communication component 808 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device may access a Wireless network based on a communication standard, such as Wi-Fi (Wireless-Fidelity, wireless network).
It will be appreciated that the configuration shown in fig. 15 is merely illustrative and that the electronic device may include more or fewer components than shown in fig. 15 or have different components than shown in fig. 15. Each of the components shown in fig. 15 may be implemented in hardware, software, or a combination thereof.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software, or may be implemented by hardware, and the described units may also be disposed in a processor. Wherein the names of the elements do not in some way constitute a limitation on the elements themselves.
Yet another aspect of the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist separately without being incorporated in the electronic device.
It should be noted that the computer readable storage medium of the embodiments of the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM), a flash Memory, an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with a computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
Another aspect of the application also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided in the various embodiments described above.
It should be further noted that the data updating method disclosed in the embodiments of the present application may be applied to a block chain system. The Blockchain (Blockchain) is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. The block chain, which is essentially a decentralized database, is a string of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, which is used for verifying the validity (anti-counterfeiting) of the information and generating a next block.
Each variable storage block may correspond to one block chain and is marked as a first block chain, and each data storage block may correspond to one block chain and is marked as a second block chain; each metadata base storage block may correspond to a block chain, denoted as a third block chain. The first block chain is used for storing data change operation information, the second block chain is used for storing data and index identification numbers corresponding to the data, and the third block chain is used for storing the address of the first block chain, the address of the second block chain and the range of the index identification numbers corresponding to the second block chain, so that trusted storage is realized. After obtaining the data change operation information, a node in the block chain system accesses a third block chain to search an address of the first block chain from the third block chain, stores the obtained data change operation information to the first block chain based on the address of the first block chain, reads the data change operation information from the first block chain when the occupied storage space of the first block chain reaches a preset first storage threshold value, searches a second block chain corresponding to the read data change operation information based on the address of each second block chain stored in the third block chain and an index identification number range corresponding to each second block chain, and updates data on the searched second block chain based on the read data change operation information. The process of the data updating method can also be maintained based on an intelligent contract, so that trusted computing is realized. It should be understood that a smart contract is a computer protocol intended to propagate, verify, or execute contracts in an informational manner, with code that automatically executes the "execute that if it happens" result in a traditional contract; the developer can define the contract logic through some programming language, issue it to the block chain (contract registration), and call the key or other event trigger execution according to the logic of the contract terms, complete the contract logic, and the contract can be upgraded or unregistered.
The above description is only a preferred exemplary embodiment of the present application, and is not intended to limit the embodiments of the present application, and one of ordinary skill in the art can easily make various changes and modifications according to the main concept and spirit of the present application, so that the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (10)

1. A data update method applied to a data storage system, the system including a variable storage block and a data storage block, the method comprising:
acquiring data change operation information;
writing the acquired data change operation information into the variable storage block, and reading the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first storage threshold value;
and updating the data stored in the data storage block based on the read data change operation information.
2. The method of claim 1, wherein the writing the obtained data change operation information to the alterable storage block comprises:
determining the storage sequence of the acquired data change operation information in the variable storage block based on an index identification number contained in the acquired data change operation information;
and storing the acquired data change operation information to the variable storage block based on the determined storage sequence.
3. The method of claim 2, wherein the method further comprises:
reading second data change operation information from the variable storage block after reading first data change operation information from the variable storage block and updating data stored in the data storage block based on the first data change operation information; the storage position of the second data change operation information is positioned behind the first data change operation information, and the storage position of the second data change operation information is adjacent to the storage position of the first data change operation information;
searching the index identification number contained in the second data change operation information from the data storage block by taking the first position as a starting point so as to determine a second position of the index identification number contained in the second data change operation information in the data storage block; the first position is the position of an index identification number contained in the first data change operation information in the data storage block;
updating the data stored in the second location based on the second data change operation information.
4. The method of claim 1, wherein the number of variable memory blocks is plural, wherein one variable memory block is an active variable memory block;
the writing the obtained data change operation information into the variable storage block, and reading the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first storage threshold value, includes:
writing the acquired data change operation information into the active variable storage block;
when the occupied storage space of the active variable storage block reaches a preset second storage threshold value, switching the active variable storage block into a frozen variable storage block; wherein writing of data is prohibited in the frozen alterable memory block;
selecting a variable storage block which does not store data from a plurality of variable storage blocks as a new active variable storage block, and skipping to the step of writing the acquired data change operation information into the active variable storage block;
and reading data change operation information from all the frozen variable storage blocks when the number of the frozen variable storage blocks is monitored to reach a preset number threshold.
5. The method of claim 4, wherein the system further comprises a metadata storage block;
the reading of the data change operation information from all the frozen variable storage blocks includes:
acquiring the sequence among the frozen variable storage blocks from the metadata storage blocks; the sequence among the frozen variable storage blocks is sorted based on the time of each frozen variable storage block, the later the time is, the later the sorting is, the time for freezing the variable storage blocks is the acquisition time of the data change operation information which is written into the frozen variable storage blocks firstly;
and sequentially reading the data change operation information from the frozen variable storage blocks according to the sequence among the frozen variable storage blocks.
6. The method of claim 1, wherein the system further comprises a plurality of metadata storage blocks;
the updating the data stored in the data storage block based on the read data change operation information comprises the following steps:
acquiring the sequence among a plurality of data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block; wherein the sequence among the plurality of data storage blocks is sorted based on the index identification number range of each data storage block;
according to the sequence among the data storage blocks, sequentially comparing the index identification number range of each data storage block with the index identification number contained in the read data change operation information so as to find out a first target data storage block from the data storage blocks; the index identification number range of the first target data storage block is matched with the index identification number contained in the read data change operation information;
and acquiring the address of the first target data storage block from the metadata storage block, and updating the data stored in the address of the first target data storage block based on the read data change operation information.
7. The method of claim 6, wherein the data change operation information includes at least one of data addition operation information, data replacement operation information, and data deletion operation information;
the updating the data stored in the address of the first target data storage block based on the read data change operation information includes:
if the data adding operation information is read, storing data to be added and an index identification number of the data to be added contained in the data adding operation information into a storage area corresponding to the address of the first target data storage block;
if the data replacement operation information is read, searching an index identification number contained in the data replacement operation information from a storage area corresponding to the address of the first target data storage block, and replacing the data corresponding to the searched index identification number with the data to be replaced contained in the data replacement operation information;
and if the data deleting operation information is read, searching an index identification number contained in the data deleting operation information from a storage area corresponding to the address of the first target data storage block, and deleting data corresponding to the searched index identification number.
8. The method of claim 6, wherein the data in the data storage block is stored in order according to an index identification number; the method further comprises the following steps:
receiving a data searching request sent by a terminal; the data searching request comprises an index identification number of the data to be searched;
acquiring the sequence among the data storage blocks and the index identification number range corresponding to each data storage block from the metadata storage block;
according to the sequence among the data storage blocks, sequentially comparing the index identification number range corresponding to each data storage block with the index identification number of the data to be searched so as to search out a second target data storage block from the data storage blocks; the index identification number range of the second target data storage block is matched with the index identification number of the data to be searched;
sequentially comparing the index identification number in the second target data storage block with the index identification number of the data to be searched so as to search the data to be searched from the second target storage block;
and sending the searched data to the terminal.
9. The method of claim 1, wherein the alterable storage block and the data storage block are deployed at different storage nodes.
10. A data update apparatus for use in a data storage system, the system comprising a alterable memory block and a data memory block, the apparatus comprising:
the acquisition module is configured to acquire data change operation information;
the reading module is configured to write the acquired data change operation information into the variable storage block, and read the data change operation information stored in the occupied storage space from the variable storage block when the occupied storage space of the variable storage block reaches a preset first threshold;
and the updating module is configured to update the data stored in the data storage block based on the read data change operation information.
HK42023070464.5A 2023-03-23 Data update method and apparatus HK40081512A (en)

Publications (1)

Publication Number Publication Date
HK40081512A true HK40081512A (en) 2023-05-19

Family

ID=

Similar Documents

Publication Publication Date Title
JP7046172B2 (en) Computer implementation methods, computer program products, and systems for storing records in shard database shard tables, computer implementation methods, computer program products, and systems for retrieving records from shard database shard tables. System, as well as a system for storing shard databases
US7447839B2 (en) System for a distributed column chunk data store
US10783115B2 (en) Dividing a dataset into sub-datasets having a subset of values of an attribute of the dataset
KR20170054299A (en) Reference block aggregating into a reference set for deduplication in memory management
US20130339314A1 (en) Elimination of duplicate objects in storage clusters
US20100161780A1 (en) Hot data management method based on hit counter
WO2009042920A2 (en) Lazy updates to indexes in a database
CN113760847A (en) Log data processing method, device, equipment and storage medium
US10706022B2 (en) Space-efficient secondary indexing on distributed data stores
US10579262B2 (en) Optimization of data deduplication
US10915533B2 (en) Extreme value computation
US20110040788A1 (en) Coherent File State System Distributed Among Workspace Clients
US20070061543A1 (en) Method for a distributed column chunk data store
CN115145954A (en) Data query method, data storage method and device
JP2022078972A (en) Change data capture log history, or method of tracking cdc log history, and computer program (tracking change data capture log history)
CN103793391B (en) Method and system for processing and verifying remote dynamic data
US20200019539A1 (en) Efficient and light-weight indexing for massive blob/objects
CN115686343A (en) Data updating method and device
HK40081512A (en) Data update method and apparatus
US20130218851A1 (en) Storage system, data management device, method and program
CN119106035A (en) A block storage method and related equipment
CN118939838A (en) Data storage method, device and electronic device
KR20230096180A (en) Spatial lsm tree apparatus and method for indexing blockchain based geospatial point data
JP2010191903A (en) Distributed file system striping class selecting method and distributed file system
CN118779310A (en) Database management method and system