CN118550447A - A data processing method, device, equipment and storage medium - Google Patents

Abstract

The present application discloses a data processing method, device, equipment and storage medium, which are applied to the field of data processing technology. The method first obtains the data to be processed, and stores the data to be processed in the memory cache of the storage partition corresponding to the primary key of the data to be processed; in response to satisfying the storage condition, at least part of the data to be processed stored in the memory cache of the storage partition is stored in the disk storage space of the storage partition. In this way, the storage partition storing the data to be processed can be determined based on the primary key of the data to be processed, which is convenient for subsequent data query in the storage partition corresponding to the target primary key of the query, thereby narrowing the scope of the data query and improving the speed of querying data.

Description

A data processing method, device, equipment and storage medium

Technical Field

The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium.

Background Art

At present, data processing technology has been applied to many industries. During the application process, a large amount of data is usually stored in the data management platform. When data query or data analysis is required later, relevant data is obtained from the data management platform. Taking log data as an example, log data can be stored in the data management platform for anomaly detection and fault diagnosis. When troubleshooting is required, relevant log data can be queried from the data management platform.

Currently, querying data from the data management platform is inefficient and time-consuming.

Summary of the invention

In view of this, the present application provides a data processing method, apparatus, device and storage medium, which can improve the efficiency of querying data.

To solve the above problems, the technical solutions provided by this application are as follows:

In a first aspect, the present application provides a data processing method, the method comprising:

Get the data to be processed;

Storing the data to be processed in a memory cache of a storage partition corresponding to the primary key of the data to be processed;

In response to the storage condition being met, at least a portion of the to-be-processed data stored in the memory cache of the storage partition is stored in the disk storage space of the storage partition.

In a second aspect, the present application provides a data processing device, the device comprising:

A first acquisition unit, used to acquire data to be processed;

A first storage unit, used to store the data to be processed into a memory cache of a storage partition corresponding to a primary key of the data to be processed;

The second storage unit is used to store at least part of the to-be-processed data stored in the memory cache of the storage partition into the disk storage space of the storage partition in response to satisfying the storage condition.

In a third aspect, the present application provides an electronic device, including:

one or more processors;

a storage device having one or more programs stored thereon,

When the one or more programs are executed by the one or more processors, the one or more processors implement any method described in the first aspect.

In a fourth aspect, the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, any method described in the first aspect is implemented.

In a fifth aspect, the present application provides a computer program product, which, when executed on a device, enables the device to execute any method described in the first aspect.

It can be seen that this application has the following beneficial effects:

The present application provides a data processing method, device, equipment and storage medium, which first obtains the data to be processed, and stores the data to be processed in the memory cache of the storage partition corresponding to the primary key of the data to be processed; in response to satisfying the storage condition, at least part of the data to be processed stored in the memory cache of the storage partition is stored in the disk storage space belonging to the storage partition. In this way, the storage partition storing the data to be processed can be determined based on the primary key of the data to be processed, which facilitates the subsequent data query in the storage partition corresponding to the target primary key of the query, thereby narrowing the scope of the data query and improving the speed of querying data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;

FIG2 is a flow chart of a data processing method provided in an embodiment of the present application;

3 is a schematic diagram of storing data to be processed in a memory table of a storage partition corresponding to a primary key of the data to be processed, provided by an embodiment of the present application;

FIG4 is a schematic diagram of the structure of an SST file provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a block in an SST file provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a data block provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of a leaf index block provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of a root index block provided in an embodiment of the present application;

FIG9 is a schematic diagram of a merged SST file provided in an embodiment of the present application;

FIG10 is a schematic diagram of a data query process provided by an embodiment of the present application;

FIG11 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to facilitate the understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

For data with large amounts of data, such as log data, it is usually stored in a data management platform dedicated to storing data. When users need to use data, they query the pre-stored log data that meets the query conditions in the data management platform and obtain the query results fed back by the data management platform. However, the efficiency of querying data using the data management platform is currently low and the latency is long.

Based on this, the present application provides a data processing method, device, equipment and storage medium, which first obtains the data to be processed, and stores the data to be processed in the memory cache of the storage partition corresponding to the primary key of the data to be processed; in response to satisfying the storage condition, at least part of the data to be processed stored in the memory cache of the storage partition is stored in the disk storage space belonging to the storage partition. In this way, the storage partition storing the data to be processed can be determined based on the primary key of the data to be processed, which facilitates the subsequent data query in the storage partition corresponding to the target primary key of the query, thereby narrowing the scope of the data query and improving the speed of querying data.

In order to facilitate understanding of the data processing method provided in the embodiment of the present application, the following is an explanation in conjunction with the scenario example shown in Figure 1. Referring to Figure 1, this figure is a schematic diagram of the framework of an exemplary application scenario provided in the embodiment of the present application.

As an example, the data processing method provided in the embodiment of the present application is applied to a data management platform. The data management platform can realize the storage and management of a large amount of data. The data management platform can obtain the pending data that needs to be stored reported by the reporting end, and store the pending data. After obtaining the query instruction later, the pending data obtained by the query is fed back. The data management platform first writes the reported pending data into the message queue. The data management platform then reads the pending data from the message queue, and stores the pending data in the memory cache of the storage partition corresponding to the primary key of the pending data. In this way, in the subsequent data query process, data can be queried in the corresponding storage partition according to the target primary key included in the query instruction, narrowing the scope of the query data and improving the query efficiency. When the storage condition is met, for example, the amount of data to be processed stored in the memory cache of the storage partition is greater than the data amount threshold, all or part of the pending data stored in the memory cache of the storage partition is stored in the disk storage space of the storage partition. In this way, a two-level storage structure of the memory cache and the disk storage space can be formed. For data with a later storage time, it may be stored in the memory cache, and you can query it first in the memory cache. If the query fails, you can query it in the disk storage space. For data with an earlier storage time, it may have been stored in the disk storage space, and you can query it directly in the disk storage space. In this way, you can quickly query data with different storage times, avoiding the problem of being unable to query due to not being stored in the disk storage space.

In order to facilitate understanding of the technical solution provided by the embodiment of the present application, a data processing method provided by the embodiment of the present application is described below in conjunction with the accompanying drawings. First, the storage process of the data to be processed is introduced. Referring to FIG. 2, the figure is a flow chart of a data processing method provided by the embodiment of the present application. The data processing method includes S201-S203.

S201: Obtain data to be processed.

The acquired data to be processed is data reported by the reporting end and needs to be stored by the data management platform. The embodiment of the present application does not limit the data type of the data to be processed. As an example, the data to be processed is log data. Log data is data generated by the process of users using the service for implementing operation and maintenance. As another example, the data to be processed is video detection data recording a specific location. The reporting end is a network node that processes the data to be processed. For example, the data to be processed is log data of a live broadcast service, and the log data is generated by the network node providing the live broadcast service and reported to the data management platform. The reporting end is each network node providing the live broadcast service. For another example, the data to be processed is detection data. The reporting end can be a detection device that generates video detection data.

The embodiment of the present application does not limit the source of the data to be processed. In a possible implementation, the data to be processed is obtained from the message queue. The reporting end sends the reported data to be processed to the message queue so that the data management platform obtains the data to be processed from the message queue. For example, the message queue is BMQ (a stream-batch integrated message queue based on a storage-computing separation architecture). The message queue includes multiple queue partitions. The data to be processed is randomly distributed in multiple queue partitions of the message queue. The embodiment of the present application does not limit the specific implementation method of allocating the data to be processed in the message queue. As an example, when the reporting end reports the data to be processed, a random algorithm can be used to implement the random distribution of the data to be processed in multiple queue partitions of the message queue. In this way, it is possible to avoid allocating the queue partitions for the distribution of the data to be processed according to the primary key of the data to be processed, and the amount of data to be processed in some queue partitions in the message queue is large, resulting in excessive load on some queue partitions and the problem of losing data to be processed.

In another possible implementation, the data to be processed is obtained from other data management platforms or systems. For example, the data to be processed is obtained from Kafka (a messaging system). As an example, the Flink (a framework and distributed processing engine) Kafka connector can be used to obtain the data to be processed. The Flink Kafka connector is used to establish a connection between the data management platform and Kafka.

The embodiment of the present application also does not limit the method of obtaining the data to be processed. As an example, obtaining the data to be processed is implemented by a Flink task. The data management platform establishes a data source (source) operator for the Flink task. The data source operator is used to enable the data management platform to obtain the data to be processed from a message queue or kafka.

S202: Storing the data to be processed in the memory cache of the storage partition corresponding to the primary key of the data to be processed.

A storage partition is a pre-divided storage space for storing data to be processed. Each storage partition has a corresponding primary key. A storage partition is used to store data to be processed whose primary key is the primary key corresponding to the storage partition. A storage partition includes a memory cache and a disk storage space. The data to be processed is first stored in the memory cache of the storage partition, and then stored in the disk storage space of the storage partition after the storage conditions are met. In one possible implementation, the data to be processed can be stored in a memory table of the memory cache of the storage partition. The embodiments of the present application do not limit the storage engine used by the storage partition. As an example, a storage partition can use an LSM-tree (Log-Structured Merge Tree) storage engine to implement storage of data to be processed.

As an example, the data management platform stores the data to be processed by Flink tasks. The data management platform establishes a data output (sink) operator. The data source operator established by the data management platform is used to obtain the data to be processed from the message queue or kafka, and according to Flink's custom routing mechanism, the data to be processed is routed to the data output operator responsible for writing the data to the storage partition corresponding to the primary key of the data to be processed according to the primary key of the data to be processed. The data output operator outputs the obtained data to be processed to the memory cache of the storage partition. The custom routing mechanism predetermines the correspondence between the storage partition and the primary key. In this way, the data to be processed that is randomly distributed in the queue partition of the message queue can be reorganized and stored according to the primary key, so that when the data is queried later, the storage partition corresponding to the target primary key can be queried, which narrows the scope of the query data and improves the efficiency of the query data.

The number of data source operators established by the data management platform can be one or more, and the number of data output operators established is consistent with the number of storage partitions. Taking obtaining unprocessed data from a message queue as an example, the message queue has multiple queue partitions, and each data source operator is used to read one or more queue partitions of the message queue. The number of data source operators is less than or equal to the number of queue partitions. The number of data source operators is independent of the number of data output operators. In some possible implementations, the number of data output operators is greater than the number of data source operators.

As an example, the data to be processed is obtained from the message queue. Referring to FIG3, this figure is a schematic diagram of a memory cache of a storage partition corresponding to the primary key of the data to be processed provided by an embodiment of the present application. The data management platform establishes two data source operators. The two data source operators obtain the data to be processed from the queue partition of the message queue, and determine the storage partition to be stored in which the data to be processed is to be stored based on the primary key of the data to be processed and the correspondence between the storage partition and the primary key predefined by the custom routing mechanism. The data source operator sends the data to be processed to the data output operator responsible for writing data to the storage partition. The data output operator stores the obtained data to be processed in the memory cache of the storage partition.

S203: In response to the storage condition being met, storing at least a portion of the to-be-processed data stored in the memory table of the storage partition into the disk storage space of the storage partition.

The storage resources of the memory cache of the storage partition are limited. When the storage condition is met, the data to be processed stored in the memory cache of the storage partition is stored in the disk storage space of the storage partition. As an example, the storage condition is that the data to be processed stored in the memory cache reaches a data volume threshold. As another example, the storage condition is that the storage time of the data to be processed stored in the memory table exceeds a time threshold. The storage condition can be set based on storage needs, and the embodiments of the present application do not limit this.

In addition, when the storage conditions are met, all or part of the pending data stored in the memory cache of the storage partition can be stored in the disk storage space of the storage partition. For example, as an example, the storage condition is that the pending data stored in the memory cache reaches the data volume threshold, and the data volume threshold is 1G (Gigabyte). 1.5G of pending data is stored in the memory cache. 1G of pending data in the memory cache is stored in the disk storage space of the storage partition, and 0.5G of pending data remains.

Based on the above S201-S203, it can be known that the storage partition for storing the data to be processed can be determined based on the primary key, so that when querying data later, data query can be performed in the storage partition corresponding to the target primary key of the query, thereby narrowing the scope of the data query and improving the speed of querying data. In addition, based on the two-level storage structure of memory cache and disk storage space included in the storage partition, it is possible to query the data to be processed stored at different times, improve the efficiency of querying data, and reduce latency.

In a possible implementation, the data to be processed is stored in an SST (sorted string table) file in a disk storage space.

During a disk write, that is, during a process of storing the data to be processed in the disk storage space, a sorted string table file is generated. The sorted string table file stores the data to be processed stored in the disk storage space this time. The disk storage file of a storage partition can include one or more SST files. The SST files belonging to each storage partition are independent of each other.

The generated SST files can be stored in HDFS (Hadoop Distributed File System) to reduce the cost of storing data.

SST files are stored in blocks. SST files include multiple data blocks. Each of the data blocks is used to store data to be processed in the format of key-value pairs. The data to be processed stored in the data block is sorted according to the primary key. Multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block. For example, the data to be processed are sorted from small to large according to the primary key. For example, the SST file includes data block A and data block B. Data block A includes data to be processed with primary key values from 90 to 100. Data block B includes data to be processed with primary key values from 95 to 105. The data to be processed stored in data block A and data block B are stored in order from small to large according to the primary key. The primary key of the first data to be processed in data block A is 90, which is smaller than the primary key of the first data to be processed in data block B, which is 95. In the SST file, data block A is sorted before data block B.

This makes it easier to build a clustered index when querying data later, and to query the data of the target primary key in an orderly manner, thereby improving the efficiency of data query.

In addition, the SST file can also include an index block. The index block is used to indicate the primary key of the first data stored in each data block and the storage position of the data block in the sorted string table file. According to the index block, the data block storing the data to be processed of the target primary key can be determined, and the storage position of the data block in the SST file can be determined, and then the data block can be read to query and obtain the data to be processed of the target primary key.

In a scenario where the number of data blocks in an SST file is large, the index block of the SST file includes multiple leaf index blocks and a root index block. A leaf index block is used to store the index information of the corresponding one or more data blocks. The root index block is used to store the index information of all leaf index blocks in the SST file. The leaf index block stores the primary key of the first data stored in each corresponding data block and the storage position of each corresponding data block in the sorted string table file. The root index block stores the first primary key of the data stored in each leaf index block and the storage position of the leaf index block in the sorted string table file.

As an example, an embodiment of the present application provides an SST file. The format of the SST file provided in the embodiment of the present application is introduced below.

Refer to Figure 4, which is a schematic diagram of the structure of an SST file provided in an embodiment of the present application. The SST file includes a data block (Data Block), a leaf index block (Leaf index block), a root index block (Root index block), a meta-info block (Meta-Info Block) and additional data (Fixed-Length Meta-Info Trailer). The additional data includes the file attribute information required for opening the SST file for the first time. The root index block, meta-information block and additional data will be read and cached when the SST file is opened. The data block and leaf index block are the parts related to the query instruction that are read when the query instruction is executed.

The structure of each block (data block, leaf index block, root index block and meta information block) in the SST file is shown in Figure 5. Each block consists of block header information (Block header) and block data (block data). The block header information has five fixed fields. The five fields represent the block type (Block Type), the amount of data occupied by the block on the disk (On-Disksize), the original block size after decompression (Uncompressed Raw Size), the offset of the previous block of the same type (Previous Block Offset) and the number of data entries in the block (Number of Entries). The block type is one of the data block, leaf index block, root index block and meta information block. Block data is the specific information stored in the block. The block data of the data block is the data to be processed. The block data of the leaf index block and the root index block is the index. The block data of the meta information block is metadata.

The specific structures of data blocks, leaf index blocks and root index blocks are introduced below.

Refer to Figure 6, which is a structural diagram of a data block provided in an embodiment of the present application. A data block is the smallest data storage unit in an SST file. A data block includes block header information (Block Header) and multiple pieces of data to be processed. The data to be processed is stored in the form of key-value pairs. As an example, a piece of data to be processed includes a key length (Key Length), a value length (Value Length), a key (Key) and a value (Value). The data to be processed stored inside the data block is compressed. When reading the data to be processed in the data block, the data block needs to be decompressed into the memory and then read.

The data to be processed in a data block are arranged in order according to the primary key. The data blocks in an SST file are also arranged in order according to the primary key of the first data to be processed stored in the data block.

Referring to FIG. 7 , this figure is a schematic diagram of the structure of a leaf index block provided in an embodiment of the present application. A leaf index block contains index information of one or more data blocks. A leaf index block consists of block header information (Block Header), a meta index (Meta index) and multiple leaf indexes (Entries). Among them, the meta index is used to directly locate the storage location of a single leaf index according to the subscript, thereby performing a binary search on the leaf index. The meta index includes: the number of index entries in the leaf index block (Number of Index Entries), the offset of each leaf index relative to the first leaf index (Relative Offsets to Entries) and the total size of all leaf indexes (Total Size of Entries). Each leaf index contains index information of a data block. Each leaf index contains the offset of the data block (Data Block Offset), the data size occupied by the data block on the disk (Data Block On-Disk Size), and the first key of the data block (First Key in the Data Block).

Refer to Figure 8, which is a schematic diagram of the structure of a root index block provided in an embodiment of the present application. The root index block contains the index information of all leaf indexes in the SST file. A root index block consists of block header information (Block header) and several root indexes (Entries). Each root index contains the index information of a leaf index. Each root index includes the offset of the leaf index (Leaf Block Offset), the size of the leaf index on the disk (Leaf Block On-DiskSize), and the first key of the leaf index block (First Key in the Leaf Block).

SST files are immutable read-only files. Once written, they cannot be appended or changed. With multiple writes to disk, the number of SST files in a storage partition gradually increases. In subsequent queries, a query may involve multiple SST files, which may reduce query efficiency.

In order to prevent the number of SST files from growing continuously over time, the SST files can be merged to combine multiple SST files into an ordered SST file, thereby increasing the aggregation of the data and facilitating subsequent data queries.

It should be noted that in some possible implementations, the data to be processed, such as log data, is generated in chronological order as the user uses the service. When merging SST files, the time span of the merged SST files needs to be similar to facilitate centralized data query.

For such data to be processed, an embodiment of the present application provides a data processing method, which, in addition to the above steps, further includes the following steps:

In response to the number of SST files belonging to the same storage partition whose creation times belong to a preset time period being greater than or equal to a threshold, the SST files whose creation times belong to the preset time period are merged.

For SST files belonging to a storage partition, the creation time of each SST file is obtained. The creation time of each SST file is the time when the SST file is created.

If the number of SST files whose creation time belongs to the preset time period is greater than or equal to the threshold, it means that the number of SST files whose creation time belongs to the preset time period is large and the time span is not large, and they can be merged. It should be noted that in the process of merging SST files, the unprocessed data stored in the SST files to be merged will be re-sorted according to the primary key, and the unprocessed data included in the merged SST files will still be sorted according to the primary key.

The preset time period can be pre-set and adjusted over time. As an example, see Figure 9, which is a schematic diagram of merging SST files provided in an embodiment of the present application. The time period is divided into multiple preset time periods, such as one preset time period for each hour. If the number of SST files belonging to a preset time period is greater than or equal to a threshold, the SST files that belong to the same preset time period at the creation time are merged. Then merge multiple consecutive preset time periods as the updated preset time period. For example, merge four adjacent one-hour periods to obtain an updated preset time period of four hours. If the number of SST files that belong to the updated preset time period is greater than or equal to a threshold, then merge the SST files that belong to the same preset time period at the creation time. And so on, until the stop merging condition is met, stop merging the SST files. The stop merging condition is a pre-set condition for stopping merging SST files. The stop merging condition is, for example, that the preset time period exceeds the limit. For example, the preset time period cannot exceed 8 hours. Another example of the stop merging condition is that the size of the merged SST file exceeds the limit. For example, the size of the merged SST file exceeds 4GB (Giga Byte). In this way, as the preset time period is adjusted based on the passage of time, the SST files can be gradually merged, the aggregation of data is increased, and subsequent data query is facilitated.

The above is an introduction to the storage process of the data to be processed. The following is an explanation of the data query process based on the above storage method.

After the data to be processed is stored, it is possible to query the data to be processed. The embodiment of the present application provides a data processing method, which, in addition to the above steps, also includes the following steps:

A1: Obtain a query instruction, where the query instruction includes a target primary key.

The query instruction is used to query the stored target primary key corresponding to the to-be-processed data. The query instruction can include one or more target primary keys.

The embodiment of the present application does not limit the triggering method of the query instruction. In a possible implementation method, refer to Figure 10, which is a flow chart of a data query provided by an embodiment of the present application. The user writes the query instruction into redis (Remote Dictionary Server) through the client that implements the data query function. The data management platform periodically queries redis to obtain the query instruction. The query instruction includes a target primary key. Among them, the target primary key is the primary key of the data to be processed that needs to be queried.

A2: query the to-be-processed data corresponding to the target primary key in one or more of the memory cache of the storage partition corresponding to the target primary key and the disk storage space of the storage partition corresponding to the target primary key to obtain a query result.

Based on the target primary key included in the query instruction, one or more of the memory cache and disk storage space of the storage partition corresponding to the target primary key can be queried. Performing data query in the storage partition corresponding to the target primary key can achieve query within a smaller data range, thereby improving the efficiency of data query.

The embodiments of the present application do not limit the storage space for query data, and can be set based on the needs of the query. For example, in one possible implementation, the pending data corresponding to the target primary key can be queried jointly in the memory cache and disk storage space of the storage partition. For another example, in another possible implementation, the query instruction also includes a storage time range. The storage time range is the time range when the pending data is reported to the data management platform. When the time difference for transmitting the pending data is ignored, the storage time range is the time range when the pending data is generated. In some cases, the storage time range is a single moment, or a time period. Based on the storage time range, it can be determined whether the pending data corresponding to the target primary key is stored in the memory cache of the storage partition or in the disk storage space of the storage partition.

As an example, the relationship between the storage time range and the disk drop time is determined. The disk drop time is the time when at least part of the to-be-processed data in the memory cache is stored in the disk storage space for the storage partition corresponding to the target primary key, which is closest to the current time.

If the storage time range is later than the disk drop time, it means that the pending data to be queried may still be stored in the memory cache. For example, the storage time range is 10:00-10:05. The disk drop time is 9:30. This means that the pending data to be queried is stored in the data management platform during the period of 10:00-10:05, and is not stored in the disk storage space at 9:30. Query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key. If the pending data corresponding to the target primary key is obtained through the query, the pending data obtained through the query is used as the query result to obtain the query result. If the pending data corresponding to the target primary key is not found, query the pending data corresponding to the target primary key in the disk storage space of the storage partition corresponding to the target primary key. If the pending data corresponding to the target primary key can be queried in the disk storage space, the pending data obtained through the query is used as the query result. If the pending data corresponding to the target primary key cannot be queried, the query result is empty.

If the storage time range is earlier than the disk-stack time, it means that the pending data to be queried may have been stored in the disk storage space. Query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key.

As an example, an embodiment of the present application provides a specific implementation method for querying pending data from a memory cache. In a possible implementation method, a memory table of the memory cache stores pending data. The pending data stored in the memory table is sorted based on the primary key. The subscript of the memory table is the primary key of the pending data. The query obtains the pending data whose subscript is the same as the target primary key. As an example, when querying data on the memory table, a corresponding iterator can be created for the memory table. The iterator is used to locate the starting position of the pending data whose primary key is the target primary key in the memory table. The data management platform establishes a minimum heap to manage each iterator. The minimum heap reads the pending data whose primary key is the target primary key in the memory table from the starting position located by the iterator, in sequence, until the primary key of the read pending data is not the target primary key, and obtains the pending data corresponding to the target primary key, that is, the query result.

As another example, an embodiment of the present application provides a specific implementation method for querying the data to be processed from the disk storage space. Taking the SST file shown in Figure 4 above as an example, since the data to be processed stored in the SST file is sorted based on the primary key, in the process of querying the data, the value of the smallest primary key stored in the SST file can be queried. If the primary keys of the data to be processed stored in the SST file are sorted from small to large, the smallest primary key stored in the SST file is the primary key of the first data to be processed stored in the first data block. If the smallest primary key of the SST file is larger than the target primary key, it means that the SST file does not include the data to be processed corresponding to the target primary key, and there is no need to query in the SST file.

Referring to FIG. 10 , a query result obtained by querying the SST file for the data to be processed corresponding to the target primary key is introduced in detail.

Get the query instruction sent by the client capable of implementing the data query function to the remote dictionary service. The query instruction includes the target primary key. The remote dictionary service sends the query instruction to the data management platform. The data management platform searches for the pending data corresponding to the target primary key in the memory table of the memory cache of the storage partition corresponding to the target primary key and the SST file of the disk storage space.

In the root index block of the SST file, the first key of each leaf index block of the SST file and the storage position of each leaf index block in the SST file are stored. Read the root index block of the SST file, query the target leaf index block corresponding to the target primary key, and the first storage position of the target leaf index block in the SST file. The first storage position can specifically be the offset of the target leaf index block in the SST file. Then read the target leaf index block from the first storage position, and query the second storage position of the target data block corresponding to the target primary key stored in the target leaf index block. The second storage position is the starting position of the target data block in the SST file. Specifically, the second storage position can be the offset of the target data block in the SST file. The data in the SST file is sorted from small to large according to the primary key. Starting from the second storage position, read the data to be processed whose primary key is the target primary key in sequence to generate the query result. In this way, based on the two-layer index structure of the root index block and the leaf index block, the data to be queried can be located in the SST file more quickly, and based on the sorting by the primary key, the data to be processed whose primary key is the target primary key can be obtained more concentratedly in the SST file, thereby realizing fast data query.

As an example, the data management platform can create an iterator for each SST file that needs to be queried. The iterator is used to locate the starting position of the data to be processed whose primary key is the target primary key in the SST file. The data management platform establishes a minimum heap to manage the iterators of each SST file. It should be noted that if the data to be processed of the target primary key is queried in the memory table and the SST file, the minimum heap manages the iterators of each SST file and the iterator of the memory table. The minimum heap reads the data to be processed whose primary key is the target primary key in the data block in sequence from the starting position located by the iterator until the primary key of the data to be processed read is not the target primary key, and obtains the data to be processed corresponding to the target primary key, which is the query result. Establishing a corresponding iterator for each SST file that needs to be queried for data query can realize parallel query of data and improve the speed of querying data.

In some possible implementations, the query instruction also includes a filtering condition. The filtering condition is a condition that the pending data to be queried needs to meet in addition to the target primary key. As an example, the filtering condition is that the client version number is greater than the target version number. Before obtaining the query result, the queried pending data can also be filtered based on the filtering condition to obtain filtered data that meets the filtering condition. The filtered data is used as the query result. In this way, more accurate data query can be achieved.

In a possible implementation, after obtaining the query results, a result file including the query results can be output. As an example, the SST file is stored in HDFS, and the result file is output to HDFS. As an example, the query instruction also includes an output result path. The output result path is used to indicate the location where the result file is output. Based on the output result path, the data management platform can output the result file including the query results to the location indicated by the output result path, so that the user can obtain the query results.

Based on a data processing method provided by the above method embodiment, an embodiment of the present application also provides a data processing device, which will be described below in conjunction with the accompanying drawings.

Referring to FIG11 , this figure is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG11 , the data processing device includes:

A first acquisition unit 1101 is used to acquire data to be processed;

The first storage unit 1102 is used to store the data to be processed into a memory cache of a storage partition corresponding to a primary key of the data to be processed;

The second storage unit 1103 is used to store at least part of the to-be-processed data stored in the memory cache of the storage partition into the disk storage space of the storage partition in response to the storage condition being met.

In a possible implementation, the data to be processed is stored in a sorted string table file in a disk storage space, the sorted string table file includes multiple data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data block is sorted according to a primary key, and the multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block.

In a possible implementation, the sorted string table file further includes an index block, where the index block is used to indicate a primary key of a first to-be-processed data stored in each data block and a storage position of the data block in the sorted string table file.

In one possible implementation, the index block includes multiple leaf index blocks and a root index block, each leaf index block stores the primary key of the first data stored in the corresponding data block and the storage position of the corresponding data block in the sorted string table file, and the root index block stores the first primary key of the data stored in each leaf index block and the storage position of the leaf index block in the sorted string table file.

In a possible implementation manner, the device further includes:

The merging unit is used to merge the sorted string table files whose creation time belongs to the preset time period in response to the number of sorted string table files belonging to the same storage partition and whose creation time belongs to the preset time period being greater than or equal to a threshold.

In a possible implementation manner, the device further includes:

A second acquisition unit, used to acquire a query instruction, wherein the query instruction includes a target primary key;

A query unit is used to query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key, and/or to query the pending data corresponding to the target primary key in the disk storage space of the storage partition corresponding to the target primary key to obtain a query result.

In a possible implementation, the query instruction also includes a storage time range, and the query unit is used to query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key if the storage time range is later than the disk drop time, and the disk drop time is the storage partition corresponding to the target primary key, which is closest to the current time, and the time when at least part of the pending data in the memory cache is stored in the disk storage space; if the pending data corresponding to the target primary key is queried, the queried pending data corresponding to the target primary key is used as the query result to obtain the query result; if the pending data corresponding to the target primary key is not queried, the pending data corresponding to the target primary key is queried in the disk storage space of the storage partition corresponding to the target primary key to obtain the query result.

In a possible implementation, the data to be processed is stored in a sorted string table file of a disk storage space, the sorted string table file includes a plurality of data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data block is sorted according to a primary key, and the plurality of data blocks are sorted according to the primary key of the first data to be processed stored in each data block, the sorted string table file also includes a plurality of leaf index blocks and a root index block, each of the leaf index blocks stores the primary key of the first data stored in the corresponding data block and the storage position of the corresponding data block in the sorted string table file, the root index block stores the first primary key of the data stored in each leaf index block and the storage position of the leaf index block in the sorted string table file, the query unit is used to query the data to be processed corresponding to the target primary key in the disk storage space of the storage partition corresponding to the target primary key, and obtain the query result, including:

The query unit is used to query the first storage position of the target leaf index block corresponding to the target primary key in the root index block of the sorted string table file of the storage partition corresponding to the target primary key, and the first storage position is the starting position for storing the target leaf index block; based on the first storage position, query the second storage position of the target data block corresponding to the target primary key in the target leaf index block, and the second storage position is the starting position for storing the target data block; starting from the second storage position, read the data to be processed whose primary key is the target primary key in sequence.

In a possible implementation manner, the query instruction further includes a filtering condition, and the device further includes:

A filtering unit, used to filter the to-be-processed data corresponding to the target primary key obtained by the query to obtain the filtered data that meets the filtering condition;

The query unit is used to obtain the query result, including:

The query unit is used to use the filtered data as a query result.

In a possible implementation, the first acquiring unit 1101 is used to acquire the data to be processed from the message queue, and the data to be processed is randomly distributed in multiple queue partitions of the message queue.

Based on a data processing method provided by the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any of the above embodiments.

Reference is made to FIG12, which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing an embodiment of the present application. The terminal device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. The electronic device shown in FIG12 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present application.

As shown in FIG12 , the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are also stored. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input/output (I/O) interface 1205 is also connected to the bus 1204.

Typically, the following devices may be connected to the I/O interface 1205: input devices 1208 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication device 1209 may allow the electronic device 1200 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 12 shows an electronic device 1200 with various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

The electronic device provided in the embodiment of the present application and the data processing method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

Based on a data processing method provided in the above method embodiment, an embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the program is executed by a processor, the data processing method as described in any of the above embodiments is implemented.

It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the data processing method.

Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or flow chart, and the combination of the boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in this application may be implemented by software or hardware. The name of a unit/module does not limit the unit itself in some cases. For example, a voice data acquisition module may also be described as a "data acquisition module".

The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present application, [Example 1] provides a data processing method, the method comprising:

Get the data to be processed;

Storing the data to be processed in a memory cache of a storage partition corresponding to the primary key of the data to be processed;

In response to the storage condition being met, at least a portion of the to-be-processed data stored in the memory cache of the storage partition is stored in the disk storage space of the storage partition.

According to one or more embodiments of the present application, [Example 2] provides a data processing method, wherein the data to be processed is stored in a sorted string table file in a disk storage space, the sorted string table file includes multiple data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data block is sorted according to a primary key, and the multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block.

According to one or more embodiments of the present application, [Example Three] provides a data processing method, wherein the sorted string table file also includes an index block, which is used to indicate the primary key of the first data to be processed stored in each data block and the storage position of the data block in the sorted string table file.

According to one or more embodiments of the present application, [Example 4] provides a data processing method, wherein the index block includes multiple leaf index blocks and a root index block, each leaf index block stores the primary key of the first data stored in the corresponding data block and the storage position of the corresponding data block in the sorted string table file, and the root index block stores the first primary key of the data stored in each leaf index block and the storage position of the leaf index block in the sorted string table file.

According to one or more embodiments of the present application, [Example 5] provides a data processing method, the method further comprising:

In response to the number of sorted string table files belonging to the same storage partition and whose creation times belong to a preset time period being greater than or equal to a threshold, the sorted string table files whose creation times belong to the preset time period are merged.

According to one or more embodiments of the present application, [Example 6] provides a data processing method, the method further comprising:

Obtaining a query instruction, wherein the query instruction includes a target primary key;

The pending data corresponding to the target primary key is queried in the memory cache of the storage partition corresponding to the target primary key, and/or the pending data corresponding to the target primary key is queried in the disk storage space of the storage partition corresponding to the target primary key to obtain a query result.

According to one or more embodiments of the present application, [Example 7] provides a data processing method, wherein the query instruction further includes a storage time range, and the querying of the to-be-processed data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key, and/or the querying of the to-be-processed data corresponding to the target primary key in the disk storage space of the storage partition corresponding to the target primary key, to obtain a query result, including:

If the storage time range is later than the disk-falling time, query the memory cache of the storage partition corresponding to the target primary key for the data to be processed corresponding to the target primary key, and the disk-falling time is the time when at least part of the data to be processed in the memory cache is stored in the disk storage space of the storage partition corresponding to the target primary key, which is closest to the current time;

If the to-be-processed data corresponding to the target primary key is found, the to-be-processed data corresponding to the target primary key is taken as the query result to obtain the query result;

If the to-be-processed data corresponding to the target primary key is not found, the to-be-processed data corresponding to the target primary key is searched in the disk storage space of the storage partition corresponding to the target primary key to obtain a query result.

According to one or more embodiments of the present application, [Example 8] provides a data processing method, wherein the data to be processed is stored in a sorted string table file in a disk storage space, the sorted string table file includes data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data block is sorted according to the primary key, and the multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block, and the data to be processed corresponding to the target primary key is queried in the disk storage space of the storage partition corresponding to the target primary key to obtain the query result, including:

In the root index block of the sorted string table file of the storage partition corresponding to the target primary key, query the first storage position of the target leaf index block corresponding to the target primary key, the first storage position is the starting position for storing the target leaf index block, and the root index block stores the correspondence between the target primary key and the target leaf index block and the first storage position of the target leaf index block;

Based on the first storage position, querying the second storage position of the target data block corresponding to the target primary key in the target leaf index block, where the second storage position is the starting position for storing the target data block, and the target leaf index block stores the correspondence between the target primary key and the target data block and the second storage position of the target data block;

Starting from the second storage location, the data to be processed whose primary key is the target primary key is read in sequence.

According to one or more embodiments of the present application, [Example 9] provides a data processing method, wherein the query instruction further includes a filtering condition, and before obtaining the query result, the method further includes:

Filter the to-be-processed data corresponding to the target primary key obtained by the query to obtain the filtered data that meets the filtering condition;

The query result obtained includes:

The filtered data is used as the query result.

According to one or more embodiments of the present application, [Example 10] provides a data processing method, wherein obtaining the data to be processed includes:

The data to be processed is obtained from a message queue, where the data to be processed is randomly distributed in a plurality of queue partitions of the message queue.

According to one or more embodiments of the present application, [Example 11] provides a data processing device, the device comprising:

A first acquisition unit, used to acquire data to be processed;

A first storage unit, used to store the data to be processed into a memory cache of a storage partition corresponding to a primary key of the data to be processed;

The second storage unit is used to store at least part of the to-be-processed data stored in the memory cache of the storage partition into the disk storage space of the storage partition in response to satisfying the storage condition.

According to one or more embodiments of the present application, [Example 12] provides a data processing device, wherein the data to be processed is stored in a sorted string table file in a disk storage space, the sorted string table file includes multiple data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data block is sorted according to a primary key, and the multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block.

According to one or more embodiments of the present application, [Example 13] provides a data processing device, wherein the sorted string table file also includes an index block, which is used to indicate the primary key of the first data to be processed stored in each data block and the storage position of the data block in the sorted string table file.

According to one or more embodiments of the present application, [Example 14] provides a data processing device, wherein the index block includes multiple leaf index blocks and a root index block, each leaf index block stores the primary key of the first data stored in the corresponding data block and the storage position of the corresponding data block in the sorted string table file, and the root index block stores the first primary key of the data stored in each leaf index block and the storage position of the leaf index block in the sorted string table file.

According to one or more embodiments of the present application, [Example 15] provides a data processing device, the device further comprising:

The merging unit is used to merge the sorted string table files whose creation time belongs to the preset time period in response to the number of sorted string table files belonging to the same storage partition and whose creation time belongs to the preset time period being greater than or equal to a threshold.

According to one or more embodiments of the present application, [Example 16] provides a data processing device, the device further comprising:

A second acquisition unit, used to acquire a query instruction, wherein the query instruction includes a target primary key;

A query unit is used to query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key, and/or to query the pending data corresponding to the target primary key in the disk storage space of the storage partition corresponding to the target primary key to obtain a query result.

According to one or more embodiments of the present application, [Example 17] provides a data processing device, wherein the query instruction further includes a storage time range, and the query unit is used to query the to-be-processed data corresponding to the target primary key in at least one of the memory cache and the disk storage space of the storage partition corresponding to the target primary key, and obtain the query result, including:

The query unit is used to query the pending data corresponding to the target primary key in the memory cache of the storage partition corresponding to the target primary key if the storage time range is later than the disk placement time, and the disk placement time is the time when at least part of the pending data in the memory cache is stored in the disk storage space of the storage partition corresponding to the target primary key, which is closest to the current time; if the pending data corresponding to the target primary key is not found, the disk storage space of the storage partition corresponding to the target primary key is queried for the pending data corresponding to the target primary key to obtain the query result.

According to one or more embodiments of the present application, [Example 18] provides a data processing device,

The data to be processed is stored in a sorted string table file in a disk storage space, the sorted string table file includes data blocks, each of the data blocks is used to store the data to be processed in a key-value pair format, the data to be processed stored in the data blocks are sorted according to the primary key, and the multiple data blocks are sorted according to the primary key of the first data to be processed stored in each data block, the query unit is used to query the first storage position of the target leaf index block corresponding to the target primary key in the root index block of the sorted string table file of the storage partition corresponding to the target primary key, the first storage position is the starting position for storing the target leaf index block, the root index block stores the correspondence between the target primary key and the target leaf index block and the first storage position of the target leaf index block; based on the first storage position, query the second storage position of the target data block corresponding to the target primary key in the target leaf index block, the second storage position is the starting position for storing the target data block, the target leaf index block stores the correspondence between the target primary key and the target data block and the second storage position of the target data block; starting from the second storage position, read the data to be processed whose primary key is the target primary key in sequence.

According to one or more embodiments of the present application, [Example 19] provides a data processing device, wherein the query instruction further includes a filtering condition, and the device further includes:

A filtering unit, used to filter the to-be-processed data corresponding to the target primary key obtained by the query to obtain the filtered data that meets the filtering condition;

The query unit is used to obtain the query result, including:

The query unit is used to use the filtered data as a query result.

According to one or more embodiments of the present application, [Example 20] provides a data processing device, wherein the first acquisition unit is used to obtain data to be processed from a message queue, and the data to be processed is randomly distributed in multiple queue partitions of the message queue.

According to one or more embodiments of the present application, [Example 21] provides an electronic device, including:

one or more processors;

a storage device having one or more programs stored thereon,

When the one or more programs are executed by the one or more processors, the one or more processors implement any method described in [Example 1] to [Example 10].

According to one or more embodiments of the present application, [Example 22] provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, it implements any method described in [Example 1] to [Example 10].

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And/or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character "/" generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims (13)

1. A method of data processing, the method comprising:

Acquiring data to be processed;

Storing the data to be processed into a memory cache of a storage partition corresponding to a main key of the data to be processed;

and in response to the meeting of the storage condition, storing at least part of data to be processed stored in the memory cache of the storage partition into the disk storage space of the storage partition.

2. The method of claim 1, wherein the data to be processed is stored in an ordering string table file in a disk storage space, the ordering string table file including a plurality of data blocks, each of the data blocks being configured to store the data to be processed in a key-value pair format, the data to be processed stored in the data blocks being ordered according to a primary key, the plurality of data blocks being ordered according to a primary key of a first data to be processed stored in each data block.

3. The method of claim 2, wherein the sort string table file further comprises an index block for indicating a primary key of the first data to be processed stored by each data block and a storage location of the data block in the sort string table file.

4. A method according to claim 3, wherein the index blocks comprise a plurality of leaf index blocks and root index blocks, each of the leaf index blocks storing a primary key of a first data to be processed stored by a corresponding respective data block and a corresponding storage location of the respective data block in the ordered string table file, the root index block storing a primary key of a first data stored by a respective leaf index block and a storage location of the leaf index block in the ordered string table file.

5. The method according to claim 2, wherein the method further comprises:

And in response to the fact that the number of the ordered string table files belonging to the same storage partition and the creation time belongs to a preset time period is greater than or equal to a threshold value, merging the ordered string table files of which the creation time belongs to the preset time period.

6. The method according to claim 1, wherein the method further comprises:

Acquiring a query instruction, wherein the query instruction comprises a target primary key;

And querying the data to be processed corresponding to the target main key in the memory cache of the storage partition corresponding to the target main key, and/or querying the data to be processed corresponding to the target main key in the disk storage space of the storage partition corresponding to the target main key, so as to obtain a query result.

7. The method of claim 6, wherein the querying instruction further includes a storage time range, the querying the memory cache of the storage partition corresponding to the target primary key for the data to be processed corresponding to the target primary key, and/or querying the disk storage space of the storage partition corresponding to the target primary key for the data to be processed corresponding to the target primary key, to obtain a query result, and the querying result includes:

If the storage time range is later than the disk-falling time, inquiring the data to be processed corresponding to the target main key in a memory cache of a storage partition corresponding to the target main key, wherein the disk-falling time is the time when at least part of the data to be processed in the memory cache is stored in a disk storage space, and the storage partition corresponds to the target main key and is closest to the current time;

If the data to be processed corresponding to the target main key is inquired, the inquired data to be processed corresponding to the target main key is used as an inquiry result, and the inquiry result is obtained;

And if the data to be processed corresponding to the target main key is not queried, querying the data to be processed corresponding to the target main key in the disk storage space of the storage partition corresponding to the target main key, and obtaining a query result.

8. The method according to claim 6 or 7, wherein the data to be processed is stored in an ordering string table file in a disk storage space, the ordering string table file includes data blocks, each data block is used for storing the data to be processed in a format of key value pairs, the data to be processed stored in the data blocks are ordered according to a primary key, a plurality of data blocks are ordered according to a primary key of first data to be processed stored in each data block, the data to be processed corresponding to the target primary key is queried in the disk storage space of a storage partition corresponding to the target primary key, and a query result is obtained, and the method includes:

Querying a first storage position of a target leaf index block corresponding to the target main key in a root index block of an ordered string table file of a storage partition corresponding to the target main key, wherein the first storage position is a starting position for storing the target leaf index block, and the root index block stores a corresponding relation between the target main key and the target leaf index block and the first storage position of the target leaf index block;

Inquiring a second storage position of a target data block corresponding to the target main key in the target leaf index block based on the first storage position, wherein the second storage position is a starting position for storing the target data block, and the target leaf index block stores the corresponding relation between the target main key and the target data block and the second storage position of the target data block;

And sequentially reading the data to be processed, of which the main key is the target main key, from the second storage position.

9. The method of claim 6 or 7, wherein the query instruction further comprises a filter condition, the method further comprising, prior to the obtaining the query result:

Screening the data to be processed corresponding to the target primary key obtained by inquiry to obtain filtering data meeting the filtering condition;

The obtaining the query result comprises the following steps:

and taking the filtered data as a query result.

10. The method according to any one of claims 1-7, wherein the acquiring data to be processed comprises:

And acquiring data to be processed from the message queue, wherein the data to be processed are randomly distributed in a plurality of queue partitions of the message queue.

11. A data processing apparatus, the apparatus comprising:

the first acquisition unit is used for acquiring data to be processed;

The first storage unit is used for storing the data to be processed into a memory cache of a storage partition corresponding to a main key of the data to be processed;

And the second storage unit is used for responding to the condition of meeting the storage condition and storing at least part of data to be processed stored in the memory cache of the storage partition into the disk storage space of the storage partition.

12. An electronic device, comprising:

One or more processors;

a storage device having one or more programs stored thereon,

When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-10.

13. A computer readable medium, characterized in that a computer program is stored thereon, wherein the program, when executed by a processor, implements the method according to any of claims 1-10.