CN110389967B - Data storage method, device, server and storage medium - Google Patents

Abstract

The present application discloses a data storage method, device, server and storage medium, which relate to the field of databases. The method includes: obtaining a message to be stored; extracting the data to be stored in the message to be stored; if the data to be stored meets the target query rules, storing the data to be stored in a non-relational database, and storing the data to be stored in a time series database, and the concurrent query requirements and query delay requirements of the data that meet the target query rules are higher than the data that do not meet the target query rules; if the data to be stored does not meet the target query rules, storing the data to be stored in a time series database. Since non-relational databases can provide high-concurrency and low-latency data query services, storing data with higher concurrent query requirements and query delay requirements in non-relational databases can increase the speed of subsequent data queries.

Description

Data storage method, device, server and storage medium

Technical Field

The embodiments of the present application relate to the field of databases, and in particular to a data storage method, device, server, and storage medium.

Background technique

Massive amounts of data are generated on the Internet every day. In order to effectively manage the massive amounts of data, Internet service providers will store the generated data in a database (DataBase, DB).

In the related art, relational databases are usually used to store data. Common relational databases include MySQL and Oracle. In relational databases, data is stored in the form of tables. Each field in the table is pre-defined, and the reliability and stability are high. In addition, relational databases use structured query language for data query, and support the addition, deletion, modification, and query of data in the database as well as cross-table query functions.

However, faced with a large number of data query requests, the speed of data query based on relational databases is slow (especially when performing cross-table queries) and cannot meet the query needs.

Summary of the invention

The embodiments of the present application provide a data storage method, device, server and storage medium, which can solve the problem that the data query speed in the related art is slow and cannot meet the query requirements. The technical solution is as follows:

In one aspect, an embodiment of the present application provides a data storage method, the method comprising:

Get the message to be stored;

Extracting the data to be stored in the message to be stored;

If the data to be stored meets the target query rule, the data to be stored is stored in a non-relational database, and the data to be stored is stored in a time series database, wherein the concurrent query requirement and query delay requirement of the data meeting the target query rule are higher than those of the data not meeting the target query rule;

If the data to be stored does not conform to the target query rule, the data to be stored is stored in the time series database.

On the other hand, an embodiment of the present application provides a data storage device, the device comprising:

A first acquisition module, used to acquire a message to be stored;

An extraction module, used for extracting the data to be stored in the message to be stored;

A first storage module is used to store the data to be stored in a non-relational database and store the data to be stored in a time series database when the data to be stored meets the target query rule, wherein the concurrent query requirement and the query delay requirement of the data meeting the target query rule are higher than those of the data not meeting the target query rule;

The second storage module is used to store the data to be stored in the time series database when the data to be stored does not meet the target query rule.

On the other hand, an embodiment of the present application provides a server, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the data storage method as described in the above aspects.

On the other hand, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data storage method as described in the above aspects.

On the other hand, a computer program product is provided. When the computer program product is executed on a computer, the computer is enabled to execute the data storage method as described in the above aspects.

The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

After obtaining the message to be stored, extract the data to be stored in the message to be stored, and detect whether the data to be stored meets the target query rules. When the data to be stored meets the target query rules, store the data to be stored in the non-relational database and the time series database. When the data to be stored does not meet the target query rules, store the data to be stored in the time series database. Since non-relational databases can provide high concurrency and low-latency data query services, storing data with high concurrent query requirements and query latency requirements in non-relational databases can improve the speed of subsequent data queries. At the same time, storing data with low concurrent query requirements and query latency requirements in the time series database facilitates multi-dimensional combined queries in subsequent data queries.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 shows a system architecture diagram of a data classification storage query system provided by an exemplary embodiment of the present application;

FIG2 is a schematic diagram showing the principle of a data storage method provided in an embodiment of the present application;

FIG3 shows a flow chart of a data storage method provided by an exemplary embodiment of the present application;

FIG4 shows a flow chart of a data storage method provided by another exemplary embodiment of the present application;

FIG5 is a schematic diagram of an interface for manually configuring a dimension set process;

FIG6 is a flow chart showing a process of storing log messages in a database according to an embodiment;

FIG. 7 is a flow chart showing a process of querying log messages from a database according to an embodiment;

FIG8 is a flowchart of a dimension set generation process provided by an exemplary embodiment;

9 to 11 are schematic diagrams of data compression formats;

FIG12 is a structural block diagram of a data storage device provided by an exemplary embodiment of the present application;

FIG. 13 shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

To facilitate understanding, some terms involved in the embodiments of the present application are briefly introduced below.

Non-relational database (Not only SQL, NoSQL): refers to a non-relational, distributed, and non-ACID database design pattern. Unlike relational databases that use tables to store data, non-relational databases use column-oriented structures, key-value (KV) structures, and document-oriented structures to store data. The non-relational database in the embodiment of the present application uses a KV structure to store data, which can provide high concurrency and low latency data query services.

Time series database: Also known as time series database, it is mainly used to store and process data with time tags. Data with time tags is also called time series data. It is often used to store massive data that changes over time. Common time series databases include Druid, TimescaleDB, KairosDB, InfluxDB, etc.

Please refer to FIG. 1 , which shows a system structure diagram of a data classification storage query system provided by an exemplary embodiment of the present application. The system includes: a classification server 11 , a first database server 12 and a second database server 13 .

The classification server 11 is a server for providing data classification storage and data classification query services, which can be a server, a server cluster composed of several servers, or a cloud computing center. Among them, data classification storage refers to the process of storing the data to be stored in at least one database according to the data classification standard; data classification query refers to the process of routing the query request to the specified database according to the data classification standard and providing feedback on the data query results fed back by the database.

In a possible implementation manner, the classification server 11 corresponds to a visualization operation platform for receiving data classification standards set by a user, or receiving a query request triggered by a user.

The classification server 11 is connected to the first database server 12 and the second database server 13 via a wired network or a wireless network.

The first database server 12 is a server with a non-relational database, which can be a single server, a server cluster consisting of several servers, or a cloud computing center. In the embodiment of the present application, the first database server 12 is used to store and query data with high concurrency and low latency query requirements, and the non-relational database uses KV format to store data.

The second database server 13 is a server with a time-series database, which can be a single server, a server cluster consisting of several servers, or a cloud computing center. In the embodiment of the present application, the second database server 13 is used to store and query multi-dimensional combined data.

In a possible application scenario, after the classification server 11 obtains the message to be stored, it detects whether the data to be stored in the message to be stored has data with high concurrency and low latency query requirements. If so, the classification server 11 stores the data to be stored in the first database server 12 and the second database server 13 at the same time; if not, the classification server 11 only stores the data to be stored in the second database server 13. Since the data with high concurrency and low latency query requirements are stored in the first database server 12, when the classification server 11 subsequently receives a query request for data with high concurrency and low latency query requirements, high concurrency and low latency data query can be implemented through the first database server 12; and when a query request for multi-dimensional combination data is received, data query for multi-dimensional combination data can be implemented through the second database server 13.

As shown in FIG2 , in the data storage method provided by the embodiment of the present application, after the classification server 21 obtains the message to be stored from the message queue 22, it detects whether the data to be stored contained in the message to be stored meets the high concurrency and low latency query requirements; if it meets, in order to achieve high concurrency and low latency data query, and historical data can be traced back, the classification server 21 stores the data to be stored in the non-relational database 23 (high concurrency and low latency data query) and the time series database 24 (historical data can be traced back) at the same time; if it does not meet, in order to achieve multi-dimensional combination data query, the classification server 21 stores the data to be stored in the time series database 24. When a query request 25 is subsequently received, the classification server 21 detects whether the data to be queried by the query request 25 meets the high concurrency and low latency query requirements; if it meets, the classification server 21 will route the query request to the non-relational database 23, and the non-relational database 23 will provide high concurrency and low latency data query services; if it does not meet, the classification server 21 will route the query request to the time series database 24, and the time series database 24 will provide multi-dimensional combination data query services.

Compared with the use of a single relational database to store data in the related art, which cannot meet the data query requirements, the embodiment of the present application creatively proposes a data storage query architecture of "classification server + dual database", in which the classification server classifies the data to be stored into the corresponding database according to "whether there is a high concurrency and low latency data query requirement", and further routes the query request to the corresponding database, which provides data query services, thus realizing high concurrency and low latency data query, and realizing multi-dimensional combined data query. The following uses an illustrative embodiment to illustrate the data storage and query process.

Please refer to Fig. 3, which shows a flow chart of a data storage method provided by an exemplary embodiment of the present application. This embodiment takes the method used in the classification server 11 shown in Fig. 1 as an example for explanation, and the method includes the following steps.

Step 301, obtaining the message to be stored.

In a possible implementation, the classification server obtains the message to be stored from the message queue, and the message queue stores the log message generated during the operation of the system. For example, for a payment system, the log message can be a payment success log message or a payment failure log message generated by a payment behavior, etc. The embodiment of the present application does not limit the specific type of the message to be stored.

Step 302: extract the data to be stored in the message to be stored.

In a possible implementation, the classification server extracts the data to be stored from the message to be stored through a message extraction template, and the data to be stored may include dimension data and indicator data, wherein the dimension data may further include dimension names and dimension values, and the indicator data may further include indicator names and indicator values.

In an illustrative example, the dimension data extracted by the server includes a dimension name of channel number and a dimension value of 001, and the indicator data includes an indicator name of payment success and an indicator value of 1 (indicating payment success).

Optionally, after extracting the data to be stored, the classification server further obtains the target query rule and detects whether the data to be stored conforms to the target query rule. If so, step 303 is executed; if not, step 304 is executed.

Step 303: If the data to be stored meets the target query rules, the data to be stored is stored in a non-relational database, and the data to be stored is stored in a time series database, wherein the concurrent query requirements and query delay requirements of the data that meet the target query rules are higher than those of the data that do not meet the target query rules.

Optionally, the target query rule is manually set by a user, or the target query rule is automatically learned and generated by a classification server.

In one possible implementation, the target query rule is used to indicate the dimensional characteristics of data with high concurrency and low latency query requirements, that is, data with specific dimensional characteristics need to achieve high concurrency and low latency data query. Accordingly, when the dimensional data in the data to be stored meets the dimensional characteristics indicated by the target query rule, the classification server determines that the data to be stored is data with high concurrency and low latency query requirements, and thus stores it in a non-relational database.

Optionally, in order to implement multi-dimensional combined query in a non-relational database, before storing the data to be stored in the non-relational database, the classification server also needs to combine and split the dimensional data in the data to be stored according to predefined dimensional combination rules, so as to store the data generated after the combination and splitting of the data to be stored in the non-relational database.

Since the data stored in the non-relational database is processed data to be stored rather than the original data to be stored, in order to make historical data traceable, the classification server needs to store the data to be stored in the time series database while storing the data in the non-relational database.

In a possible implementation, when storing the data to be stored in the time series database, the classification server inputs the data to be stored into the time series database storage program, and the storage program aggregates and stores the data to be stored according to a predefined data format. The embodiment of the present application does not limit the storage process of data in the time series database.

Step 304: If the data to be stored does not meet the target query rule, the data to be stored is stored in the time series database.

In order to implement multi-dimensional combined query in non-relational databases, the data stored in non-relational databases needs to be processed by combination and splitting, which will cause data expansion, resulting in the actual amount of stored data being much larger than the amount of original data. If all the data to be stored is stored in a non-relational database, a large amount of storage resources will be consumed, and the combination and splitting process will take a long time, resulting in low storage efficiency.

Although the time series database needs to go through the exchange of disk files and memory, as well as the memory decompression step of index files when performing data query, and the data query speed is slower than that of non-relational databases, data expansion does not occur when it stores data. Therefore, in the embodiment of the present application, when it is determined that the data to be stored does not meet the target query rules, that is, the data to be stored is not data with high concurrency and low latency query requirements, the classification server only stores the data to be stored in the time series database, and does not store it in the non-relational database.

In one possible implementation, the time series database is Druid (a real-time big data processing platform), which implements a storage model based on a log schema and provides a query function for any combination of log dimensions without combining or splitting the log dimensions (that is, the stored data will not experience data expansion).

To summarize, in the data storage method provided in the embodiment of the present application, after obtaining the message to be stored, the data to be stored in the message to be stored is extracted, and it is detected whether the data to be stored meets the target query rules. When the data to be stored meets the target query rules, the data to be stored is stored in a non-relational database and a time series database; when the data to be stored does not meet the target query rules, the data to be stored is stored in a time series database; since non-relational databases can provide high concurrency and low-latency data query services, after storing data with high concurrency query requirements and query delay requirements in the non-relational database, the speed of subsequent data queries can be improved; at the same time, after storing data with low concurrency query requirements and query delay requirements in the time series database, it is convenient to perform multi-dimensional combined queries in subsequent data query processes.

In a possible implementation, the non-relational database in the embodiment of the present application uses KV format for data storage, and accordingly, when storing the data to be stored in the non-relational database, the data to be stored needs to be converted into KV format data before being stored.

Please refer to Fig. 4, which shows a flow chart of a data storage method provided by another exemplary embodiment of the present application. This embodiment takes the method used in the classification server 11 shown in Fig. 1 as an example for explanation, and the method includes the following steps.

Step 401, obtaining the message to be stored.

Step 402: extract the data to be stored in the message to be stored, where the data to be stored includes dimension data and indicator data.

The implementation process of the above steps 401 to 402 may refer to steps 301 to 302, and this embodiment will not be repeated here.

Step 403: If the data to be stored includes preset filtering dimension data, format the data in the data to be stored that matches the preset filtering dimension data.

In a possible implementation, a filtering program is provided in the classification server, and the filtering program is used to format abnormal dimensional data in the data to be stored.

Optionally, the filter program is configured with dimension configuration information that needs to be formatted, and the dimension configuration information includes the dimension name and dimension value corresponding to the abnormal dimension data. Accordingly, the filter program detects whether the extracted dimension data contains abnormal dimension data. If so, the abnormal dimension data is formatted 4; if not, it is determined that the data to be stored does not contain abnormal dimension data, and there is no need to format the data to be stored.

Optionally, when formatting the abnormal dimension data, the filter program replaces the latitude value corresponding to the abnormal dimension data with a preset value, for example, the preset value is error.

After completing the formatting of the data to be stored, the classification server further detects whether the data to be stored meets the target query rules. Since dimensions are usually used for data query, in a possible implementation, the target query rules include a dimension set, which is used to indicate the dimensional features (including dimension names) of the data that meets the target query rules (high concurrency and low latency query requirements), wherein the dimension set is manually set or automatically learned and generated by the classification server.

Schematically, as shown in FIG5 , a dimension configuration interface 51 displays several dimension sets that have been set. When a trigger operation for configuring a corresponding modification control for a dimension set is received, a modification interface 52 is displayed. Through the modification interface, the user can modify the interface, dimension, and time granularity.

Optionally, the classification server detects whether the dimensional data in the data to be stored belongs to the dimension set. If so, it determines that the data to be stored meets the target query rules and executes the following steps 404 to 407; if not, it determines that the data to be stored does not meet the target query rules and executes the following steps 408 to 409.

Step 404: If the dimension data belongs to the dimension set, it is determined that the data to be stored meets the target query rule.

In an illustrative example, the dimension set is {channel number + payment method, payment method + platform, payment method + source, payment method + error code}. Since the dimension data extracted from the message to be stored includes the dimension name "channel number + payment method", the classification server determines that the data to be stored meets the target query rules.

Step 405, generating a target key value according to the dimension data, and generating a target value according to the indicator data.

Since the non-relational database uses the KV format to store data, before storing the data to be stored in the non-relational database, the classification server also needs to generate a target key value based on the dimension data and generate a target value based on the indicator data.

In a possible implementation, in order to enable a non-relational database to support multi-dimensional combined data queries, the classification server combines the extracted dimensional data, thereby generating multiple target key values based on the combined dimensional data, and then stores the multiple target key values and their corresponding target value values in the non-relational database.

In an illustrative example, when five dimensional data A, B, C, D, and E are extracted from the message to be stored, the classification server combines the dimensional data to obtain AB, AC, AD, AE, BC, BD, BE, CD, CE, DE, ABC, ABD, ABE, BCD, BCE, BDE, CDE, ABCD, ABCE, BCDE, and ABCDE.

However, when the dimensional data is directly combined in the above manner, it will cause serious data expansion, thereby occupying a large amount of storage space in the non-relational database.

In order to avoid serious data expansion, in a possible implementation, the classification server generates a target key value only based on the dimension data belonging to the dimension set, and generates a corresponding target value based on the indicator data, so as to reduce the degree of data expansion. This step may include the following steps.

1. Determine the dimension data belonging to the dimension set as the target dimension data, and generate the target key value according to the target dimension data.

Since only the dimension data belonging to the dimension set needs to support high-concurrency, low-latency queries, the classification server determines the dimension data belonging to the dimension set in the data to be stored as the target dimension data, and then combines the target dimension data to generate the target key value.

In an illustrative example, when five dimensional data, A, B, C, D, and E, are extracted from the message to be stored, and the dimensional set includes dimensions A and B, the classification server determines dimension data A and dimension data B as target dimensional data, and combines dimension data A and B to generate a target key value.

Compared with directly combining dimensional data, the target dimensional data is first determined based on dimensional combination, and then the target dimensional data is combined to generate the target key value. Under the premise of realizing multi-dimensional combined query function, it can reduce the expansion of stored data, save storage space of non-relational servers, and help improve the efficiency of data entry.

2. Aggregate the indicator data according to the target granularity to generate the target value.

In a possible implementation, when the granularity of the extracted indicator data is inconsistent with the target granularity, the classification server also needs to aggregate the indicator data according to the target granularity to generate a target value that meets the target granularity. The target granularity may be a time granularity, such as minutes, hours, days, etc., and the target granularity may be manually set.

For example, when the time granularity of the extracted indicator data is seconds and the target granularity is minutes, the classification server aggregates the indicator data for 60 consecutive seconds to generate a target value with a time granularity of minutes.

Of course, when the granularity of the indicator data is consistent with the target granularity, the classification server may also directly generate a value according to the indicator data, which is not limited in this embodiment.

Step 406, storing the target key value and the target value in a non-relational database.

In a possible implementation, the above steps 405 and 406 may be executed by a Flink real-time processing system in the classification server, and the Flink real-time processing system stores the generated target key value and target value in a non-relational database.

Optionally, the target key value may also be hashed to obtain a corresponding hash value, thereby storing the hash value domain target value in a non-relational database to improve the efficiency of subsequent queries.

Step 407: store the data to be stored in the time series database.

Since the target key value may be generated based on only some dimensional data in the data to be stored (that is, some dimensions may not be stored in the non-relational database), in order to ensure the traceability of historical data, the classification server also needs to store the data to be stored completely in the time series database, and the time series database can realize data query of any dimensional combination.

Step 408: If the dimension data does not belong to the dimension set, it is determined that the data to be stored does not meet the target query rule.

In an illustrative example, the dimension set is {channel number + payment method, payment method + platform, payment method + source, payment method + error code}. Since the dimension data extracted from the message to be stored includes the dimension name "business + id", the classification server determines that the data to be stored does not meet the target query rules.

Step 409: store the data to be stored in the time series database.

When the data to be stored does not meet the target query rules, it indicates that the data to be stored does not need to support high-concurrency, low-latency data queries. Therefore, the classification server can only store the data to be stored in the time series database instead of the non-relational database, thereby avoiding data expansion caused by storing the data in the non-relational database.

Through the above steps, the classification server completes the classification storage of data. When receiving a query request, the classification server further sends the query request to the corresponding database according to the concurrency and delay requirements of the query request.

Step 410: receiving a query request, wherein the query request includes query dimension data.

In a possible implementation, the classification server provides a unified query entry through which users can initiate two types of query requests: KV and multi-dimensional combination. Accordingly, after receiving the query request, the classification server parses the query dimension data contained in the query request, thereby determining the type of the query request based on the query dimension data.

Optionally, the classification server detects whether the query dimension data conforms to the target query rule. If so, it determines that the query request is a KV type query request and executes step 411; if not, it determines that the query request is a multi-dimensional combination type query request and executes step 412. The process of detecting whether the query dimension data conforms to the target query rule can refer to the process of detecting whether the extracted dimension data conforms to the target query rule in the above steps, and this embodiment will not be repeated here.

Step 411: If the query dimension data meets the target query rule, a first query request is sent to the non-relational database according to the query dimension data.

When the query dimension data meets the target query rule, it indicates that the query request is a query request with high concurrency and low latency requirements, and the classification server sends a first query request to the non-relational database according to the query dimension data.

In a possible implementation, the classification server generates a query key value based on the query dimension data, thereby sending a first query request containing the query key value to the non-relational database, and the non-relational database obtains the corresponding value based on the query key value and feeds the value back to the classification server.

Since non-relational databases store data in KV format, they can achieve high-concurrency, low-latency data queries, thereby shortening data query time and improving the feedback speed of query results.

Step 412: If the query dimension data does not meet the target query rule, a second query request is sent to the time series database according to the query dimension data.

When the query dimension data meets the target query rule, it indicates that the query request has low requirements for concurrency and query latency, and the classification server sends a second query request to the time series database according to the query dimension data.

In a possible implementation, the classification server constructs a query parameter for querying the time series database based on the query dimension data, thereby sending a second query request containing the query parameter to the time series database, and the time series database performs a multi-dimensional combined query based on the query parameter and feeds back the query result to the classification server.

In this embodiment, the classification server filters the extracted data to be stored based on preset filtering dimension data, and formats abnormal dimensional data in the data to be stored to avoid the abnormal dimensional data from affecting subsequent data storage.

In addition, in this embodiment, the classification server detects whether the data to be stored meets the target query rule based on the preset dimension set, thereby storing the data to be stored containing the specified dimension in the non-relational database, which helps to improve the efficiency of subsequent queries on the specified dimension data; at the same time, when storing the data to be stored that meets the target query rule in the non-relational database, the classification server determines the dimensional data belonging to the dimension set as the target dimensional data, and generates a target key value based on the target dimensional data. On the premise of realizing multi-dimensional combined query, the expansion rate of data storage is reduced, saving storage space of the non-relational database.

In addition, in this embodiment, the classification server routes the query request to the corresponding database based on the target query rules and the query dimension parameters in the query request, which can meet both high concurrency and low latency data query requirements and multi-dimensional combination data query requirements.

In combination with the method provided in the above embodiment, in an illustrative example, when applied to the scenario of storing log messages, the process of the classification server storing log messages in the database is shown in FIG6 .

1. The classification server obtains the real-time written log messages from the message queue;

2. The classification server parses the log messages and formats and filters the abnormal dimensions;

3. The classification server detects whether the log message contains dimension combination statistics items with high concurrent queries. If so, execute 4 and 5. If not, execute 6.

4. If it exists, the classification server calculates the key value and indicator value of the parsed log message according to the dimension combination rules and writes them into the NoSQL database;

5. If it exists, the classification server writes the parsed log message into the Druid database according to the predefined dimension format;

6. If it does not exist, the classification server writes the parsed log message to the Druid database according to the predefined dimension format.

Correspondingly, the process of the classification server querying the log message from the database is shown in FIG7 .

1. The classification server receives the client's query request through the unified query portal;

2. The classification server parses the query request in JSON format;

3. The classification server detects whether it needs to query the NoSQL database according to the data query rules. If so, execute 4 and 5. If not, execute 6 and 7.

4. If necessary, the classification server converts the query request into the key value to be queried and queries the NoSQL database;

5. The classification server returns the query results returned by the NoSQL database to the client through a unified query portal;

6. If not required, the classification server converts the query request into the request format required by Druid multi-dimensional combination query and queries the Druid database;

7. The classification server returns the query results returned by the Druid database to the client through a unified query entry.

In the above embodiment, the dimension set is manually configured as an example. However, with the increasing number of dimensions, manual configuration is more difficult and less efficient. In order to improve the efficiency of dimension set configuration, the classification server can automatically generate dimension sets through a self-learning mechanism.

In a possible implementation, as shown in FIG8 , the classification server may generate a dimension set by including the following steps.

Step 801: for each preset time period, count the occurrence frequency of each dimension data in the message to be stored within the preset time period.

Since dimensions that appear frequently in messages are usually important dimensions, and the probability of subsequent high-concurrency, low-latency queries on important dimensions is high, in a possible implementation, each time the classification server obtains a message to be stored, it updates the frequency of occurrence of each dimension data according to the dimension data (single dimension data or dimension data combination) contained in the message to be stored. Of course, in order to reduce the processing pressure of the classification processor, the frequency of occurrence can also be updated based on sampling of the messages to be stored, and the dimension data contained in the sampled messages to be stored is updated, which is not limited in this embodiment.

Optionally, the classification server counts the frequency of occurrence of each dimension data in each preset time period according to the message time of each message to be stored. The preset time period may be a time bucket divided into a predetermined time length, each corresponding to a respective bucket ID, for example, the preset time period is a 2-hour time bucket.

Step 802: determine first candidate dimension data according to the occurrence frequency, where the occurrence frequency of the first candidate dimension data is higher than the occurrence frequency of other dimension data.

Furthermore, the classification server determines, based on the occurrence frequencies corresponding to the respective dimensional data, first candidate dimensional data having a higher occurrence frequency than other dimensional data.

In a possible implementation, the classification server determines the dimension data with the top N% occurrence frequency as the first candidate dimension data. For example, the classification server determines the dimension data with the top 5% occurrence frequency as the first candidate dimension data.

Step 803: Generate a dimension set based on the first candidate dimension data.

Further, the classification server generates a dimension set based on the first candidate dimension data.

In a possible implementation, for the determined first candidate dimension data, the classification server further determines whether the indicator data corresponding to the first candidate dimension data meets the indicator condition, and generates a dimension set according to the first candidate dimension condition if it meets the indicator condition. The indicator condition may be a normal distribution condition.

In a possible implementation, the classification server adds a tuple corresponding to the first candidate dimension data to the dimension set, and the tuple may include a dimension name, a dimension value, an indicator name, and a bucket ID.

Step 804: for each preset time period, when a query request is received within the preset time period, the query dimension data included in the query request is obtained.

In addition to determining dimension data with a high frequency of occurrence as candidate dimension data, the classification server can also determine candidate dimension data based on query dimension data in the query request, that is, determine candidate dimension data based on historical data query conditions.

In a possible implementation, when the classification server receives a query request, it obtains the query dimension data contained in the query request, and determines the preset time period to which the query request belongs according to the query time corresponding to the query request. The preset time period may be a time bucket divided into predetermined time lengths, each corresponding to a respective bucket ID, for example, the preset time period is a 2-hour time bucket.

Step 805: determine the query dimension data as second candidate dimension data, and generate a dimension set based on the second candidate dimension data.

Further, the classification server determines the query dimension data as second candidate dimension data, thereby generating a dimension set according to the second dimension data.

In a possible implementation, the classification server adds a tuple corresponding to the second candidate dimension data to the dimension set, where the tuple may include a (query) dimension name, a (query) dimension value, an indicator name, and a bucket ID.

It should be noted that the above steps 801 to 803 and steps 804 to 805 can be performed one by one or simultaneously, that is, the classification server can generate a dimension set based on the intersection of the first candidate dimension data and the second candidate dimension data, and this embodiment does not limit this.

Step 806: When a query request is received, and the query dimension data contained in the query request belongs to a dimension set, the query timer corresponding to the query dimension data in the dimension set is incremented by one.

In order to count the number of times each candidate dimension data in the dimension set is queried, in a possible implementation, each candidate dimension data in the dimension set also corresponds to its own query counter (which can be set in the tuple corresponding to the candidate dimension data), wherein the initial value of the query counter of the first candidate dimension data in the dimension set is 0, and the initial value of the query counter of the second candidate dimension data is 1.

Furthermore, when receiving a query request, the classification server detects whether the query dimension data contained in the query request belongs to the dimension set, and when the query dimension data belongs to the dimension set, adds one to the query timer corresponding to the query dimension data in the dimension combination.

Step 807, obtaining the count value of the query counter corresponding to each candidate dimension data in the dimension set within a predetermined time period before the current moment.

For candidate dimension data in a dimension set, if no query request for the candidate dimension data is received for a long time, it indicates that the importance of the query candidate dimension data is relatively low. Therefore, in order to avoid the dimension set being too large, the classification server can obtain the count value of the query counter corresponding to each candidate dimension data in the dimension set, so as to determine whether the candidate dimension data needs to be deleted based on the count value.

In a possible implementation, in order to ensure timeliness, the classification server obtains the count value of the query counter corresponding to the candidate dimension data within a predetermined time period before the current moment. For example, the classification server can determine whether it is within a predetermined time period before the current moment based on the bucket ID corresponding to the candidate dimension data, for example, the predetermined time period can be 1 day or 1 week.

Step 808: If the count value of the query counter corresponding to the candidate dimension data is less than the count threshold, the candidate dimension data is deleted from the dimension set.

Furthermore, the classification server detects whether the obtained count value is less than the count threshold, and if so, deletes the corresponding candidate dimension data, and if so, retains the corresponding candidate dimension data. For example, the count threshold may be 1, 2, 5, and so on.

In one possible implementation, if the count value of the query counter corresponding to the candidate dimension data is less than the count threshold, the classification server may mark the candidate dimension data as to be deleted and set a deletion delay period (for example, 1 hour); if the count value of the query counter is still less than the count threshold when the deletion delay period is reached, the candidate dimension data is deleted; if the count value of the query counter is greater than the count threshold when the deletion delay period is reached, the to-be-deleted mark of the candidate dimension data is cleared.

In this embodiment, the classification server determines candidate dimension data based on the frequency of occurrence of each dimension data and/or based on the query dimension data in the query request, and then generates a dimension set based on the candidate dimension data, thereby eliminating the process of users manually configuring the dimension set and simplifying the configuration efficiency and speed of the dimension set.

In addition, the classification server updates the query times of the candidate dimension data according to the received query requests, and periodically clears the candidate dimension data with fewer query times, thereby avoiding the dimension set from being too large.

After the data to be stored is stored in the non-relational database, in order to further compress the storage space of the data, the data stored in the non-relational database can be compressed by a data compression mechanism. In a possible implementation, the data compression process can be performed by the non-relational database or by the classification server. The following takes the data compression process performed by the classification server as an example for explanation.

In a possible implementation, the classification server obtains data with the same key value in the non-relational database and detects whether the time series corresponding to each key value is continuous. If the time series of the key value is not continuous, the classification server uses the format shown in FIG9 to perform data compression.

In the format shown in Figure 9, the value portion corresponding to the key value includes the FLG field, LEN field, INDEX field and datapoint field. Among them, the FLG field is 1 bit, used to indicate whether the time series is continuous (0 means discontinuous, 1 means continuous); the LEN field is 3 bits, used to indicate the data storage byte length of each time series corresponding to the datapoint; the INDEX field is 1 byte, used to indicate the index corresponding to the time series; the datapoint field is used to store the data of the datapoint, and the INDEX field appears in pairs with the datapoint field.

By using the above format for data compression, the value values corresponding to the same key value in the same time bucket can be compressed into one value, thereby saving storage space.

If the time series of the key value is continuous, the classification server uses the format shown in Figure 10 for data compression. Compared with the format shown in Figure 9, since the key value is continuous, the value value can no longer store the INDEX field, but directly store the datapoint data and set the value of the FLG field to 1.

Optionally, if the time series of the key value is continuous and the data point data storage byte length is greater than the preset byte length, the classification server can further perform an exclusive OR operation (exclusive OR, XOR) on the data point data corresponding to the adjacent time series, thereby compressing the data point data according to the exclusive OR operation result.

In one possible implementation, the datapoint data corresponding to the first time series is directly stored without being compressed, and for the datapoint data corresponding to the nth time series (n is greater than or equal to 2), the classification server performs an XOR operation on the datapoint data corresponding to the nth time series and the n-1th time series to obtain an XOR operation result, and generates a compressed value of the datapoint data corresponding to the nth time series based on the XOR operation result.

In an illustrative example, the classification server obtains the original datapoints (previous value and current value) corresponding to 13:00:00 and 13:00:01 as shown in Table 1.

Table I

13:00:00 previous value 00000000 00111100 13:00:01 current value 00000000 00111101 current xor 00000000 00000001

By performing an XOR operation on the previous value and the current value, the XOR operation result (current XOR) is 0000000000000001. Further, according to the XOR operation result 0000000000000001, the classification server generates a compression value corresponding to the current value as shown in Table 2.

Table II

Among them, the first bit of the compressed value is used to indicate whether the XOR operation result is 0 (if not 0, it is set to 1, and if it is 0, it is set to 0), the second bit is used to indicate whether the number of leading 0 bits in the XOR operation result is the same as the number of leading 0 bits in the previous XOR operation result (if they are the same, they are set to 1, and if they are different, they are set to 0), the third to fourth bits are used to indicate the number of leading 0 bits in the XOR operation result (in units of 4 bits, 11 means that the number of leading 0 bits is 3×4 bits), and the fifth to eighth bits are used to store the part of the XOR operation result other than the leading 0 (i.e., 0001). Schematically, the compressed data obtained by the above method is shown in Figure 11.

FIG. 12 is a structural block diagram of a data storage device provided by an exemplary embodiment of the present application. The device may be provided in the classification server described in the above embodiment. As shown in FIG. 12 , the device includes:

The first acquisition module 1201 is used to acquire the message to be stored;

An extraction module 1202 is used to extract the data to be stored in the message to be stored;

The first storage module 1203 is used to store the data to be stored in a non-relational database and store the data to be stored in a time series database when the data to be stored meets the target query rule, wherein the concurrent query requirement and query delay requirement of the data meeting the target query rule are higher than those of the data not meeting the target query rule;

The second storage module 1204 is configured to store the data to be stored in the time series database when the data to be stored does not meet the target query rule.

Optionally, the data to be stored includes dimension data and indicator data, and the non-relational database uses a key-value structure to store data;

The first storage module 1203 includes:

A generating unit, configured to generate a target key value according to the dimension data and a target value according to the indicator data when the data to be stored meets the target query rule;

A storage unit is used to store the target key value and the target value in the non-relational database.

Optionally, the target query rule includes a dimension set, and the dimension set is used to indicate dimensional features of data that meets the target query rule;

The generating unit is used for:

If the dimension data belongs to the dimension set, determining that the data to be stored meets the target query rule;

Determine the dimension data belonging to the dimension set as target dimension data, and generate the target key value according to the target dimension data;

The indicator data is aggregated according to the target granularity to generate the target value.

Optionally, the device further includes a first generating module, wherein the first generating module is configured to:

For each preset time period, counting the occurrence frequency of each dimension data in the message to be stored within the preset time period;

Determine first candidate dimension data according to the occurrence frequency, where the occurrence frequency of the first candidate dimension data is higher than the occurrence frequency of other dimension data;

The dimension set is generated according to the first candidate dimension data.

Optionally, the device further includes a second generating module, wherein the second generating module is configured to:

For each preset time period, when a query request is received within the preset time period, the query dimension data included in the query request is obtained;

The second generating module is used to determine the query dimension data as second candidate dimension data, and generate the dimension set according to the second candidate dimension data.

Optionally, the device further comprises:

A first receiving module, configured to, when receiving a query request, and the query dimension data contained in the query request belongs to the dimension set, add one to a query timer corresponding to the query dimension data in the dimension set;

A second acquisition module is used to obtain the count value of the query counter corresponding to each candidate dimension data in the dimension set within a predetermined time period before the current moment;

The deletion module is used to delete the candidate dimension data from the dimension set when the count value of the query counter corresponding to the candidate dimension data is less than the count threshold.

Optionally, the device further comprises:

A second receiving module, configured to receive a query request, wherein the query request includes query dimension data;

A first sending module, configured to send a first query request to the non-relational database according to the query dimension data if the query dimension data meets the target query rule;

The second sending module is used to send a second query request to the time series database according to the query dimension data if the query dimension data does not meet the target query rule.

Optionally, the device further comprises:

The formatting module is used for formatting the data in the data to be stored that matches the preset filtering dimension data if the data to be stored includes the preset filtering dimension data.

To summarize, in an embodiment of the present application, after obtaining the message to be stored, the data to be stored in the message to be stored is extracted, and it is detected whether the data to be stored meets the target query rules. When the data to be stored meets the target query rules, the data to be stored is stored in a non-relational database and a time series database; when the data to be stored does not meet the target query rules, the data to be stored is stored in a time series database; since non-relational databases can provide high concurrency and low-latency data query services, storing data with higher concurrent query requirements and query delay requirements in the non-relational database can increase the speed of subsequent data queries; at the same time, storing data with lower concurrent query requirements and query delay requirements in the time series database facilitates multi-dimensional combined queries in subsequent data query processes.

It should be noted that the data storage device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the server is divided into different functional modules to complete all or part of the functions described above. In addition, the data storage device provided in the above embodiment belongs to the same concept as the data storage method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

Please refer to Figure 13, which shows a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. Specifically, the server 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The server 1300 also includes a basic input/output system (I/O system) 1306 that helps transmit information between various devices in the computer, and a large-capacity storage device 1307 for storing an operating system 1313, an application program 1314, and other program modules 1315.

The basic input/output system 1306 includes a display 1308 for displaying information and an input device 1309 such as a mouse and a keyboard for user inputting information. The display 1308 and the input device 1309 are connected to the central processing unit 1301 through an input/output controller 1310 connected to the system bus 1305. The basic input/output system 1306 may also include an input/output controller 1310 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input/output controller 1310 also provides output to a display screen, a printer, or other types of output devices.

The mass storage device 1307 is connected to the central processing unit 1301 through a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the server 1300. That is, the mass storage device 1307 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.

Without loss of generality, the computer readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above. The above-mentioned system memory 1304 and mass storage device 1307 can be collectively referred to as memory.

The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1301 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

According to various embodiments of the present application, the server 1300 may also be connected to a remote computer on a network through a network such as the Internet. That is, the server 1300 may be connected to a network 1312 through a network interface unit 1311 connected to the system bus 1305, or the network interface unit 1311 may be used to connect to other types of networks or remote computer systems (not shown).

The memory also includes one or more programs, which are stored in the memory and include steps executed by the classification server in the method provided in the embodiment of the present application.

An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the data storage method described in any of the above embodiments.

The present application also provides a computer program product. When the computer program product runs on a server, the computer executes the data storage methods provided by the above-mentioned various method embodiments.

A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can be a computer-readable storage medium contained in the memory in the above embodiments; or a computer-readable storage medium that exists independently and is not assembled in the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the data storage method described in any of the above method embodiments.

Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims (10)

1. A method of data storage, the method comprising:

Acquiring a message to be stored;

Extracting data to be stored in the message to be stored, wherein the data to be stored comprises dimension data and index data;

If the dimension data belong to a dimension set, determining that the data to be stored accord with a target query rule; determining the dimension data belonging to the dimension set as target dimension data, and combining the target dimension data to generate a target key value; performing aggregation treatment on the index data according to the target granularity to generate a target value; storing the target key value and the target value into a non-relational database, wherein the non-relational database stores data by adopting a key-value structure; storing the data to be stored into a time sequence database, wherein the target query rule is used for indicating the dimension characteristics of data with high concurrency and low delay query requirements, the concurrency query requirements of the data conforming to the target query rule and the query delay requirements are higher than those of the data not conforming to the target query rule, the dimension set is contained in the target query rule, and the dimension set is used for indicating the dimension characteristics of the data conforming to the target query rule;

if the dimension data does not belong to the dimension set, determining that the data to be stored does not accord with the target query rule; storing the data to be stored into the time sequence database;

Receiving a query request, wherein the query request comprises query dimension data;

If the query dimension data accords with the target query rule, a first query request is sent to the non-relational database according to the query dimension data;

And if the query dimension data does not accord with the target query rule, sending a second query request to the time sequence database according to the query dimension data.

2. The method of claim 1, wherein after the retrieving the message to be stored, the method further comprises:

counting the occurrence frequency of each dimension data in the message to be stored in the preset time period for each preset time period;

determining first candidate dimension data according to the occurrence frequency, wherein the occurrence frequency of the first candidate dimension data is higher than that of other dimension data;

And generating the dimension set according to the first candidate dimension data.

3. The method of claim 1, wherein after the retrieving the message to be stored, the method further comprises:

for each preset time period, when a query request is received within the preset time period, acquiring query dimension data contained in the query request;

And determining the query dimension data as second candidate dimension data, and generating the dimension set according to the second candidate dimension data.

4. A method according to claim 2 or 3, characterized in that the method further comprises:

when a query request is received and query dimension data contained in the query request belongs to the dimension set, adding one operation to a query counter corresponding to the query dimension data in the dimension set;

Acquiring count values of the query counters corresponding to each candidate dimension data in the dimension set within a preset time before the current moment;

And if the count value of the query counter corresponding to the candidate dimension data is smaller than the count threshold value, deleting the candidate dimension data from the dimension set.

5. A method according to any one of claims 1 to 3, wherein after said extracting the data to be stored in the message to be stored, the method further comprises:

If the data to be stored contains preset filtering dimension data, formatting the data matched with the preset filtering dimension data in the data to be stored.

6. A data storage device, the device comprising:

The first acquisition module is used for acquiring the message to be stored;

The extraction module is used for extracting data to be stored in the message to be stored, wherein the data to be stored comprises dimension data and index data;

The first storage module is used for determining that the data to be stored accords with a target query rule when the dimension data belongs to a dimension set; determining the dimension data belonging to the dimension set as target dimension data, and combining the target dimension data to generate a target key value; performing aggregation treatment on the index data according to the target granularity to generate a target value; storing the target key value and the target value into a non-relational database, wherein the non-relational database stores data by adopting a key-value structure; storing the data to be stored into a time sequence database, wherein the target query rule is used for indicating the dimension characteristics of data with high concurrency and low delay query requirements, the concurrency query requirements of the data conforming to the target query rule and the query delay requirements are higher than those of the data not conforming to the target query rule, the dimension set is contained in the target query rule, and the dimension set is used for indicating the dimension characteristics of the data conforming to the target query rule;

the second storage module is used for determining that the data to be stored does not accord with the target query rule when the dimension data does not belong to the dimension set; storing the data to be stored into the time sequence database;

The second receiving module is used for receiving a query request, wherein the query request comprises query dimension data;

The first sending module is used for sending a first query request to the non-relational database according to the query dimension data if the query dimension data accords with the target query rule;

And the second sending module is used for sending a second query request to the time sequence database according to the query dimension data if the query dimension data does not accord with the target query rule.

7. The apparatus of claim 6, further comprising a first generation module to:

counting the occurrence frequency of each dimension data in the message to be stored in the preset time period for each preset time period;

determining first candidate dimension data according to the occurrence frequency, wherein the occurrence frequency of the first candidate dimension data is higher than that of other dimension data;

And generating the dimension set according to the first candidate dimension data.

8. The apparatus of claim 6, further comprising a second generation module configured to:

for each preset time period, when a query request is received within the preset time period, acquiring query dimension data contained in the query request;

And the second generation module is used for determining the query dimension data as second candidate dimension data and generating the dimension set according to the second candidate dimension data.

9. A server comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by the processor to implement the data storage method of any one of claims 1 to 5.

10. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, the at least one instruction, the at least one program, the set of codes, or the set of instructions being loaded and executed by a processor to implement the data storage method of any one of claims 1 to 5.