CN106919617B - Compression storage method and device - Google Patents

Abstract

Embodiments of the present invention provide a compression storage method and device, wherein the method specifically includes: acquiring compression parameters of data to be stored, and/or acquiring device parameters of a storage device corresponding to the data to be stored; according to the compression parameters and/or device parameters, select a target compression scheme corresponding to the data to be stored; use the target compression scheme to compress and store the data to be stored. The embodiments of the present invention can maximize the advantages of the target compression scheme, thereby greatly improving the throughput and other performance of the database. Moreover, the embodiment of the present invention can select a target compression algorithm that best fits the current number of CPU cores, so that while ensuring database performance, the stability of compressed storage can be increased, and resources of storage devices in the database cluster can be saved as much as possible.

Description

A compressed storage method and device

technical field

The present invention relates to the field of computer technology, and in particular, to a compression storage method and device.

Background technique

With the development of information technology, especially the development of Internet technology, the amount of information in various fields has shown an explosive growth trend, and massive data larger than 10 ¹² bytes emerge in an endless stream. In order to effectively manage massive data, compressed database technology has been proposed at present. Compressed database technology can improve the storage efficiency of massive data, and can improve the performance of the database such as throughput.

Existing solutions usually use the zlib compression algorithm provided by InnoDB to compress and store massive data; among them, InnoDB is the most widely used storage engine in MySQL databases, and zlib can provide a function library for data compression, which can save about 25% -50% storage space, and can reduce IO (input and output, Input Output) consumption, and improve the throughput of the database.

However, in the process of using the existing solution, we sometimes encounter such a problem: in the case of a large data base, when frequent insert (insert) or update (update) operations are continued, the B-tree used by InnoDB The nature of the (B-tree) index itself tends to cause a significant drop in database performance, which may, for example, result in a drop in database throughput.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention is proposed to provide a compressed storage method and apparatus that overcomes the above problems or at least partially solves the above problems.

According to one aspect of the present invention, a compression storage method is provided, comprising:

Obtain the compression parameters of the data to be stored, and/or obtain the device parameters of the storage device corresponding to the data to be stored;

Selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters;

The data to be stored is compressed and stored by using the target compression scheme.

Optionally, the compression parameters include application scenario parameters and/or database architecture parameters, and the step of selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters further includes: :

Select a target storage engine corresponding to the application scenario parameters and/or database schema parameters from at least one storage engine in the database;

A corresponding target compression algorithm is selected from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

Optionally, the step of selecting a target storage engine corresponding to the application scenario parameter from at least one storage engine in the database further includes:

If the application scenario parameter is the first application scenario parameter, select the corresponding first storage engine as the target storage engine; or,

If the application scenario parameter is the second application scenario parameter, then select the corresponding second storage engine as the target storage engine;

Wherein, the first application scenario parameters include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

Wherein, the second application scenario parameters include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

Optionally, the step of selecting a target storage engine corresponding to the database schema parameter from at least one storage engine in the database further includes:

If the database architecture parameter is the first database architecture parameter, the corresponding third storage engine is selected as the target storage engine; or,

If the database architecture parameter is the second database architecture parameter, the corresponding fourth storage engine is selected as the target storage engine;

Wherein, the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameters include: non-master-slave architecture parameters.

Optionally, the compression parameters include compression index parameters, and the step of selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters further includes:

A target compression algorithm that fits the compression index parameter and/or the device parameter is selected from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

Optionally, the device parameters include at least one of the following parameters: CPU parameters, disk parameters and memory parameters.

Optionally, the compression index parameter includes at least one of the following parameters: a compression time parameter and a compression ratio parameter.

Optionally, the target compression algorithm conforming to the compression index parameter and/or the device parameter includes:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

According to another aspect of the present invention, a compressed storage device is provided, comprising:

an acquisition module, used for acquiring compression parameters of the data to be stored, and/or acquiring device parameters of the storage device corresponding to the data to be stored;

a selection module, configured to select a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters; and

A compression storage module, configured to compress and store the data to be stored by using the target compression scheme.

Optionally, the compression parameters include application scenario parameters and/or database architecture parameters, and the selection module further includes:

a first selection submodule, configured to select a target storage engine corresponding to the application scenario parameters and/or database architecture parameters from at least one storage engine of the database; and

The second selection sub-module is configured to select a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

Optionally, the first selection submodule further includes:

a first selection unit, configured to select the corresponding first storage engine as the target storage engine when the application scenario parameter is the first application scenario parameter; or,

a second selection unit, configured to select the corresponding second storage engine as the target storage engine when the application scenario parameter is the second application scenario parameter;

Wherein, the first application scenario parameters include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

Wherein, the second application scenario parameters include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

Optionally, the first selection submodule further includes:

a third selection unit, configured to select the corresponding third storage engine as the target storage engine when the database schema parameter is the first database schema parameter; or,

a fourth selection unit, configured to select the corresponding fourth storage engine as the target storage engine when the database schema parameter is the second database schema parameter;

Wherein, the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameters include: non-master-slave architecture parameters.

Optionally, the compression parameters include compression index parameters, and the selection module further includes:

A third selection sub-module, configured to select a target compression algorithm that fits the compression index parameter and/or the device parameter from at least one compression algorithm of the target storage engine, the target storage engine and its corresponding target compression algorithm compose the target compression scheme.

Optionally, the device parameters include at least one of the following parameters: CPU parameters, disk parameters and memory parameters.

Optionally, the compression index parameter includes at least one of the following parameters: a compression time parameter and a compression ratio parameter.

Optionally, the target compression algorithm conforming to the compression index parameter and/or the device parameter includes:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

According to a compression storage method and device according to an embodiment of the present invention, a target compression scheme corresponding to the data to be stored can be selected according to the compression parameters and/or device parameters corresponding to the data to be compressed; Parameters related to the characteristics of the data to be compressed, or the above compression parameters can also be used to represent parameters related to the compression requirements of the data to be compressed, or the above compression parameters can also be used to represent parameters related to the storage environment of the data to be compressed, so In the embodiment of the present invention, a target compression scheme that best suits the characteristics, compression requirements, and storage environment of the data to be compressed can be selected according to the compression parameters. Therefore, using the target compression scheme to compress and store the data to be stored can maximize the The advantages of the target compression scheme are brought into play, so that the performance such as the throughput of the database can be greatly improved.

In addition, since the above device parameters can be used to represent the performance of the storage device corresponding to the data to be stored, the embodiment of the present invention can select a target compression algorithm that best fits the current number of CPU cores, so that while ensuring the performance of the database, the compressed storage can be increased. It is stable and can save the resources of storage devices in the database cluster as much as possible.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of alternative embodiments. The drawings are for the purpose of illustrating alternative embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 shows a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention;

FIG. 2 shows a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention;

FIG. 3 shows a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention; and

FIG. 4 shows a schematic structural diagram of a compressed storage device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

Referring to FIG. 1, it shows a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention, which may specifically include the following steps:

Step 101: Obtain compression parameters of the data to be stored, and/or obtain device parameters of a storage device corresponding to the data to be stored;

Step 102, according to the compression parameters and/or device parameters, select a target compression scheme corresponding to the data to be stored;

Step 103: Compress and store the data to be stored by using the target compression scheme.

The embodiment of the present invention is applicable to the compressed storage of massive data in the database, wherein the database may include databases such as ORACLE (Oracle), DB2, MySQL, etc. It can be understood that the embodiment of the present invention does not specify the specific type of the database. limit. For the convenience of description, the embodiments of the present invention are described by taking MySQL as an example, and other types of databases may be referred to each other.

In this embodiment of the present invention, the above-mentioned compression parameters may be used to represent parameters related to the characteristics of the data to be compressed, such as parameters used to measure the data size of the data to be compressed; or, the above-mentioned compression parameters may also be used to represent the parameters related to the data to be compressed. Parameters related to compression requirements, for example, application scenario parameters used to measure the application scenario of the data to be compressed, or compression index parameters used to measure the compression index of the data to be compressed; Parameters related to the storage environment of the data to be compressed, such as database architecture parameters used to measure the database environment of the data to be compressed. Since the embodiment of the present invention can select a target compression scheme that best fits the above compression parameters according to the compression parameters, using the target compression scheme to compress and store the data to be stored can maximize the performance of the target compression scheme. Advantages, which can greatly improve the performance of the database throughput and so on.

In the embodiment of the present invention, the above-mentioned target compression scheme may specifically include: a target storage engine and a target compression algorithm corresponding to the target engine. The above-mentioned target storage engine may be a storage engine designated by a user, or may be a storage engine selected through the embodiment of the present invention. Among them, the above-mentioned storage engine can be used for a functional system that is responsible for a series of transactions such as storage of data in the database, establishment of indexes for stored data, and updating of data such as querying.

In an optional embodiment of the present invention, the storage engine of the MySQL database may specifically include: ISAM, MyISAM, HEAP, InnoDB, TokuDB, etc., then the embodiment of the present invention can select the target that best fits the above compression parameters according to the compression parameters. storage engine.

In an application example 1 of the present invention, compared with the existing solution, in the case of a large data base, frequent insert or update operations continue to cause a significant drop in database performance, the embodiment of the present invention can According to the large data volume base, the corresponding large data volume parameters or online analysis and processing scenario parameters can be obtained, or, according to the large data volume base and the characteristics of continuous frequent insert or update operations, the corresponding batch loading can be obtained. and read-intensive scenario parameters, and select TokuDB that fits the above large data volume parameters or online analytical processing scenario parameters or batch loading and read-intensive scenario parameters as the target storage engine; since the index structure of TokuDB is Fractal Tree, and Fractal Tree is relative to B -tree has the advantages of stronger insert performance and less "fragmentation", so TokuDB has the advantages of higher compression ratio and better batch loading performance. Therefore, the TokuDB selected in the embodiment of the present invention can be used for the above application example 1. The data to be compressed is effectively compressed and stored, so that the performance such as the throughput of the database can be improved as much as possible.

In this embodiment of the present invention, the foregoing device parameters may be used to represent the performance of the storage device corresponding to the data to be stored. In an optional embodiment of the present invention, the device parameters may specifically include at least one of the following parameters: CPU parameters, disk parameters, and memory parameters.

In an optional embodiment of the present invention, the CPU consumption generated by different compression algorithms is different, so the corresponding target CPU parameters can be obtained for each compression algorithm, and according to the current CPU of the storage device corresponding to the data to be stored The parameter selects the corresponding compression algorithm. For example, the current number of CPU cores (cores) of the storage device is 6 cores, and the first target compression algorithm requires more than 8 CPU cores, the second target compression algorithm requires CPU cores between 4 cores and 8 cores, and the third target compression algorithm requires the number of CPU cores to be between 4 and 8. The three-target compression algorithm requires the number of CPU cores to be less than 4 cores. Therefore, you can choose the target compression algorithm that best fits the current number of CPU cores, thereby ensuring database performance and saving storage device resources as much as possible.

It should be noted that, the compressed storage process in the embodiment of the present invention may be executed by a compressed storage device. In an optional embodiment of the present invention, those skilled in the art can determine the compression parameters of the data to be compressed according to the characteristics of the data to be compressed, compression requirements, storage environment and other factors and parameters, and send the data to the above-mentioned data through the first preset interface. The compression storage device inputs the determined compression parameters. Alternatively, those skilled in the art may first determine the storage device corresponding to the data to be stored, then obtain the device parameters of the storage device, and input the obtained device parameters into the above-mentioned compressed storage device through the second preset interface.

In another optional embodiment of the present invention, the above-mentioned compressing storage device may first obtain factors such as characteristics of the data to be compressed, compression requirements, storage environment, etc., and then determine the compression parameters of the data to be compressed according to the above-mentioned factors and parameters, wherein , the above-mentioned factor parameters can be acquired through a third preset interface, or can be obtained by the above-mentioned compressed storage device by accessing a database. The embodiment of the present invention does not limit the specific acquisition method of the above-mentioned factor parameters. Alternatively, the above-mentioned compressed storage apparatus may also obtain the device parameters of the storage device by accessing the storage device corresponding to the data to be stored. It can be understood that the embodiments of the present invention do not limit the specific acquisition process of the above-mentioned compression parameters and the above-mentioned device parameters.

To sum up, the embodiment of the present invention selects a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters corresponding to the data to be compressed; since the above compression parameters can be used to represent parameters related to the characteristics of the data to be compressed , or the above compression parameters can also be used to represent parameters related to the compression requirements of the data to be compressed, or the above compression parameters can also be used to represent parameters related to the storage environment of the data to be compressed, so the embodiment of the present invention can be based on the compression parameters Select the target compression scheme that best fits the characteristics, compression requirements, and storage environment of the data to be compressed. Therefore, using the target compression scheme to compress and store the to-be-stored data can maximize the advantages of the target compression scheme , which can greatly improve the throughput and other performance of the database.

In addition, since the above device parameters can be used to represent the performance of the storage device corresponding to the data to be stored, the embodiment of the present invention can select a target compression algorithm that best fits the current number of CPU cores, so that while ensuring the performance of the database, the compressed storage can be increased. It is stable and can save the resources of storage devices in the database cluster as much as possible.

Referring to FIG. 2, it shows a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention, which may specifically include the following steps:

Step 201, obtaining compression parameters of the data to be stored, and/or obtaining device parameters of a storage device corresponding to the data to be stored;

Step 202, according to the compression parameters and/or device parameters, select a target compression scheme corresponding to the data to be stored;

Step 203, using the target compression scheme to compress and store the data to be stored;

With respect to the embodiment shown in FIG. 1 , the compression parameters in this embodiment may specifically include application scenario parameters and/or database architecture parameters, and the foregoing step 202 may further include:

Step 221: Select a target storage engine corresponding to the application scenario parameters and/or database architecture parameters from at least one storage engine in the database;

Step 222: Select a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm may constitute the target compression scheme.

In this embodiment, a target storage engine that best fits the application scenario parameters and/or database architecture parameters is selected from at least one storage engine in the database, and the above-mentioned target storage engine is used to compress and store. In the case of the advantages of the target storage engine, the data to be compressed is effectively compressed and stored, so that the performance such as the throughput of the database can be improved as much as possible.

In an optional embodiment of the present invention, the resource information of each storage engine of the database can be captured through the Internet, and the above resource information can be analyzed to obtain the advantage keywords and disadvantage keywords corresponding to each storage engine, In this way, in the process of implementing step 221, the application scenario parameters and/or database architecture parameters can be matched with the above-mentioned advantageous keywords or disadvantaged keywords. If the matching with the disadvantaged keywords is successful, the corresponding storage engine can be excluded. , if it is successfully matched with the dominant keyword, the corresponding storage engine can be used as an alternative storage engine. Wherein, when the number of alternative storage engines is greater than 1, one of the alternative storage engines may be selected as the target storage engine according to the number of successfully matched advantageous keywords, etc., or all alternative storage engines may be output to make The user selects a target storage engine from it. When the number of candidate storage engines is equal to 1, the candidate storage engine can be directly used as the target storage engine.

It can be understood that the above process of obtaining the advantageous keywords and disadvantaged keywords is only an optional embodiment. In fact, those skilled in the art can also adopt other selection processes of the target storage engine according to actual application requirements, such as through the fifth preset interface. Receive the advantage keywords and disadvantage keywords input by the user, or, by analyzing the historical usage records of the storage engine or the user's comment data on the storage engine, to obtain the advantage keywords and disadvantage keywords corresponding to each storage engine, etc., The embodiments of the present invention do not limit the acquisition process of the advantageous keywords and disadvantaged keywords corresponding to each storage engine.

Embodiments of the present invention may provide the following selection scheme for selecting a target storage engine corresponding to the application scenario parameters and/or database architecture parameters from at least one storage engine of a database:

Choose option 1

In option 1, the step of selecting a target storage engine corresponding to the application scenario parameter from at least one storage engine in the database may further include:

Step A1, the application scenario parameter is the first application scenario parameter, then select the corresponding first storage engine as the target storage engine; or,

Step A2, the application scenario parameter is the second application scenario parameter, then select the corresponding second storage engine as the target storage engine;

Wherein, the first application scenario parameters may specifically include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

The second application scenario parameters may specifically include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

In the embodiment of the present invention, OLTP (On-Line Transaction Processing) is the main application of a traditional relational database, and is mainly used for basic, daily transaction processing, such as bank transactions. Since OLTP scenarios usually involve operations such as insert, update, delete, and query of a single or few database records, InnoDB can make full use of the advantages of CPU multi-core/high frequency, while reducing IO consumption and improving throughput, which is especially suitable for read-intensive applications , therefore, in an optional embodiment of the present invention, the above-mentioned first storage engine may specifically include: InnoDB. It can be understood that any storage engine that conforms to the online transaction processing scenario parameters and/or the read-intensive scenario parameters can be used as the above-mentioned first storage engine, and the embodiment of the present invention does not limit the specific first storage engine.

OLAP (Online Analytical Processing) is the main application of the data warehouse system, which supports complex analysis operations, focuses on decision support, and provides intuitive and easy-to-understand query results. Since the OLTP scenario usually involves the loading of batch database records, and TokuDB has the advantages of better batch loading performance and better insertion performance, in an optional embodiment of the present invention, the above-mentioned second storage engine may specifically include: TokuDB. It can be understood that any storage engine that conforms to the online analytical processing scenario parameters and/or the batch loading and read-intensive scenario parameters can be used as the second storage engine, and the embodiment of the present invention does not limit the specific second storage engine.

Choose option 2

In option 2, the step of selecting a target storage engine corresponding to the database schema parameter from at least one storage engine in the database may further include:

Step B1, the database architecture parameter is the first database architecture parameter, then select the corresponding third storage engine as the target storage engine; or,

Step B2, the database architecture parameter is the second database architecture parameter, then select the corresponding fourth storage engine as the target storage engine;

Wherein, the first database architecture parameters may specifically include: master-slave architecture parameters; or, the second database architecture parameters may specifically include: non-master-slave architecture parameters.

In the embodiment of the present invention, the above-mentioned master-slave architecture parameters may be used to represent the architecture in which the database architecture supports master-slave replication. Using the master-slave architecture, when the master server fails, the slave server can be used to provide services, so the stability of the database can be provided. Moreover, processing user requests separately on the master and slave servers can improve data processing efficiency. In addition, the data on the master server is copied to the slave server to protect the data from accidental loss.

It is found through research in the embodiment of the present invention that TokuDB currently has defects in the support of the master-slave architecture, that is, the compatibility between TokuDB and the master-slave architecture is problematic, while InnoDB can better support the master-slave architecture. Therefore, in an optional embodiment of the present invention, the first storage engine may specifically include: InnoDB, and the second storage engine may specifically include: TokuDB. It can be understood that any storage engine that conforms to the parameters of the master-slave architecture can be used as the above-mentioned first storage engine, and any storage engine that conforms to the parameters of the non-master-slave architecture can be used as the above-mentioned second storage engine. The engine and the second storage engine are not limited.

Referring to FIG. 3, a schematic flowchart of steps of a compressed storage method according to an embodiment of the present invention is shown, which may specifically include the following steps:

Step 301: Obtain compression parameters of the data to be stored, and/or obtain device parameters of a storage device corresponding to the data to be stored;

Step 302, according to the compression parameters and/or device parameters, select a target compression scheme corresponding to the data to be stored;

Step 303, using the target compression scheme to compress and store the data to be stored;

Compared with the embodiment shown in FIG. 1 , the compression parameters in this embodiment may specifically include compression index parameters, and the foregoing step 302 may further include:

Step 321: Select a target compression algorithm that fits the compression index parameter and/or the device parameter from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm can form the target compression scheme.

The compression index parameter may be used to represent the compression index of the data to be stored. For example, the compression index parameter may specifically include at least one of the following parameters: a compression time parameter and a compression ratio parameter. The above compression time parameter can be used to represent the time spent for compressing storage, and the compression ratio parameter can be used to represent the space ratio of the compressed data relative to the data to be stored. In this embodiment, a target compression algorithm that fits the compression index parameters and/or the device parameters is selected from at least one compression algorithm of the target storage engine, which can ensure the stability of compressed storage and save the time of storage devices in the database cluster. At the same time, the optimal compression index is achieved as much as possible.

In an optional embodiment of the present invention, assuming that the above-mentioned target storage engine has three or more target compression algorithms, the target compression algorithm conforming to the compression index parameter and/or the device parameter is specific Can include:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

In an application example of the present invention, it is assumed that the above-mentioned target storage engine is TokuDB, and TokuDB has the lzma algorithm, the zlib algorithm and the quicklz algorithm, wherein the compression ratio parameters corresponding to the lzma algorithm, the zlib algorithm and the quicklz algorithm are respectively: 2.626857, 2.285948 and 1.700141; then the first-level compression index parameters, the second-level compression index parameters, and the third-level compression index parameters corresponding to the lzma algorithm, the zlib algorithm, and the quicklz algorithm can be set according to the above-mentioned compression ratio parameters, for example, set for the lzma algorithm The first interval of the first-level compression index parameter is [2.3, 2.7], the second interval of the second-level compression index parameter is set to [1.8, 2.29] for the zlib algorithm, and the third-level compression index parameter is set for the quicklz algorithm. If the three intervals are [2, 2.28], the current compression index parameter can be matched with the above-mentioned first interval, second interval and third interval, and the algorithm corresponding to the successful matching is used as the current target compression algorithm. For example, if the current compression index parameter is 2.4, the current target compression algorithm may be the lzma algorithm.

In another application example of the present invention, it is assumed that the number of CPU cores corresponding to the lzma algorithm, the zlib algorithm and the quicklz algorithm are respectively above 8 cores, between 4 cores and 8 cores, and below 4 cores; The parameters are matched with the above three CPU cores, and the algorithm corresponding to the successful matching is used as the current target compression algorithm. For example, if the current number of CPU cores of the storage device is 6 cores, the zlib algorithm that best fits the current number of CPU cores can be selected.

It can be understood that the above compression ratio parameter and the number of CPU cores can be used in combination. Specifically, the matching results of the two compression index parameters can be weighted and averaged, and the compression algorithm corresponding to the highest weighted average result can be selected as the target compression algorithm, etc., It can be understood that the embodiment of the present invention does not limit the specific process of using the compression ratio parameter and the number of CPU cores in combination.

It should be noted that the above-mentioned number of CPU cores is only an optional embodiment of the device parameters. In fact, the above-mentioned device parameters may also include: disk parameters used to characterize disk performance and memory parameters used to characterize memory performance, etc. The present invention The embodiment does not limit specific device parameters.

In addition, it should be noted that the target storage engine in this embodiment may be the target storage engine selected by using the embodiment shown in FIG. 2, or may be the storage engine specified by the user. The acquisition process is not limited.

For the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the described sequence of actions, because according to the embodiments of the present invention , certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

Referring to FIG. 4, a schematic structural diagram of a compressed storage device according to an embodiment of the present invention is shown, which may specifically include the following modules:

an obtaining module 401, configured to obtain compression parameters of the data to be stored, and/or obtain device parameters of a storage device corresponding to the data to be stored;

a selection module 402, configured to select a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters; and

The compression storage module 403 is configured to compress and store the data to be stored by using the target compression scheme.

In an optional embodiment of the present invention, the compression parameters may specifically include application scenario parameters and/or database architecture parameters, and the selection module 402 may further include:

a first selection submodule, configured to select a target storage engine corresponding to the application scenario parameters and/or database architecture parameters from at least one storage engine of the database; and

The second selection sub-module is configured to select a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm can form the target compression scheme.

In another optional embodiment of the present invention, the first selection sub-module may further include:

a first selection unit, configured to select the corresponding first storage engine as the target storage engine when the application scenario parameter is the first application scenario parameter; or,

a second selection unit, configured to select the corresponding second storage engine as the target storage engine when the application scenario parameter is the second application scenario parameter;

Wherein, the first application scenario parameters may specifically include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

The second application scenario parameters may specifically include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

In yet another optional embodiment of the present invention, the first selection sub-module may further include:

a third selection unit, configured to select the corresponding third storage engine as the target storage engine when the database schema parameter is the first database schema parameter; or,

a fourth selection unit, configured to select the corresponding fourth storage engine as the target storage engine when the database schema parameter is the second database schema parameter;

Wherein, the first database architecture parameters may specifically include: master-slave architecture parameters; or, the second database architecture parameters may specifically include: non-master-slave architecture parameters.

In another optional embodiment of the present invention, the compression parameters may specifically include compression index parameters, and the selection module 402 may further include:

A third selection sub-module, configured to select a target compression algorithm that fits the compression index parameter and/or the device parameter from at least one compression algorithm of the target storage engine, the target storage engine and its corresponding target compression algorithm The target compression scheme can be composed.

In an optional embodiment of the present invention, the device parameters may specifically include at least one of the following parameters: CPU parameters, disk parameters, and memory parameters.

In another optional embodiment of the present invention, the compression index parameter may specifically include at least one of the following parameters: a compression time parameter and a compression ratio parameter.

In yet another optional embodiment of the present invention, the target compression algorithm conforming to the compression index parameter and/or the device parameter may specifically include:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

As for the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and reference may be made to the partial description of the method embodiment for related parts.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not directed to any particular programming language. It is to be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or its description. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and further they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although some of the embodiments described herein include certain features, but not others, included in other embodiments, that combinations of features of different embodiments are intended to be within the scope of the invention within and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all of the components in the compressed storage method and apparatus according to the embodiments of the present invention. The present invention can also be implemented as apparatus or apparatus programs (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet platforms, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

The invention discloses A1, a compression storage method, comprising:

Obtain the compression parameters of the data to be stored, and/or obtain the device parameters of the storage device corresponding to the data to be stored;

Selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters;

The data to be stored is compressed and stored by using the target compression scheme.

A2. The method according to A1, wherein the compression parameters include application scene parameters and/or database architecture parameters, then the target compression scheme corresponding to the data to be stored is selected according to the compression parameters and/or device parameters The steps further include:

Select a target storage engine corresponding to the application scenario parameters and/or database schema parameters from at least one storage engine in the database;

A corresponding target compression algorithm is selected from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

A3. The method according to A2, wherein the step of selecting a target storage engine corresponding to the application scenario parameter from at least one storage engine in the database further includes:

If the application scenario parameter is the first application scenario parameter, select the corresponding first storage engine as the target storage engine; or,

If the application scenario parameter is the second application scenario parameter, then select the corresponding second storage engine as the target storage engine;

Wherein, the first application scenario parameters include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

Wherein, the second application scenario parameters include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

A4. The method according to A2, wherein the step of selecting a target storage engine corresponding to the database schema parameter from at least one storage engine in the database further comprises:

If the database architecture parameter is the first database architecture parameter, the corresponding third storage engine is selected as the target storage engine; or,

If the database architecture parameter is the second database architecture parameter, the corresponding fourth storage engine is selected as the target storage engine;

Wherein, the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameters include: non-master-slave architecture parameters.

A5. The method according to A1, wherein the compression parameters include compression index parameters, and the step of selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters further includes:

A target compression algorithm that fits the compression index parameter and/or the device parameter is selected from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

A6. The method according to A5, wherein the device parameters include at least one of the following parameters: a CPU parameter, a disk parameter, and a memory parameter.

A7. The method according to A5, wherein the compression index parameter includes at least one of the following parameters: a compression time parameter and a compression ratio parameter.

A8. The method according to any one of A5 to A7, wherein the target compression algorithm conforming to the compression index parameter and/or the device parameter includes:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

The present invention discloses B9, a compressed storage device, comprising:

an acquisition module, used for acquiring compression parameters of the data to be stored, and/or acquiring device parameters of the storage device corresponding to the data to be stored;

a selection module, configured to select a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters; and

A compression storage module, configured to compress and store the data to be stored by using the target compression scheme.

B10. The device according to B9, wherein the compression parameters include application scenario parameters and/or database architecture parameters, and the selection module further includes:

a first selection submodule, configured to select a target storage engine corresponding to the application scenario parameters and/or database architecture parameters from at least one storage engine of the database; and

The second selection sub-module is configured to select a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, and the target storage engine and its corresponding target compression algorithm constitute the target compression scheme.

B11. The device according to B10, wherein the first selection submodule further comprises:

a first selection unit, configured to select the corresponding first storage engine as the target storage engine when the application scenario parameter is the first application scenario parameter; or,

a second selection unit, configured to select the corresponding second storage engine as the target storage engine when the application scenario parameter is the second application scenario parameter;

Wherein, the first application scenario parameters include: at least one of online transaction processing scenario parameters and read-intensive scenario parameters;

Wherein, the second application scenario parameters include: online analytical processing scenario parameters, and at least one of batch loading and read-intensive scenario parameters.

B12. The apparatus according to B10, wherein the first selection submodule further comprises:

a third selection unit, configured to select the corresponding third storage engine as the target storage engine when the database schema parameter is the first database schema parameter; or,

a fourth selection unit, configured to select the corresponding fourth storage engine as the target storage engine when the database schema parameter is the second database schema parameter;

Wherein, the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameters include: non-master-slave architecture parameters.

B13. The device according to B9, wherein the compression parameters include compression index parameters, and the selection module further includes:

A third selection sub-module, configured to select a target compression algorithm that fits the compression index parameter and/or the device parameter from at least one compression algorithm of the target storage engine, the target storage engine and its corresponding target compression algorithm compose the target compression scheme.

B14. The apparatus according to B13, wherein the device parameters include at least one of the following parameters: a CPU parameter, a disk parameter, and a memory parameter.

B15. The apparatus according to B13, wherein the compression index parameter includes at least one of the following parameters: a compression time parameter and a compression ratio parameter.

B16. The apparatus according to any one of B13 to B15, wherein the target compression algorithm conforming to the compression index parameter and/or the device parameter includes:

The first target compression algorithm that fits the first-level compression index parameters; or

a second-target compression algorithm that fits the parameters of the second-level compression indicator; or

A third target compression algorithm that conforms to the third-level compression index parameter; wherein the compression index corresponding to the first-level compression index parameter, the second-level compression index parameter, and the third-level compression index parameter decreases; or

the first target compression algorithm that fits the first-level device parameters; or

a second-target compression algorithm that fits the second-level device parameters; or

A third target compression algorithm conforming to the third-level device parameters; wherein, the device performance corresponding to the first-level device parameters, the second-level device parameters, and the third-level device parameters decreases.

Claims (14)

1. A method of compressed storage, comprising:

acquiring compression parameters of data to be stored, and/or acquiring equipment parameters of storage equipment corresponding to the data to be stored;

selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or the equipment parameters;

compressing and storing the data to be stored by utilizing the target compression scheme; if the compression parameters include application scenario parameters and/or database architecture parameters, the step of selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or device parameters further includes:

selecting a target storage engine corresponding to the application scene parameters and/or the database architecture parameters from at least one storage engine of a database;

and selecting a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, wherein the target storage engine and the corresponding target compression algorithm form the target compression scheme.

2. The method of claim 1, wherein the step of selecting the target storage engine corresponding to the application scenario parameter from at least one storage engine of the database further comprises:

if the application scene parameters are first application scene parameters, selecting a corresponding first storage engine as a target storage engine; or,

if the application scene parameters are second application scene parameters, selecting a corresponding second storage engine as a target storage engine;

wherein the first application scene parameter comprises: at least one of an online transaction scenario parameter and a read intensive scenario parameter;

wherein the second application scenario parameter comprises: the on-line analysis processes at least one of scene parameters, and batch load and read intensive scene parameters.

3. The method of claim 1, wherein the step of selecting a target storage engine from at least one storage engine of a database corresponding to the database architecture parameter further comprises:

if the database architecture parameter is a first database architecture parameter, selecting a corresponding third storage engine as a target storage engine; or,

if the database architecture parameter is a second database architecture parameter, selecting a corresponding fourth storage engine as a target storage engine;

wherein the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameter includes: non-master-slave architecture parameters.

4. The method according to claim 1, wherein the compression parameter includes a compression index parameter, and the step of selecting the target compression scheme corresponding to the data to be stored according to the compression parameter and/or the device parameter further includes:

and selecting a target compression algorithm matched with the compression index parameter and/or the equipment parameter from at least one compression algorithm of a target storage engine, wherein the target storage engine and the corresponding target compression algorithm form the target compression scheme.

5. The method of claim 4, wherein the device parameters comprise at least one of: CPU parameters, disk parameters, and memory parameters.

6. The method of claim 4, wherein the compression indicator parameter comprises at least one of: a compression time parameter and a compression ratio parameter.

7. The method according to any one of claims 4 to 6, wherein the target compression algorithm fitting the compression index parameter and/or the device parameter comprises:

a first target compression algorithm fitting the first-stage compression index parameter; or

A second target compression algorithm fitting the second-stage compression index parameter; or

A third target compression algorithm fitting the third-stage compression index parameter; the compression indexes corresponding to the first-stage compression index parameter, the second-stage compression index parameter and the third-stage compression index parameter are decreased progressively; or

A first target compression algorithm fitting the parameters of the first-stage equipment; or

A second target compression algorithm fitting the parameters of the second-stage equipment; or

A third target compression algorithm fitting the third-stage equipment parameters; and the device performance corresponding to the first-stage device parameter, the second-stage device parameter and the third-stage device parameter is decreased progressively.

8. A compressed memory device, comprising:

the acquisition module is used for acquiring compression parameters of data to be stored and/or acquiring device parameters of storage equipment corresponding to the data to be stored;

the selection module is used for selecting a target compression scheme corresponding to the data to be stored according to the compression parameters and/or the equipment parameters; and

the compression storage module is used for compressing and storing the data to be stored by utilizing the target compression scheme; the compression parameters include application scenario parameters and/or database architecture parameters, and the selecting module further includes:

the first selection submodule is used for selecting a target storage engine corresponding to the application scene parameters and/or the database architecture parameters from at least one storage engine of a database; and

and the second selection submodule is used for selecting a corresponding target compression algorithm from at least one compression algorithm of the target storage engine, and the target storage engine and the corresponding target compression algorithm form the target compression scheme.

9. The apparatus of claim 8, wherein the first selection submodule further comprises:

the first selection unit is used for selecting the corresponding first storage engine as a target storage engine when the application scene parameter is the first application scene parameter; or,

the second selection unit is used for selecting the corresponding second storage engine as a target storage engine when the application scene parameters are the second application scene parameters;

wherein the first application scene parameter comprises: at least one of an online transaction scenario parameter and a read intensive scenario parameter;

wherein the second application scenario parameter comprises: the on-line analysis processes at least one of scene parameters, and batch load and read intensive scene parameters.

10. The apparatus of claim 8, wherein the first selection submodule further comprises:

the third selection unit is used for selecting a corresponding third storage engine as a target storage engine when the database architecture parameter is the first database architecture parameter; or,

the fourth selecting unit is used for selecting a corresponding fourth storage engine as a target storage engine when the database architecture parameter is the second database architecture parameter;

wherein the first database architecture parameters include: master-slave architecture parameters; or, the second database architecture parameter includes: non-master-slave architecture parameters.

11. The apparatus of claim 8, wherein the compression parameter comprises a compression index parameter, the selection module further comprising:

and the third selection submodule is used for selecting a target compression algorithm which is matched with the compression index parameter and/or the equipment parameter from at least one compression algorithm of a target storage engine, and the target storage engine and the corresponding target compression algorithm form the target compression scheme.

12. The apparatus of claim 11, wherein the device parameter comprises at least one of: CPU parameters, disk parameters, and memory parameters.

13. The apparatus of claim 11, wherein the compression indicator parameter comprises at least one of: a compression time parameter and a compression ratio parameter.

14. The apparatus according to any one of claims 11 to 13, wherein the target compression algorithm fitting the compression index parameter and/or the device parameter comprises:

a first target compression algorithm fitting the first-stage compression index parameter; or

A second target compression algorithm fitting the second-stage compression index parameter; or

A third target compression algorithm fitting the third-stage compression index parameter; the compression indexes corresponding to the first-stage compression index parameter, the second-stage compression index parameter and the third-stage compression index parameter are decreased progressively; or

A first target compression algorithm fitting the parameters of the first-stage equipment; or

A second target compression algorithm fitting the parameters of the second-stage equipment; or

A third target compression algorithm fitting the third-stage equipment parameters; and the device performance corresponding to the first-stage device parameter, the second-stage device parameter and the third-stage device parameter is decreased progressively.

Images