HK40020782B - Data transmission method and device basedon cloud storage and computer equipment - Google Patents

Description

Cloud storage-based data transmission methods, devices, and computer equipment

Technical Field

This invention relates to the field of cloud storage technology, and in particular to a data transmission method, apparatus, computer equipment, and storage medium based on cloud storage.

Background Technology

Currently, when writing data from a Hive database (Hive is a data warehouse tool that can map structured data files to a database table) to HBase (HBase is a distributed, column-oriented open-source database), offline batch writing or streaming writing is generally used. However, both of these methods use the put command (put is one of the data insertion methods in HBase) to write data to HBase. When inserting data using the put command, sorting is done at the same time, which affects the data processing efficiency of the HBase cluster and results in low data writing efficiency.

Summary of the Invention

This invention provides a data transmission method, apparatus, computer device, and storage medium based on cloud storage, aiming to solve the problem that in the prior art, data is written to HBase using the put method, and data is inserted while sorting, which affects the data processing efficiency of the HBase cluster and results in low data writing efficiency.

In a first aspect, embodiments of the present invention provide a data transmission method based on cloud storage, comprising:

Receive and store all data uploaded from a Hive database; wherein the Hive database is a data warehouse database.

Get the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open source database, and each pre-partition in the HBase database corresponds to a partition server;

Based on the number of pre-partitions and the row keys of each data in the full data, the full data is partitioned to obtain the corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server;

Sort the data in each partition in ascending order according to the column and row keys to obtain the corresponding sorted partition data; and

The sorted partition data is sent to the partition server corresponding to the HBase database for storage.

Secondly, embodiments of the present invention provide a data transmission device based on cloud storage, comprising:

A receiving unit is used to receive and store the full amount of data uploaded from the Hive database; wherein the Hive database is a data warehouse database.

The partition count acquisition unit is used to obtain the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open source database, and each pre-partition in the HBase database corresponds to a partition server;

A partitioning unit is used to partition the full data according to the number of pre-partitions and the row key of each data in the full data to obtain corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server;

The sorting unit is used to sort the data in each partition in ascending order according to the column and row keys, thus obtaining the corresponding sorted partition data; and

The transmission unit is used to send the sorted partition data to the partition server corresponding to the HBase database for storage.

Thirdly, embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the cloud storage-based data transmission method described in the first aspect.

Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the cloud storage-based data transmission method described in the first aspect.

This invention provides a data transmission method, apparatus, computer device, and storage medium based on cloud storage, which enables the sorting process to be completed in the cloud before all data is written to the HBase database, thereby improving the efficiency of writing data to the HBase database.

Attached Figure Description

To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

Figure 1 is a schematic diagram of an application scenario of the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 2 is a flowchart illustrating the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 3 is a schematic diagram of a sub-process of the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 4 is a schematic diagram of another sub-process of the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 5 is a schematic diagram of another sub-process of the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 6 is a schematic diagram of another sub-process of the data transmission method based on cloud storage provided in an embodiment of the present invention;

Figure 7 is a schematic block diagram of a cloud storage-based data transmission device provided in an embodiment of the present invention;

Figure 8 is a schematic block diagram of a subunit of a cloud storage-based data transmission device provided in an embodiment of the present invention;

Figure 9 is a schematic block diagram of another sub-unit of the cloud storage-based data transmission device provided in an embodiment of the present invention;

Figure 10 is a schematic block diagram of another sub-unit of the cloud storage-based data transmission device provided in an embodiment of the present invention;

Figure 11 is a schematic block diagram of another sub-unit of the cloud storage-based data transmission device provided in an embodiment of the present invention;

Figure 12 is a schematic block diagram of a computer device provided in an embodiment of the present invention.

Detailed Implementation

The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and/or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and/or collections thereof.

It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

It should also be further understood that the term "and/or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

Please refer to Figures 1 and 2. Figure 1 is a schematic diagram of an application scenario of the data transmission method based on cloud storage provided in an embodiment of the present invention; Figure 2 is a schematic diagram of the process of the data transmission method based on cloud storage provided in an embodiment of the present invention. The data transmission method based on cloud storage is applied to a server and is executed by application software installed on the server.

As shown in Figure 2, the method includes steps S110 to S150.

S110. Receive and store the full amount of data uploaded from the Hive database; wherein the Hive database is a data warehouse database.

In this embodiment, the technical solution is described from the perspective of a cloud computing platform. Specifically, the cloud computing platform used in this application is Spark. Spark is a fast and general-purpose computing engine designed for large-scale data processing. Spark enables in-memory distributed datasets, which, in addition to providing interactive queries, can also optimize iterative workloads.

After the cloud computing platform receives the full amount of data uploaded from the Hive database, it generates logical dataframes (a dataframe is a collection of rows of a dataset, and a dataset is a new interface added in Spark 1.6+) for physical storage (physical storage is a combination of memory and disk storage).

S120. Obtain the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open-source database, and each pre-partition in the HBase database corresponds to a partition server.

In this embodiment, after the storage of all data is completed on the cloud computing platform, in order to know how many partitions the full data will be divided into for subsequent storage, it is necessary to first obtain the number of pre-partitions from the HBase database.

The HBase database is a distributed open-source database, and each pre-partition in the HBase database corresponds to a partition server. It is a highly reliable, high-performance, column-oriented, and scalable distributed storage system based on Hadoop. Using HBase technology, large-scale structured storage clusters can be built on inexpensive computer servers.

In one embodiment, as shown in FIG3, step S120 includes:

S121. Send an RPC request to the HBase database; wherein, the RPC request is a Remote Procedure Call protocol request;

S122. Receive the metadata sent by the HBase database according to the RPC request, and obtain the number of pre-partitions based on the metadata.

In this embodiment, after the cloud computing platform has completed storing all the data, it initiates an RPC request (Remote Procedure Call, a request to request services from a remote computer program over a network) to access the ZooKeeper metadata (a distributed, open-source distributed application coordination service) of the HBase database. The ZooKeeper metadata stores the partition information of the pre-built tables in HBase, thus revealing the number of pre-partitions in the HBase database. Knowing the number of pre-partitions in the HBase database allows for accurate division of the entire dataset into the same number of partitions.

S130. Based on the number of pre-partitions and the row key of each data in the full data, the full data is partitioned to obtain the corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server.

In this embodiment, the entire data stored in the dataframe of the cloud computing platform is distributed into corresponding partitions based on the HexStringSplit pre-partitioning method. HexStringSplit is a pre-partitioning method suitable for rows whose row keys are prefixed with hexadecimal strings.

In one embodiment, as shown in FIG4, step S130 includes:

S131. Obtain the row key corresponding to each data in the full data;

S132. Generate the corresponding hash value for the row key of each data using the MD5 encryption algorithm or the SHA-256 encryption algorithm;

S133. Take the modulo of the hash value corresponding to each row key with the number of pre-partitions to obtain the remainder corresponding to each row key;

S134. Store the data corresponding to each row key into the partition corresponding to the remainder of that row key to obtain the corresponding partition data.

In this embodiment, each piece of data in Spark has a corresponding row key. The row key of each piece of data is obtained first, so that the data can be processed and divided into the corresponding regions.

Then, when the row keys of each data item are calculated using the MD5 or SHA encryption algorithm, a corresponding hash value is generated. The MD5 algorithm is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure the integrity and consistency of transmitted information. The SHA-256 algorithm is a secure hash algorithm that calculates a fixed-length string (also known as a message digest) corresponding to a digital message. By generating hash values for the row keys using MD5 or SHA-256, the hash values are distributed across the corresponding partitions, ensuring that data with the same row key remainder is grouped into the same partition. This method achieves fast and efficient partitioning of the entire dataset.

Since each pre-partition in the HBase database corresponds to a partition server, and each partition's data uniquely corresponds to a partition server, the correspondence between partition data and partition servers can be pre-set. For example, partition 1 corresponds to partition server 1, ..., partition N corresponds to partition server N. Once the correspondence between each partition's data and its partition server is known, targeted storage can be implemented during subsequent data storage, improving storage efficiency.

S140. Sort each partition's data in ascending order according to the column and row keys to obtain the corresponding sorted partition data.

In this embodiment, after the full data is partitioned according to the number of pre-defined partitions in the cloud computing platform, the data in each partition needs to be sorted. Once sorting is complete, the data is sent to the HBase database for fast storage. When sorting the data in each partition, the column value and row key value can be selected for sorting.

In one embodiment, as shown in FIG5, step S140 includes:

S141. Obtain the data with the same row key from each partition data, sort the data with the same row key according to the ascending order of the column, and obtain the first sorted partition data corresponding to each partition data.

S142. Sort each first sorted partition data in ascending order according to the row key to obtain the sorted partition data corresponding to each first sorted partition data.

In this embodiment, within each partition, data with the same row key value are first grouped together. Within each partition, data with the same row key value is then sorted in ascending order by column value, resulting in the first sorted partition data. This first sorted partition data can then be further sorted according to the row key in ascending order to obtain the next sorted partition data corresponding to each first sorted partition. Therefore, sorting the partition data by column and row key allows for more systematic data storage.

S150. Send the sorted partition data to the partition server corresponding to the HBase database for storage.

In this embodiment, after sorting the data in each partition to obtain the corresponding sorted partition data, it can be directly sent to the HBase database for storage. There is no need to sort and insert data simultaneously when using the put command, which would affect the data processing efficiency of the HBase cluster. The data that has been partitioned and sorted is directly stored in the HBase database, which improves storage efficiency.

In one embodiment, as shown in FIG6, step S150 includes:

S151. Input the sorted partition data into the local HDFS layer to convert the sorted partition data into corresponding data files; wherein, the HDFS layer is a distributed file system layer.

S152. The data file is sent to the partition server corresponding to the HBase database for storage.

In this embodiment, the lowest layer of the cloud computing platform (i.e., Spark) is the HDFS layer, which stores data. The sorted partition data is input into the HDFS layer, which then converts the sorted partition data into data files. Specifically, the data files are HFile files, which consist of 7 blocks. These blocks can be categorized as follows:

a) Datablock (datablock is a block of data), which stores key-value data (i.e., key-value pair data). The default size of a datablock is 64KB.

b) Data index block, which stores the index of the data block (index is the index). The index can be a multi-level index, intermediate index, leaf index, which is usually distributed in the HFile file;

c) The bloom filter block stores the values of the bloom filter.

d) Meta data block: There are multiple meta data blocks, which are distributed continuously.

e) Meta data index, representing the index of metadata;

f) The file-info block records information about the file, such as the largest key in the HFile, the average key length, the HFile creation timestamp, and the encoding method used by the data block.

g) Trailer block (i.e., the end of the message): Every HFile file has a trailer block. The trailer length may be different for different versions of HFile (there are three versions: V1, V2, and V3, with little difference between V2 and V3). However, the trailer length of all HFiles of the same version is the same, and the last 4 bytes of the trailer must be version information.

As can be seen, the sorted partition data is stored in the local HDFS layer, and it is stored by converting it into HFile files.

Once the sorted partition data is converted into HFile files at the HSFS layer, the corresponding HFile files can be sent to the corresponding partition server in the HBase database. The HBase partition server then uses the Bulkload scheme (i.e., the main loading scheme) to write the HFiles into the HBase database. The advantages of Bulkload are that the import process does not consume partition resources; it can quickly import massive amounts of data; and it saves memory.

In one embodiment, step S150 is followed by:

If a data transmission error message sent by the HBase database is detected, the data transmission interruption point is located in the sorted partition data according to the log file corresponding to the data transmission error message.

Data located after the data transmission interruption point of each sorted partition is sent to the partition server corresponding to the HBase database for storage.

In this embodiment, if a transmission interruption occurs during the process of sending the sorted partition data to the HBase database for storage, a data transmission error message sent by the HBase database can be received. The interruption point can then be located in the sorted partition data based on the log file corresponding to the error message. Once the interruption point is identified, data transmission can resume from the data after that point, ensuring normal transmission can be restored even after an anomaly occurs.

This method enables the sorting process to be completed in the cloud before all data is written to the HBase database, thus improving the efficiency of writing data to the HBase database.

This invention also provides a cloud storage-based data transmission device, which is used to execute any of the aforementioned cloud storage-based data transmission methods. Specifically, please refer to FIG7, which is a schematic block diagram of the cloud storage-based data transmission device provided in this invention. This cloud storage-based data transmission device 100 can be configured in a server.

As shown in Figure 7, the cloud storage-based data transmission device 100 includes a receiving unit 110, a partition count acquisition unit 120, a partitioning unit 130, a sorting unit 140, and a transmission unit 150.

The receiving unit 110 is used to receive and store the full amount of data uploaded by the Hive database; wherein the Hive database is a data warehouse database.

In this embodiment, the technical solution is described from the perspective of a cloud computing platform. Specifically, the cloud computing platform used in this application is Spark. Spark is a fast and general-purpose computing engine designed for large-scale data processing. Spark enables in-memory distributed datasets, which, in addition to providing interactive queries, can also optimize iterative workloads.

After the cloud computing platform receives the full amount of data uploaded from the Hive database, it generates logical dataframes (a dataframe is a collection of rows of a dataset, and a dataset is a new interface added in Spark 1.6+) for physical storage (physical storage is a combination of memory and disk storage).

The partition count acquisition unit 120 is used to obtain the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open source database, and each pre-partition in the HBase database corresponds to a partition server.

In this embodiment, after the storage of all data is completed on the cloud computing platform, in order to know how many partitions the full data will be divided into for subsequent storage, it is necessary to first obtain the number of pre-partitions from the HBase database.

The HBase database is a distributed open-source database, and each pre-partition in the HBase database corresponds to a partition server. It is a highly reliable, high-performance, column-oriented, and scalable distributed storage system based on Hadoop. Using HBase technology, large-scale structured storage clusters can be built on inexpensive computer servers.

In one embodiment, as shown in FIG8, the partition count acquisition unit 120 includes:

The request sending unit 121 is used to send an RPC request to the HBase database; wherein the RPC request is a Remote Procedure Call protocol request;

Meta-information parsing unit 122 is used to receive meta-information sent by the HBase database according to the RPC request, and obtain the number of pre-partitions based on the meta-information.

In this embodiment, after the cloud computing platform has completed storing all the data, it initiates an RPC request (Remote Procedure Call, a request to request services from a remote computer program over a network) to access the ZooKeeper metadata (a distributed, open-source distributed application coordination service) of the HBase database. The ZooKeeper metadata stores the partition information of the pre-built tables in HBase, thus revealing the number of pre-partitions in the HBase database. Knowing the number of pre-partitions in the HBase database allows for accurate division of the entire dataset into the same number of partitions.

Partitioning unit 130 is used to partition the full data according to the number of pre-partitions and the row key of each data in the full data to obtain corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server.

In this embodiment, the entire data stored in the dataframe of the cloud computing platform is distributed into corresponding partitions based on the HexStringSplit pre-partitioning method. HexStringSplit is a pre-partitioning method suitable for rows whose row keys are prefixed with hexadecimal strings.

In one embodiment, as shown in FIG9, the partitioning unit 130 includes:

Row key acquisition unit 131 is used to acquire the row key corresponding to each data in the full data;

Hash unit 132 is used to generate a corresponding hash value for each row key of the data using the MD5 encryption algorithm or the SHA-256 encryption algorithm;

Modulo operation unit 133 is used to calculate the modulo of the hash value corresponding to each row key with the number of pre-partitions to obtain the remainder corresponding to each row key;

Data partitioning unit 134 is used to store the data corresponding to each row key into the partition corresponding to the remainder of the row key, so as to obtain the corresponding partition data.

In this embodiment, each piece of data in Spark has a corresponding row key. The row key of each piece of data is obtained first, so that the data can be processed and divided into the corresponding regions.

Then, when the row keys of each data item are calculated using the MD5 or SHA encryption algorithm, a corresponding hash value is generated. The MD5 algorithm is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure the integrity and consistency of transmitted information. The SHA-256 algorithm is a secure hash algorithm that calculates a fixed-length string (also known as a message digest) corresponding to a digital message. By generating hash values for the row keys using MD5 or SHA-256, the hash values are distributed across the corresponding partitions, ensuring that data with the same row key remainder is grouped into the same partition. This method achieves fast and efficient partitioning of the entire dataset.

Since each pre-partition in the HBase database corresponds to a partition server, and each partition's data uniquely corresponds to a partition server, the correspondence between partition data and partition servers can be pre-set. For example, partition 1 corresponds to partition server 1, ..., partition N corresponds to partition server N. Once the correspondence between each partition's data and its partition server is known, targeted storage can be implemented during subsequent data storage, improving storage efficiency.

The sorting unit 140 is used to sort the data of each partition in ascending order according to the column and row keys to obtain the corresponding sorted partition data.

In this embodiment, after the full data is partitioned according to the number of pre-defined partitions in the cloud computing platform, the data in each partition needs to be sorted. Once sorting is complete, the data is sent to the HBase database for fast storage. When sorting the data in each partition, the column value and row key value can be selected for sorting.

In one embodiment, as shown in FIG10, the sorting unit 140 includes:

The first sorting unit 141 is used to obtain data with the same row key in each partition data, sort the data with the same row key according to the ascending order of the column, and obtain the first sorted partition data corresponding to each partition data.

The second sorting unit 142 is used to sort each first sorted partition data according to the ascending order of the row key to obtain sorted partition data corresponding to each first sorted partition data.

In this embodiment, within each partition, data with the same row key value are first grouped together. Within each partition, data with the same row key value is then sorted in ascending order by column value, resulting in the first sorted partition data. This first sorted partition data can then be further sorted according to the row key in ascending order to obtain the next sorted partition data corresponding to each first sorted partition. Therefore, sorting the partition data by column and row key allows for more systematic data storage.

The transmission unit 150 is used to send the sorted partition data to the partition server corresponding to the HBase database for storage.

In this embodiment, after sorting the data in each partition to obtain the corresponding sorted partition data, it can be directly sent to the HBase database for storage. There is no need to sort and insert data simultaneously when using the put command, which would affect the data processing efficiency of the HBase cluster. The data that has been partitioned and sorted is directly stored in the HBase database, which improves storage efficiency.

In one embodiment, as shown in FIG11, the transmission unit 150 includes:

The underlying storage unit 151 is used to input the sorted partition data to the local HDFS layer to convert the sorted partition data into corresponding data files; wherein, the HDFS layer is a distributed file system layer.

The data sending unit 152 is used to send the data file to the partition server corresponding to the HBase database for storage.

In this embodiment, the lowest layer of the cloud computing platform (i.e., Spark) is the HDFS layer used for data storage. The sorted partition data is input into the HDFS layer, which then converts the sorted partition data into data files. Therefore, the sorted partition data is stored locally in the HDFS layer and is stored as HFile files.

Once the sorted partition data is converted into HFile files at the HSFS layer, the corresponding HFile files can be sent to the corresponding partition server in the HBase database. The HBase partition server then uses the Bulkload scheme (i.e., the main loading scheme) to write the HFiles into the HBase database. The advantages of Bulkload are that the import process does not consume partition resources; it can quickly import massive amounts of data; and it saves memory.

In one embodiment, the cloud storage-based data transmission device 100 further includes:

The interruption point acquisition unit is used to locate and acquire the data transmission interruption point in each sorted partition data according to the log file corresponding to the data transmission error information if a data transmission error information has been received from the HBase database.

The data transmission recovery unit is used to send the data located after the data transmission interruption point of each sorted partition data to the partition server corresponding to the HBase database for storage.

In this embodiment, if a transmission interruption occurs during the process of sending the sorted partition data to the HBase database for storage, a data transmission error message sent by the HBase database can be received. The interruption point can then be located in the sorted partition data based on the log file corresponding to the error message. Once the interruption point is identified, data transmission can resume from the data after that point, ensuring normal transmission can be restored even after an anomaly occurs.

This device enables the sorting process to be completed in the cloud before all data is written to the HBase database, thus improving the efficiency of writing data to the HBase database.

The aforementioned cloud storage-based data transmission device can be implemented as a computer program, which can run on the computer device shown in Figure 12.

Please refer to Figure 12, which is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 is a server, which can be a standalone server or a server cluster composed of multiple servers.

Referring to Figure 12, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform a cloud storage-based data transfer method.

The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a cloud storage-based data transmission method.

The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art will understand that the structure shown in FIG12 is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. A specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

The processor 502 is used to run a computer program 5032 stored in a memory to implement the cloud storage-based data transmission method disclosed in this embodiment of the invention.

Those skilled in the art will understand that the embodiments of the computer device shown in FIG12 do not constitute a limitation on the specific configuration of the computer device. In other embodiments, the computer device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, in some embodiments, the computer device may include only a memory and a processor. In such embodiments, the structure and function of the memory and processor are consistent with those shown in FIG12, and will not be repeated here.

It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the cloud storage-based data transmission method disclosed in this embodiment of the invention.

Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims (8)

1. A data transmission method based on cloud storage, characterized in that it includes: Receive and store all data uploaded from a Hive database; wherein the Hive database is a data warehouse database. Get the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open source database, and each pre-partition in the HBase database corresponds to a partition server; Based on the number of pre-partitions and the row keys of each data in the full data, the full data is partitioned to obtain the corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server; Sort the data in each partition in ascending order according to the column and row keys to obtain the corresponding sorted partition data; and The sorted partition data is sent to the partition server corresponding to the HBase database for storage; The process of obtaining the number of pre-partitions in the HBase database includes: Send an RPC request to the HBase database; wherein the RPC request is a Remote Procedure Call protocol request; Receive metadata sent by the HBase database according to the RPC request, and obtain the number of pre-partitions based on the metadata; The metadata contains partition information for tables pre-built in the HBase database. 2. The data transmission method based on cloud storage according to claim 1, characterized in that, the step of partitioning the full data according to the number of pre-partitions and the row key of each data in the full data to obtain the corresponding partition data includes: Obtain the row key corresponding to each data point in the full dataset; Generate the corresponding hash value for the row key of each data using the MD5 encryption algorithm or the SHA-256 encryption algorithm; Take the modulo of the hash value corresponding to each row key with the number of pre-partitions to obtain the remainder corresponding to each row key; The data corresponding to each row key is stored in the partition corresponding to the remainder of that row key, so as to obtain the corresponding partition data. 3. The data transmission method based on cloud storage according to claim 1, characterized in that, the step of sorting each partition data in ascending order according to the column and row keys to obtain the corresponding sorted partition data includes: In each partition, retrieve the data with the same row key, sort the data with the same row key in ascending order of the column, and obtain the first sorted partition data corresponding to each partition. Sort each first sorted partition data in ascending order according to the row key to obtain the sorted partition data corresponding to each first sorted partition data. 4. The data transmission method based on cloud storage according to claim 1, characterized in that, sending the sorted partition data to the partition server corresponding to the HBase database for storage includes: The sorted partition data is input into the local HDFS layer to convert the sorted partition data into corresponding data files; wherein, the HDFS layer is a distributed file system layer. The data file is sent to the partition server corresponding to the HBase database for storage. 5. The data transmission method based on cloud storage according to claim 1, characterized in that, after sending the sorted partition data to the partition server corresponding to the HBase database for storage, it further includes: If a data transmission error message is detected that has been received from the HBase database, the data transmission interruption point is located in the sorted partition data according to the log file corresponding to the data transmission error message. Data located after the data transmission interruption point of each sorted partition is sent to the partition server corresponding to the HBase database for storage. 6. A data transmission device based on cloud storage, characterized in that it comprises: A receiving unit is used to receive and store the full amount of data uploaded from the Hive database; wherein the Hive database is a data warehouse database. The partition count acquisition unit is used to obtain the number of pre-partitions in the HBase database; wherein, the HBase database is a distributed open source database, and each pre-partition in the HBase database corresponds to a partition server; A partitioning unit is used to partition the full data according to the number of pre-partitions and the row key of each data in the full data to obtain corresponding partition data; wherein, the total number of partitions of the partition data is equal to the number of pre-partitions, and each partition data uniquely corresponds to a partition server; The sorting unit is used to sort the data in each partition in ascending order according to the column and row keys, thus obtaining the corresponding sorted partition data; and The transmission unit is used to send the sorted partition data to the partition server corresponding to the HBase database for storage; The partition count acquisition unit includes: A request sending unit is used to send an RPC request to the HBase database; wherein the RPC request is a Remote Procedure Call protocol request; Meta-information parsing unit, used to receive meta-information sent by the HBase database according to the RPC request, and obtain the number of pre-partitions based on the meta-information; The metadata contains partition information for tables pre-built in the HBase database. 7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the data transmission method based on cloud storage as described in any one of claims 1 to 5. 8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the data transmission method based on cloud storage as described in any one of claims 1 to 5.