CN109886862B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN109886862B CN109886862B CN201910074154.8A CN201910074154A CN109886862B CN 109886862 B CN109886862 B CN 109886862B CN 201910074154 A CN201910074154 A CN 201910074154A CN 109886862 B CN109886862 B CN 109886862B
- Authority
- CN
- China
- Prior art keywords
- data set
- gpu
- compression
- controlling
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention provides a data processing method and device, which are used for solving the problems of low data transmission efficiency and high storage space overhead in the prior art. The method comprises the following steps: acquiring a first data set from a disk to a main memory; controlling the CPU to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access; acquiring the second data set from the main memory to a video memory of an image processor GPU; and controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
A Graphics Processing Unit (GPU) is a kind of hardware included in a computer, and the GPU may utilize a plurality of internal Processing cores to assist a Central Processing Unit (CPU) of the computer to perform parallel acceleration on multiple types of computing tasks, and fig. 1 is a schematic diagram of a computing and storage framework of a computer including the GPU. In the above process, when the data volume of the processed data is large, the transmission overhead of the processed data introduced from the disk into the main memory is large and a large storage space is required in the main memory, and when the data volume of the processed data is larger than the video memory of the GPU, a large amount of data switching needs to be performed between the main memory of the computer and the video memory of the GPU in the process of quantity processing, which further increases the transmission overhead and affects the transmission efficiency.
In the prior art, in order to reduce transmission overhead, before being stored in a disk, processed data is compressed by adopting an entropy-based compression method, when the compressed processed data stored in the disk needs to be decompressed, the compressed processed data stored in the disk is led into a main memory, and then the compressed processed data in the main memory is sent to a video memory.
In summary, how to reduce the overhead of the storage space while improving the data transmission efficiency is a problem to be solved at present.
Disclosure of Invention
In view of this, the present invention provides a data processing method and apparatus, which are used to solve the problems of low data transmission efficiency and large storage space overhead in the prior art.
According to a first aspect of the embodiments of the present invention, there is provided a data processing method, including: acquiring a first data set from a disk to a main memory; controlling a Central Processing Unit (CPU) to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access; acquiring the second data set from the main memory to a video memory of an image processor GPU; and controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm.
In one embodiment, after controlling the CPU to decompress the first data set for a first time according to a first decompression algorithm and determining a second data set after the decompression for the first time, the method further includes: and writing the second data set into a primary compressed data buffer of the main memory.
In an embodiment, the acquiring the second data set from the main memory to the video memory of the image processor GPU specifically includes: and acquiring the second data set from the primary compressed data buffer of the main memory to a video memory data buffer of a video memory of the GPU.
In one embodiment, after the controlling the at least one processing core of the GPU decompresses the second set of data according to a second decompression algorithm, the method includes: and removing the second data set in the video memory data buffer of the video memory of the GPU.
In an embodiment, the controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm specifically includes: and controlling a plurality of cores of the GPU to decompress the second data set for the second time in a parallel mode, determining a decompressed first original data set, and writing the first original data set into a video memory of the GPU.
In one embodiment, the controlling the multiple cores of the GPU to decompress the second data set for the second time in a parallel manner, determining a decompressed first original data set, and writing the first original data set into a video memory of the GPU specifically includes: and controlling the plurality of processing cores to acquire data from the first original data set in the video memory of the GPU for processing.
In an embodiment, the controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm specifically includes: and controlling the plurality of processing cores of the GPU to decompress the second data set for the second time, and writing the decompressed data into the registers corresponding to the plurality of processing cores respectively.
In an embodiment, after the controlling the plurality of processing cores of the GPU decompresses the second data set for the second time and writes the decompressed data into the registers corresponding to the plurality of processing cores, the method further includes: and controlling the plurality of processing cores to respectively acquire decompressed data from the corresponding registers for processing.
According to a second aspect of the embodiments of the present invention, there is provided a method of data compression, including: the CPU is controlled to segment the second original data set according to a set value, and at least one segmented first original data set is determined; controlling the CPU or the GPU to carry out first compression on the first original data set, and determining a data set after the first compression; controlling the CPU to carry out second compression on the data set after the first compression, and determining a first data set after compression; and sending the compressed first data set to a disk.
In one embodiment, the first compression is performed using a random access compression algorithm, and the second compression is performed using a non-random access compression algorithm.
According to a third aspect of the embodiments of the present invention, there is provided an apparatus for data processing, including: the first acquisition unit is used for acquiring a first data set from a disk to a main memory; the first decompression unit is used for controlling a Central Processing Unit (CPU) to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access; the second acquisition unit is used for acquiring the second data set from the main memory to the video memory of the image processor GPU; and the second decompression unit is used for controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm.
In one embodiment, the apparatus further comprises: and the writing unit is used for writing the second data set into a primary compressed data buffer area of the main memory.
In an embodiment, the second obtaining unit is specifically configured to: and acquiring the second data set from the primary compressed data buffer of the main memory to a video memory data buffer of a video memory of the image processor GPU.
In one embodiment, the apparatus further comprises: a removal unit, configured to remove the second data set in the video memory data buffer of the video memory of the GPU.
In an embodiment, the second decompression unit is specifically configured to: and controlling a plurality of cores of the GPU to decompress the second data set for the second time in a parallel mode, determining a decompressed first original data set, and writing the first original data set into a video memory of the GPU.
In an embodiment, the second decompression unit is specifically configured to: and controlling the plurality of processing cores to acquire data from the first original data set in the video memory of the GPU for processing.
In an embodiment, the second decompression unit is specifically configured to: and controlling the plurality of processing cores of the GPU to decompress the second data set for the second time, and writing the decompressed data into the registers corresponding to the plurality of processing cores respectively.
In an embodiment, the second decompression unit is specifically configured to: and controlling the plurality of processing cores to respectively acquire decompressed data from the corresponding registers for processing.
According to a fourth aspect of the embodiments of the present invention, there is provided an apparatus for data processing, including: the fragmentation unit is used for controlling the CPU to fragment the second original data set according to a set value and determining at least one first original data set after fragmentation; the first compression unit is used for controlling the CPU or the GPU to carry out first compression on the first original data set and determining a data set after the first compression; the second compression unit is used for controlling the CPU to carry out second compression on the data set after the first compression and determining a first compressed data set; and the sending unit is used for sending the compressed first data set to a disk.
In one embodiment, the first compression is performed using a random access compression algorithm, and the second compression is performed using a non-random access compression algorithm.
According to a fifth aspect of embodiments of the present invention, there is provided a computer readable storage medium having stored thereon computer program instructions, characterized in that the computer program instructions, when executed by a processor, implement the method according to the first aspect or any of the possibilities of the first aspect.
The embodiment of the invention has the beneficial effects that: firstly, a first data set is obtained from a disk to a main memory, then a Central Processing Unit (CPU) is controlled to decompress the first data set for the first time according to a first decompression algorithm, a second data set after the first decompression is determined, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access, then the second data set is obtained from the main memory to a video memory of an image processor (GPU), and finally at least one processing core of the GPU is controlled to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm. By the method, the primary data set is compressed for the second time, transmission overhead from a disk to a main memory can be reduced, storage space in the disk and the main memory is saved, the second data set subjected to the first decompression is decompressed and processed in parallel in the video memory of the GPU, decompression efficiency and data processing speed can be improved, the second data set does not need to be decompressed integrally and then processed, and storage space in the video memory of the GPU is saved.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a schematic diagram of a prior art computing and storage framework for a computer including a GPU;
FIG. 2 is a diagram of a data processing process in the prior art;
FIG. 3 is a flow chart of a method for data compression according to an embodiment of the present invention;
fig. 4 is a schematic diagram of data slice compression according to an embodiment of the present invention;
FIG. 5 is a flow chart of a method for processing data according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a data processing process according to an embodiment of the present invention;
FIG. 7 is a diagram of an apparatus for data compression according to an embodiment of the present invention;
FIG. 8 is a diagram of an apparatus for data processing according to an embodiment of the present invention;
fig. 9 is a flowchart of another data processing method according to an embodiment of the present invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present application may be practiced without these specific details. Further, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes.
Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including, but not limited to".
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only, and not for purposes of order, nor are they to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the prior art, under a schematic diagram of a computing and storage framework of a computer in fig. 1, when a GPU processes data, generally, as shown in fig. 2, an original data set is compressed by using a compression algorithm that is not randomly accessible, a compressed data set is determined, the compressed data set is stored in a disk, the disk sends the compressed data set to a main Memory, where the main Memory is a Memory shared by all processing mechanisms (e.g., a CPU), and when a Unified Memory Architecture (UMA) is used, the GPU may also access a part of a storage space in the main Memory, the CPU sends the compressed data set in the main Memory to a display Memory of the GPU, and a processing core of the GPU serially decompresses the compressed data set to obtain the original data set, where the original data set is written into the Memory, and the processing core of the GPU acquires the original data set from the display Memory to process the original data set. When the data is transmitted and stored in the above manner, serial decompression is required when the data is processed and decompressed in the video memory, the decompression time is long, and the compressed data set and the decompressed original data set need to be stored in the video memory, which results in large storage space overhead.
In order to solve the problems of low data transmission efficiency and high storage space overhead, a data compression method provided by the present invention, as shown in fig. 3 specifically, includes:
and step S300, controlling a Central Processing Unit (CPU) to segment the second original data set according to a set value, and determining at least one segmented first original data set.
Step S301, controlling the CPU or the GPU to perform first compression on the first original data set, and determining a data set after the first compression.
And step S302, controlling the CPU to carry out second compression on the data set after the first compression, and determining the first data set after compression.
Specifically, the first compression is performed by using a random access compression algorithm, and the second compression is performed by using a non-random access compression algorithm.
Step S303, sending the compressed first data set to a disk.
For example, as shown in fig. 4, assuming that a second original data set is larger than storage spaces of a disk, a memory and a video memory, the second original data set needs to be fragmented according to a set value, where the set value is determined according to the size of the storage space of the disk, the memory, the video memory and a data transmission rate, assuming that the size of the second original data set is 20G, the set value is determined to be 5G according to the size of the storage space of the disk, the memory, the video memory and the data transmission rate, before storing the second original data set in the disk, the second original data set needs to be fragmented into 4 parts, namely, a fragment 0, a fragment 1, a fragment 2 and a fragment 3, where any fragment is a first original data set, and the first original data set after each fragment is compressed by using a random access compression algorithm and then compressed by using a non-random access compression algorithm, so as to determine the compressed first data set. In the embodiment of the present invention, the sizes of different fragments may also be different.
The present invention further provides a data processing method, configured to process the first data set stored in the disk and compressed twice, specifically as shown in fig. 5, where the method includes:
step S500, acquiring a first data set from a disk to a main memory.
Specifically, a first data set subjected to twice compression is acquired from a disk and is stored in a twice-compressed data buffer of a main memory.
For example, the first data set is determined by compressing the first original data set by a random access compression algorithm and then compressing the first original data set by a non-random access compression algorithm.
Optionally, in the embodiment of the present invention, the magnetic disk may also be a hard disk or other storage media, which is not limited in the present invention.
Step S501, the CPU is controlled to decompress the first data set for the first time according to a first decompression algorithm, and a second data set after the first decompression is determined, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access.
For example, the second data set after the first decompression is a data set compressed only by a random access compression algorithm.
Optionally, after step S501, the method further includes writing the second data set into a primary compressed data buffer of a main memory.
Step S502, the second data set is obtained from the main memory to the video memory of the image processor GPU.
Specifically, the second data set is obtained from the primary compressed data buffer of the main memory to the video memory data buffer of the video memory of the image processor GPU.
Step S503, controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, where the second decompression algorithm is a random access decompression algorithm.
Specifically, the multiple cores of the GPU are controlled to decompress the second data set for the second time in a parallel manner.
In the embodiment of the invention, a first data set is firstly obtained from a disk to a main memory, then a Central Processing Unit (CPU) is controlled to decompress the first data set for the first time according to a first decompression algorithm, a second data set after the first decompression is determined, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access, then the second data set is obtained from the main memory to a video memory of an image processor (GPU), and finally at least one processing core of the GPU is controlled to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm. By the method, the primary data set is compressed for the second time, transmission overhead from a disk to a main memory can be reduced, storage space between the disk and a CPU main memory is saved, the second data set subjected to the first decompression is decompressed and processed in parallel in a video memory of the GPU, decompression efficiency and data processing speed can be improved, the second data set does not need to be decompressed integrally and then processed, and storage space in the video memory of the GPU is saved.
In the embodiment of the present invention, the processing procedure of fig. 5 may also be represented by the schematic diagram shown in fig. 6, so that the processing procedure of fig. 5 may be more clearly expressed.
Optionally, after step S503, the method includes: and removing the second data set in the video memory data buffer of the video memory of the GPU.
In the embodiment of the present invention, step S503 specifically includes two cases.
In the first case, a plurality of cores of the GPU are controlled to decompress the second data set for the second time in a parallel mode, a decompressed first original data set is determined, and the first original data set is written into a video memory of the GPU; and controlling the plurality of processing cores to acquire data from the first original data set in the video memory of the GPU for processing.
Controlling a plurality of processing cores of the GPU to decompress the second data set for the second time, and writing the decompressed data into registers corresponding to the plurality of processing cores respectively; and controlling the plurality of processing cores to respectively acquire decompressed data from the corresponding registers for processing.
Fig. 7 is a schematic diagram of an apparatus for data compression according to an embodiment of the present invention. As shown in fig. 7, the data compression apparatus of the present embodiment includes: the device comprises a slicing unit 71, a first compression unit 72, a second compression unit 73 and a sending unit 74, wherein the slicing unit is used for controlling a CPU (central processing unit) to slice a second original data set by setting a threshold value and determining at least one sliced first original data set; the first compression unit is used for controlling the CPU or the GPU to perform first compression on the first original data set and determining a data set after the first compression; the second compression unit is further configured to control the CPU to perform second compression on the data set subjected to the first compression, and determine a compressed first data set; and the sending unit is used for sending the compressed first data set to a disk.
Fig. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 8, the data processing apparatus of the present embodiment includes: a first acquisition unit 81, a first decompression unit 82, a second acquisition unit 83, and a second decompression unit 84. The first acquiring unit is used for acquiring a first data set from a disk to a main memory; the first decompression unit is used for controlling a Central Processing Unit (CPU) to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access; the second obtaining unit is used for obtaining the second data set from the main memory to a video memory of the image processor GPU; the second decompression unit is configured to control at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, where the second decompression algorithm is a random access decompression algorithm.
Optionally, the apparatus further comprises: and the writing unit is used for writing the second data set into a primary compressed data buffer area of the main memory.
Optionally, the second obtaining unit is specifically configured to: and acquiring the second data set from the primary compressed data buffer of the main memory to a video memory data buffer of a video memory of the image processor GPU.
Optionally, the apparatus further comprises: a removal unit, configured to remove the second data set in the video memory data buffer of the video memory of the GPU.
Optionally, the second decompression unit is specifically configured to: and controlling a plurality of cores of the GPU to decompress the second data set for the second time in a parallel mode, determining a decompressed first original data set, and writing the first original data set into a video memory of the GPU.
Optionally, the second decompression unit is specifically configured to: and controlling the plurality of processing cores to acquire data from the first original data set in the video memory of the GPU for processing.
Optionally, the second decompression unit is specifically configured to: and controlling the plurality of processing cores of the GPU to decompress the second data set for the second time, and writing the decompressed data into the registers corresponding to the plurality of processing cores respectively.
Optionally, the second decompression unit is specifically configured to: and controlling the plurality of processing cores to respectively acquire decompressed data from the corresponding registers for processing.
In this embodiment of the present invention, the first obtaining unit may also be referred to as a disk memory transmission module, the first decompressing unit may also be referred to as a CPU decompressing module, the second obtaining unit may also be referred to as a memory video memory transmission module, the second decompressing unit may also be referred to as a GPU decompressing module, assuming that the second raw data set is divided into slice 0, slice 1, slice 2, and slice 3,4 first raw data sets, for different processing modules, after processing one of the slices, the next slice may be processed continuously, the multiple modules may process multiple slices at the same time, assuming that the processing times are T0, T1, T2, and T3, and the specific processing flow is as shown in fig. 9, wherein the disk memory module processes slice 0 first at time T0, then the CPU decompressing module processes slice 0 at time T1, then the memory video memory transmission module processes slice 0 at time T2, and finally the GPU decompressing module processes slice 0 at time T3; the CPU decompression module processes the fragment 0 at the time of T1, and the disk memory transmission module processes the fragment 1 at the same time; when the memory video memory transmission module processes the fragment 0 at the time of T2, the CPU decompression module processes the fragment 1, and the disk memory module processes the fragment 2; and at the time of T3, the GPU decompresses the partition 0, simultaneously, the memory video memory transmission module processes the partition 1, the CPU decompresses the partition 2, and the disk memory module processes the partition 3.
Specifically, the disk memory transmission module is used for reading the twice-compressed segmented data set from the disk into twice-compressed data buffer areas of the memory, before reading data from the disk, the disk memory transmission module firstly checks whether the twice-compressed data buffer areas in the memory have spare storage spaces, and if so, the segmented data set which is not accessed yet is transmitted from the disk to the twice-compressed data buffer areas; the CPU decompression module is used for carrying out first decompression on the twice-compressed fragment data set to obtain a data set which is only compressed by random access, and before carrying out the first decompression, the CPU decompression module sequentially checks twice-compressed data buffer areas of an internal memory and once-compressed data buffer areas of the internal memory, if the twice-compressed fragment data set which is not processed still exists in the twice-compressed data buffer areas and the once-compressed data buffer areas use vacant space, part of data is read from the twice-compressed data buffer areas, the CPU is used for carrying out the first decompression, and the decompressed data is stored in the once-compressed data buffer areas; the memory video memory transmission module is used for transmitting the partitioned data set which is only compressed by random access from the primary compressed data buffer area to the video memory data buffer area, the memory video memory transmission module sequentially checks the primary compressed data buffer area and the video memory data buffer area, if the unprocessed partitioned data set which is only compressed by random access exists in the primary compressed data buffer area and the video memory data buffer area uses free space, part of data is read from the primary compressed data buffer area and is transmitted and stored to the video memory data buffer area. The GPU decompression module is used for decompressing the partitioned data sets which are only compressed by random access in the video memory data buffer area and performing subsequent processing, if the video memory data buffer area has data, only one processing core of the GPU decompresses and processes the data, and after the data processing is completed, the corresponding partitioned data sets are removed from the video memory data buffer area.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, various aspects of embodiments of the invention may take the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," module "or" system. Furthermore, various aspects of embodiments of the invention may take the form of: a computer program product embodied in one or more computer readable media having computer readable program code embodied thereon.
Any combination of one or more computer-readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of embodiments of the invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to: electromagnetic, optical, or any suitable combination thereof. The computer readable signal medium may be any of the following computer readable media: is not a computer readable storage medium and may communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of embodiments of the present invention may be written in any combination of one or more programming languages, including: object oriented programming languages such as Java, smalltalk, C + +, and the like; and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package; executing in part on a user computer and in part on a remote computer; or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention described above describe aspects of embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method of data processing, comprising:
the CPU is controlled to segment the second original data set according to a set value, and at least one segmented first original data set is determined;
controlling the CPU or the GPU to carry out first compression on the first original data set, and determining a data set after the first compression;
controlling the CPU to carry out second compression on the data set after the first compression, and determining a first data set after compression;
sending the compressed first data set to a disk;
acquiring a first data set from a disk to a main memory;
controlling a Central Processing Unit (CPU) to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access;
acquiring the second data set from the main memory to a video memory of an image processor GPU;
and controlling at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm, wherein the second decompression algorithm is a random access decompression algorithm.
2. The method as claimed in claim 1, wherein said controlling said CPU to decompress said first data set for a first time according to a first decompression algorithm, and after determining said second data set after said first decompression, the method further comprises:
and writing the second data set into a primary compressed data buffer of the main memory.
3. The method according to claim 2, wherein said obtaining the second data set from the main memory to the video memory of the image processor GPU specifically comprises:
and acquiring the second data set from the primary compressed data buffer of the main memory to a video memory data buffer of a video memory of the image processor GPU.
4. The method of claim 3, wherein after controlling the at least one processing core of the GPU to decompress the second set of data a second time according to a second decompression algorithm, the method comprises:
and removing the second data set in the video memory data buffer of the video memory of the GPU.
5. The method of claim 1, wherein the controlling the at least one processing core of the GPU to decompress the second data set a second time according to a second decompression algorithm comprises:
and controlling a plurality of cores of the GPU to decompress the second data set for the second time in a parallel mode, determining a decompressed first original data set, and writing the first original data set into a video memory of the GPU.
6. The method according to claim 5, wherein the controlling the plurality of cores of the GPU to decompress the second data set for a second time in a parallel manner, determine a decompressed first raw data set, and write the first raw data set into a video memory of the GPU specifically includes:
and controlling the plurality of processing cores to acquire data from the first original data set in the video memory of the GPU for processing.
7. The method according to claim 1, wherein the controlling the at least one processing core of the GPU to decompress the second data set for the second time according to a second decompression algorithm comprises:
and controlling the plurality of processing cores of the GPU to decompress the second data set for the second time, and writing the decompressed data into the registers corresponding to the plurality of processing cores respectively.
8. The method of claim 7, wherein the controlling the plurality of processing cores of the GPU to decompress the second set of data a second time and to write the decompressed data into respective registers of the plurality of processing cores, further comprises:
and controlling the plurality of processing cores to respectively acquire decompressed data from the corresponding registers for processing.
9. The method of claim 1, wherein the first compression is performed using a random access compression algorithm and the second compression is performed using a non-random access compression algorithm.
10. An apparatus for data processing, comprising:
the fragmentation unit is used for controlling the CPU to fragment the second original data set according to a set value and determining at least one fragmented first original data set;
the first compression unit is used for controlling the CPU or the GPU to carry out first compression on the first original data set and determining a data set after the first compression;
the second compression unit is used for controlling the CPU to carry out second compression on the data set after the first compression and determining a first data set after the compression;
a sending unit, configured to send the compressed first data set to a disk;
the first acquisition unit is used for acquiring a first data set from a disk to a main memory;
the first decompression unit is used for controlling the CPU to decompress the first data set for the first time according to a first decompression algorithm and determining a second data set after the first decompression, wherein the first decompression algorithm is a decompression algorithm corresponding to a compression method which does not support random access;
the second acquisition unit is used for acquiring the second data set from the main memory to the video memory of the image processor GPU;
a second decompression unit, configured to control at least one processing core of the GPU to decompress the second data set for a second time according to a second decompression algorithm, where the second decompression algorithm is a random access decompression algorithm.
11. A computer-readable storage medium on which computer program instructions are stored, which, when invoked by a processor, implement the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910074154.8A CN109886862B (en) | 2019-01-25 | 2019-01-25 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910074154.8A CN109886862B (en) | 2019-01-25 | 2019-01-25 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109886862A CN109886862A (en) | 2019-06-14 |
| CN109886862B true CN109886862B (en) | 2023-01-17 |
Family
ID=66926964
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910074154.8A Active CN109886862B (en) | 2019-01-25 | 2019-01-25 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109886862B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678200A (en) * | 2012-08-31 | 2014-03-26 | 三星电子株式会社 | Data processing apparatus, method for processing data, and computer readable recording medium |
| CN105120293A (en) * | 2015-08-26 | 2015-12-02 | 中国航空工业集团公司洛阳电光设备研究所 | Image cooperative decoding method and apparatus based on CPU and GPU |
| CN108206937A (en) * | 2016-12-20 | 2018-06-26 | 浙江宇视科技有限公司 | A kind of method and apparatus for promoting intellectual analysis performance |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9792350B2 (en) * | 2013-01-10 | 2017-10-17 | International Business Machines Corporation | Real-time classification of data into data compression domains |
-
2019
- 2019-01-25 CN CN201910074154.8A patent/CN109886862B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678200A (en) * | 2012-08-31 | 2014-03-26 | 三星电子株式会社 | Data processing apparatus, method for processing data, and computer readable recording medium |
| CN105120293A (en) * | 2015-08-26 | 2015-12-02 | 中国航空工业集团公司洛阳电光设备研究所 | Image cooperative decoding method and apparatus based on CPU and GPU |
| CN108206937A (en) * | 2016-12-20 | 2018-06-26 | 浙江宇视科技有限公司 | A kind of method and apparatus for promoting intellectual analysis performance |
Non-Patent Citations (1)
| Title |
|---|
| 基于GPU加速的binLBT压缩解压算法;于卓等;《北京航空航天大学学报》;20100315(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109886862A (en) | 2019-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20090284400A1 (en) | Method and System for Reducing Required Storage During Decompression of a Compressed File | |
| JP7742731B2 (en) | Parallel decompression of compressed data streams | |
| CN108632624B (en) | Image data processing method, device, terminal device and readable storage medium | |
| US9311721B1 (en) | Graphics processing unit-assisted lossless decompression | |
| US11316530B2 (en) | Adaptive compression for data services | |
| KR102546236B1 (en) | Low-latency consumption of encoded video bitstream | |
| US8300704B2 (en) | Picture processing via a shared decoded picture pool | |
| US10015274B2 (en) | Enhanced storage clients | |
| EP3198870B1 (en) | Processing parameters for operations on blocks while decoding images | |
| US11429317B2 (en) | Method, apparatus and computer program product for storing data | |
| US8452900B2 (en) | Dynamic compression of an I/O data block | |
| CN106227506A (en) | A kind of multi-channel parallel Compress softwares system and method in memory compression system | |
| US20210191620A1 (en) | Data compression system using base values and methods thereof | |
| US20110016233A1 (en) | System, method, and computer program product for inserting a gap in information sent from a drive to a host device | |
| US20170192897A1 (en) | Caching in storage clients | |
| CN105791819A (en) | Frame compression method for image and decompression method and device for image | |
| CN109886862B (en) | Data processing method and device | |
| CN103049561A (en) | Data compressing method, storage engine and storage system | |
| WO2021237513A1 (en) | Data compression storage system and method, processor, and computer storage medium | |
| US8823557B1 (en) | Random extraction from compressed data | |
| EP2897052B1 (en) | Address compression and decompression method, compressor and decompressor | |
| KR102383962B1 (en) | Deep learning accelerator with variable data encoder/decoder | |
| JP7481167B2 (en) | Compressed data decompression system, memory system, and data decompression method | |
| US10838727B2 (en) | Device and method for cache utilization aware data compression | |
| CN118132567B (en) | Multi-pipeline heterogeneous filtering acceleration method and system for column storage database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |