CN110908966B - Method, device and equipment for calculating deduplication rate and readable storage medium - Google Patents

Abstract

The invention discloses a method for calculating a deduplication rate, comprising: acquiring a file operation request, using an online deduplication method to execute the file operation request, generating corresponding log information, and adding the log information to the log; reading the log, and calculating a storage system The theoretical occupancy value of the storage system is obtained; the actual occupancy value of the storage system is obtained, and the deduplication rate is calculated by using the theoretical occupancy value and the actual occupancy value; this method uses the online deduplication method to execute the request after obtaining the execution file operation request, and generates the corresponding log information. When calculating the deduplication rate, use the log to calculate the theoretical occupancy value of the storage system, and use the theoretical occupancy value and the actual occupancy value to calculate the deduplication rate of the storage system. It is not necessary to stop receiving business operations, and the deduplication rate of the storage system at the current moment can be accurately calculated. , the accuracy of deduplication rate calculation is improved; in addition, the present invention also provides a deduplication rate calculation device, equipment and computer-readable storage medium, which also have the above beneficial effects.

Description

A kind of deduplication rate calculation method, device, device and readable storage medium

technical field

The present invention relates to the technical field of file deduplication, and in particular, to a deduplication rate calculation method, a deduplication rate calculation device, a deduplication rate calculation device and a computer-readable storage medium.

Background technique

Storage space is a limited resource. Different files uploaded by the same user, such as logs at different times, or files uploaded by different users, such as files uploaded by different people in the same company, will have a lot of the same content in these uploaded files. file or an object with the same content after segmentation. To save storage space, data deduplication can be performed on the storage cluster. After the deduplication function is enabled, it is very important to count the deduplication rate to measure and compare the efficiency and performance of different deduplication algorithms, algorithm optimization configuration parameters, and algorithm implementation details. The existing deduplication rate calculation method is changed to the offline deduplication method (that is, the background deduplication method), that is, after the file upload is completed, the capacity of the cluster is obtained, the reception of business operations is stopped, and the file deduplication function is activated. After the content of the duplicated files, obtain the capacity of the cluster storage after deduplication, and use the capacity before and after deduplication to calculate the deduplication rate. However, only the average deduplication rate in a period of time can be calculated in this way, and the calculated deduplication rate is not accurate enough.

Therefore, how to solve the problem that the deduplication rate calculated by the existing deduplication rate calculation method is not accurate enough is a technical problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

In view of this, the object of the present invention is to provide a deduplication rate calculation method, deduplication rate calculation device, deduplication rate calculation device and computer-readable storage medium, which solve the deduplication rate calculated by the existing deduplication rate calculation method. Inaccurate question.

In order to solve the above-mentioned technical problem, the present invention provides a kind of deduplication rate calculation method, including:

Obtaining a file operation request, using the online deduplication method to execute the file operation request and generating corresponding log information, and adding the log information to the log;

Read the log, and calculate the theoretical occupancy value of the storage system;

The actual occupancy value of the storage system is acquired, and the deduplication rate is calculated by using the theoretical occupancy value and the actual occupancy value.

Optionally, the reading of the log to calculate the theoretical occupancy value of the storage system includes:

Read each of the log information in the log, and determine the file volume in each of the log information and the state corresponding to the file volume; wherein, the state is a plus state or a minus state;

adding up all the file volumes with the plus state to obtain a first occupancy value, and adding up all the file volumes with the minus state to obtain a second occupancy value;

The theoretical occupancy value is obtained by subtracting the second occupancy value from the first occupancy value.

Optionally, when the file operation request is a file upload operation request, the obtaining the file operation request, using the online deduplication method to execute the file operation request and generating corresponding log information, including:

Obtaining a file upload operation request and a first file, and striping and slicing the first file to obtain a plurality of first objects;

Calculate the first fingerprint information of each of the first objects, and use each of the first fingerprint information to sequentially match with the fingerprint information database;

When the matching is successful, add one to the reference count of the first target object corresponding to the first fingerprint information; wherein, the first target object is stored in the storage system;

When the matching is unsuccessful, the first object corresponding to the first fingerprint information is stored in the storage system;

A first file volume of the first file is acquired, upload log information is generated by using the first file volume, and the upload log information is determined as the log information.

Optionally, when the file operation request is a file update operation request, obtaining the file operation request, using an online deduplication method to execute the file operation request and generating corresponding log information, including:

Obtain the file update operation request and the second file, and determine the update file specified by the file update operation request;

Obtain the update file volume of the update file, and utilize the update file volume to generate deletion log information;

Store the second file in the storage system, obtain the second file volume of the second file, and use the second file volume to generate first log information; wherein the first log information is upload log information ;

The log information is constructed using the first log information and the deletion log information.

The present invention also provides a deduplication rate calculation device, comprising:

A log update module, configured to obtain a file operation request, execute the file operation request by using an online deduplication method, generate corresponding log information, and add the log information to the log;

a first calculation module, configured to read the log and calculate the theoretical occupancy value of the storage system;

The second calculation module is configured to obtain the actual occupancy value of the storage system, and calculate the deduplication rate by using the theoretical occupancy value and the actual occupancy value.

Optionally, the first computing module includes:

a determining unit, configured to read each of the log information in the log, and determine a file volume in each of the log information and a state corresponding to the file volume; wherein, the state is a plus state or a minus state;

a first calculation unit, configured to add up all the file volumes with the plus state to obtain a first occupancy value, and add all the file volumes with the minus state to obtain a second occupancy value;

The second calculation unit is configured to subtract the second occupancy value from the first occupancy value to obtain the theoretical occupancy value.

Optionally, the log update module includes:

a first object obtaining unit, configured to obtain a file upload operation request and a first file, and perform striping and slicing on the first file to obtain a plurality of first objects;

a matching unit, configured to calculate the first fingerprint information of each of the first objects, and use each of the first fingerprint information to sequentially match with the fingerprint information database;

a count modification unit, configured to add one to the reference count of the first target object corresponding to the first fingerprint information when the matching is successful; wherein, the first target object is stored in the storage system;

a storage unit, configured to store the first object corresponding to the first fingerprint information in the storage system when the matching is unsuccessful;

A log generating unit, configured to acquire a first file volume of the first file, generate upload log information by using the first file volume, and determine the upload log information as the log information.

Optionally, the log update module includes:

a determining unit, configured to obtain the file update operation request and the second file, and determine the update file specified by the file update operation request;

a first generating unit, configured to obtain the update file volume of the update file, and use the update file volume to generate deletion log information;

a second generating unit, configured to store the second file in the storage system, obtain a second file volume of the second file, and generate first log information by using the second file volume; 1. The log information is the upload log information;

A generating unit configured to use the first log information and the deletion log information to form the log information.

The present invention also provides a deduplication rate computing device, including a memory and a processor, wherein:

the memory for storing computer programs;

The processor is configured to execute the computer program to implement the above-mentioned method for calculating the deduplication rate.

The present invention also provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned deduplication rate calculation method is implemented.

The deduplication rate calculation method provided by the present invention obtains a file operation request, uses an online deduplication method to execute the file operation request, generates corresponding log information, and adds the log information to the log. Read the log and calculate the theoretical occupancy value of the storage system. Obtain the actual occupancy value of the storage system, and use the theoretical occupancy value and the actual occupancy value to calculate the deduplication rate.

It can be seen that this method uses the online deduplication method to execute the request after obtaining the execution file operation request, generates the corresponding log information and adds the log information to the log, and uses the log to calculate the theoretical occupancy value of the storage system when calculating the deduplication rate. Detect the actual occupancy value of the storage system at this time, and use the theoretical occupancy value and the actual occupancy value to calculate the deduplication rate of the storage system. The method does not need to stop receiving business operations, can accurately calculate the deduplication rate of the storage system at the current moment, improves the accuracy of deduplication rate calculation, and solves the problem that the deduplication rate calculated by the existing deduplication rate calculation method is not accurate enough. It is of great significance to the optimization of the configuration parameters of the deduplication algorithm and the improvement of the efficiency and performance during the implementation of the algorithm.

In addition, the present invention also provides a deduplication rate calculation device, a deduplication rate calculation device, and a computer-readable storage medium, which also have the above beneficial effects.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a flowchart of a method for calculating a deduplication rate provided by an embodiment of the present invention;

Fig. 2 is a kind of theoretical value calculation flow chart provided by the embodiment of the present invention;

FIG. 3 is a flowchart for processing a file operation request provided by an embodiment of the present invention;

4 is a flowchart of another file operation request processing provided by an embodiment of the present invention;

5 is a schematic structural diagram of an apparatus for calculating a deduplication rate provided by an embodiment of the present invention;

FIG. 6 is a schematic structural diagram of a deduplication rate calculation device according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Please refer to FIG. 1 , which is a flowchart of a method for calculating a deduplication rate according to an embodiment of the present invention. The method includes:

S101: Obtain a file operation request, execute the file operation request by using an online deduplication method, generate corresponding log information, and add the log information to the log.

The file operation request is used to operate a file or object in the storage system, and the specific operation may include downloading, deleting, copying, updating, etc., and may also be used to add a new file or new object to the storage system. A file operation request may include only one request, or may include multiple requests; when multiple requests are included, the types and targeted files or objects of each request may be the same or different. The file operation request can include information such as bucket name and file name, which is used to locate the operated file or object.

Online deduplication, also known as real-time deduplication, means that when a new file is added to the storage system, the fingerprint information of the file or object is used to determine whether the file or object has been stored, and if so, the file or object is not stored. , only add the corresponding information; if not, store the file or object in the storage system. specific. This deduplication method can reduce the placement of duplicate data, and the deduplication of duplicate data is real-time. After the online deduplication method is used to execute the file operation request, the corresponding log information needs to be generated. Since the duplicate data is deleted in real time during online deduplication, it is impossible to know the occupied value of the storage system without deduplication. Therefore, corresponding log information needs to be generated. The log information records the size of the file or object corresponding to this operation. Theoretical changes to the storage system after recording the file operation request. Add log information to the storage system's log after it is generated.

S102: Read the log, and calculate the theoretical occupancy value of the storage system.

When you need to calculate the deduplication rate, you can read the log and calculate the theoretical occupancy value of the storage system. It should be noted that the embodiment of the present invention does not limit the timing of reading the log. For example, the operation of reading the log can be performed when the instruction for calculating the deduplication rate is detected. The instruction for calculating the deduplication rate can be manually input by the operator, or When a specific command is detected, an instruction to calculate the deduplication rate is automatically generated; or the deduplication rate can be calculated according to a preset period, that is, the operation of reading the log is performed according to the preset period; or the deduplication rate can be calculated in real time, that is, when the The next operation of reading the log is performed immediately after the deletion rate calculation is completed. When reading logs, you can calculate the theoretical occupancy value of the storage system, that is, the theoretical space occupancy value of the storage system when data deduplication is not performed.

S103: Obtain the actual occupancy value of the storage system, and calculate the deduplication rate by using the theoretical occupancy value and the actual occupancy value.

The actual occupancy value is the space occupancy value of the storage system after the data deduplication operation is performed. After obtaining the theoretical occupancy value and the actual occupancy value, the deduplication rate can be obtained by dividing the theoretical occupancy value and the actual occupancy value. Specifically, the deduplication rate can be expressed in the form of X:1.

Applying the deduplication rate calculation method provided by the embodiment of the present invention, after obtaining the execution file operation request, the online deduplication method is used to execute the request, the corresponding log information is generated and the log information is added to the log, and the log is used when calculating the deduplication rate. Calculate the theoretical occupancy value of the storage system, detect the actual occupancy value of the storage system at this time, and use the theoretical occupancy value and the actual occupancy value to calculate the deduplication rate of the storage system. The method does not need to stop receiving business operations, and can accurately calculate the deduplication rate of the storage system at the current moment, which improves the accuracy of deduplication rate calculation, and solves the problem that the deduplication rate calculated by the existing deduplication rate calculation method is not accurate enough. , which is of great significance to the optimization of the configuration parameters of the deduplication algorithm and the improvement of the efficiency and performance during the implementation of the algorithm.

Based on the above embodiments of the present invention, the embodiments of the present invention will describe a specific theoretical occupancy value calculation process. Please refer to FIG. 2. FIG. 2 is a flow chart of theoretical value calculation provided by an embodiment of the present invention, including:

S201: Read each log information in the log, and determine the file volume in each log information and a state corresponding to the file volume; wherein, the state is a plus state or a minus state.

In this embodiment of the present invention, the log information includes file volumes and states corresponding to file volumes, and each log information may include one file volume and one corresponding state, and may also include multiple file volumes and multiple corresponding states. The state is a plus state or a minus state, the plus state is used to indicate an increase in the theoretical occupancy value, and the minus state is used to indicate a decrease in the theoretical occupancy value. Different file operation requests have different statuses in the log information generated after they are executed. The status corresponding to each file operation request can be the same or different. For example, when the file operation request is a file upload operation request, the generated log information The status is plus status; or when the file operation request is a file deletion request, the status in the generated log information is minus status. Therefore, when calculating the theoretical occupancy value, each log information in the log needs to be read and the file volume in each log information and the state corresponding to the file volume need to be determined.

S202: Add up the volumes of all files in the plus state to obtain a first occupancy value, and add up all the file volumes in the minus state to obtain a second occupancy value.

After the file volume and the corresponding state are determined, the first occupancy value can be obtained by adding up all the file volumes with the plus state. Specifically, when a file volume with an added state is detected, it can be added to the previously read file volume with an added state, until the volumes of all files with an added state are added to obtain the first occupancy value; or When the file volume with the plus state is read, it is recorded, and after all the file volumes with the plus state are recorded, all the file volumes with the plus state are added to obtain the first occupancy value. The second occupancy value is obtained by adding up the volumes of all files with a reduced state. The specific calculation process of the second occupancy value may be the same as the first occupancy value calculation process, or may be different from the first occupancy value calculation process. This embodiment This is not limited.

S203: Subtract the second occupancy value from the first occupancy value to obtain a theoretical occupancy value.

After obtaining the first occupancy value and the second occupancy value, subtract the second occupancy value from the first occupancy value to obtain the theoretical occupancy value of the storage system, that is, the space occupancy value when the storage system does not perform the data deduplication operation.

Based on the above embodiments of the present invention, the file operation request may be a file upload operation request. The embodiment of the present invention will describe the processing process of the file upload operation request. Please refer to FIG. 3 , which is a file operation request processing provided by an embodiment of the present invention. Flowchart, including:

S301: Obtain a file upload operation request and a first file, and perform striping and slicing on the first file to obtain a plurality of first objects.

Since the file operation request is a file upload operation request, after the file upload operation request is obtained, the corresponding first file, that is, the uploaded file should also be obtained. After the first file is acquired, striping and slicing the first file is performed to obtain a plurality of first objects. This embodiment does not limit the specific rules and procedures for striping and slicing. Reference may be made to related technologies. Repeat. It should be noted that the first object may also be one. For example, when the size of the first file is smaller than the slicing threshold and cannot be sliced, the first file may be directly used as the first object.

S302: Calculate the first fingerprint information of each first object, and use each first fingerprint information to sequentially match with the fingerprint information database.

The fingerprint information is used to represent the identity of the file or object, and can be calculated by using a hash algorithm, that is, a hash algorithm, for example, calculated by using the MD5 algorithm; or calculated by using the SHA1 algorithm. When the first fingerprint information corresponding to each first object is obtained, each first fingerprint information is matched with the fingerprint information database corresponding to the storage system once. It is judged whether the first fingerprint information is stored in the fingerprint information database.

S303: When the matching is successful, add one to the reference count of the first target object corresponding to the first fingerprint information; wherein, the first target object is stored in the storage system.

When the matching is successful, it means that the first object corresponding to the first fingerprint information has been stored in the storage system, and the object corresponding to the first object in the storage system is the first target object. Therefore, when the matching is successful, the reference count of the first target object corresponding to the first fingerprint information is incremented by one.

S304: When the matching is unsuccessful, store the first object corresponding to the first fingerprint information in the storage system.

When the matching is unsuccessful, it means that the first object corresponding to the first fingerprint information is not stored in the storage system, so the first object corresponding to the laundry fingerprint information is stored in the storage system. After the first object is stored in the storage system, other operations may also be performed, for example, setting the reference count of the first object to 1, and storing the corresponding first fingerprint information in the fingerprint information database.

S305: Obtain the first file volume of the first file, generate upload log information by using the first file volume, and determine the upload log information as log information.

After all the first objects are processed, a first file volume of the first file is obtained, upload log information is generated by using the first file volume, and the upload log information is determined as log information. Specifically, the upload log information may include the state corresponding to the first file volume, that is, the plus state, which can be used to calculate the theoretical occupancy value; or the first file volume can be marked as a positive number, which can be used for subsequent calculation of the theoretical occupancy value.

Based on the above embodiments of the present invention, the file operation request may be a file update operation request. The embodiment of the present invention will describe the processing process of the file update operation request. Please refer to FIG. 4 . FIG. 4 is another file operation request provided by the embodiment of the present invention. Process flow chart, including:

S401: Obtain the file update operation request and the second file, and determine the update file specified by the file update operation request.

Since the file update operation request needs to update the update file in the storage system, after the file update operation request is obtained, the corresponding second file needs to be obtained. The update file information should be recorded in the file update operation request, for example, the bucket name and file name corresponding to the update file, or other information. With a file update operation request, the update file it specifies can be determined.

S402: Obtain the update file volume of the update file, and use the update file volume to generate deletion log information.

After obtaining the update file volume, use the update file volume to generate deletion log information. Specifically, the deletion log information may include the state corresponding to the update file volume, that is, the reduction state, which can be used to calculate the theoretical occupancy value; or the update file volume can be marked If it is a negative number, it can be used for subsequent calculation of the theoretical occupancy value.

S403: Store the second file in the storage system, obtain a second file volume of the second file, and generate first log information by using the second file volume.

It should be noted that the first log information is the upload log information. For the specific upload log information construction method, reference may be made to step S304, which will not be repeated here. It should be noted that the state or flag of the first log information and the deleted log information are different.

S404: Use the first log information and the deleted log information to form log information.

The log information corresponding to the file update operation request is formed by using the first log information and the deletion log information, and after the log information is formed, the log information is added to the log. When calculating the theoretical occupancy value, the log information can be analyzed to obtain first log information and deleted log information, and the theoretical occupancy value can be calculated by using the first log information and the deleted log information. Delete the log information to calculate the file volume of the entire log information and the status or mark corresponding to the file volume, and then use the file volume and status or mark to calculate the theoretical occupancy value; or the log information can be parsed into two log information, that is, the first log information and deletion log information, respectively using the second file volume and the corresponding state or flag and the updated file volume and the corresponding state or flag to calculate the theoretical occupancy value.

The following describes the deduplication rate calculation apparatus provided by the embodiment of the present invention. The deduplication rate calculation apparatus described below and the deduplication rate calculation method described above can be referred to each other correspondingly.

Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of an apparatus for calculating a deduplication rate according to an embodiment of the present invention, including:

The log update module 510 is used to obtain the file operation request, execute the file operation request by using the online deduplication method, generate corresponding log information, and add the log information to the log;

The first calculation module 520 is used to read the log and calculate the theoretical occupancy value of the storage system;

The second calculation module 530 is configured to obtain the actual occupancy value of the storage system, and use the theoretical occupancy value and the actual occupancy value to calculate the deduplication rate.

Optionally, the first computing module 520 includes:

A determination unit, used for reading each log information in the log, and determining the file volume in each log information and the state corresponding to the file volume; wherein, the state is a plus state or a minus state;

The first calculation unit is used to add up all the file volumes with the plus state to obtain the first occupancy value, and add all the file volumes with the minus state to obtain the second occupancy value;

The second calculation unit is configured to subtract the second occupancy value from the first occupancy value to obtain the theoretical occupancy value.

Optionally, the log update module 510 includes:

a first object obtaining unit, configured to obtain a file upload operation request and a first file, and perform striping and slicing on the first file to obtain a plurality of first objects;

a matching unit, configured to calculate the first fingerprint information of each first object, and use each first fingerprint information to match with the fingerprint information database in turn;

a count modification unit, used for adding one to the reference count of the first target object corresponding to the first fingerprint information when the matching is successful; wherein, the first target object is stored in the storage system;

a storage unit, configured to store the first object corresponding to the first fingerprint information in the storage system when the matching is unsuccessful;

The log generating unit is configured to obtain the first file volume of the first file, generate upload log information by using the first file volume, and determine the upload log information as log information.

Optionally, the log update module 510 includes:

a determining unit, used to obtain the file update operation request and the second file, and determine the update file specified by the file update operation request;

The first generation unit is used to obtain the update file volume of the update file, and utilize the update file volume to generate deletion log information;

The second generating unit is configured to store the second file in the storage system, obtain the second file volume of the second file, and use the second file volume to generate first log information; wherein, the first log information is upload log information;

The generating unit is configured to use the first log information and the deletion log information to form log information.

The following describes the deduplication rate calculation device provided by the embodiment of the present invention, and the deduplication rate calculation device described below and the deduplication rate calculation method described above may refer to each other correspondingly.

Please refer to FIG. 6. FIG. 6 is a schematic structural diagram of a deduplication rate calculation device according to an embodiment of the present invention. The deduplication rate calculation device includes a memory and a processor, wherein:

a memory 610 for storing computer programs;

The processor 620 is configured to execute a computer program to implement the above-mentioned method for calculating the deduplication rate.

The computer-readable storage medium provided by the embodiment of the present invention is introduced below, and the computer-readable storage medium described below and the deduplication rate calculation method described above can be referred to each other correspondingly.

The present invention also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned deduplication rate calculation method are implemented.

The computer-readable storage medium may include: a U disk, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc. that can store program codes. medium.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

Finally, it should also be noted that, in this context, relationships such as first and second, etc., are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation are intended to cover a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also not expressly listed Other elements, or elements that are inherent to such a process, method, article or apparatus.

The above provides a detailed introduction to a deduplication rate calculation method, a deduplication rate calculation device, a deduplication rate calculation device, and a computer-readable storage medium provided by the present invention. Specific examples are used in this paper to describe the principles and implementations of the present invention. Elaborated, the description of the above embodiment is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the idea of the present invention, there will be a For changes, in summary, the contents of this specification should not be construed as limiting the present invention.

Claims (8)

1. A file deduplication rate calculation method is characterized by comprising the following steps:

acquiring a file operation request, executing the file operation request by using an online deduplication method, generating corresponding log information, and adding the log information into a log;

reading the log, and calculating a theoretical occupation value of the storage system;

acquiring an actual occupancy value of the storage system, and calculating a deduplication rate by using the theoretical occupancy value and the actual occupancy value;

the reading the log and calculating the theoretical occupancy value of the storage system comprises the following steps:

reading each log information in the log, and determining a file volume in each log information and a state corresponding to the file volume; wherein the state is an addition state or a subtraction state; the adding state is used for indicating that the theoretical occupancy value is increased, and the subtracting state is used for indicating that the theoretical occupancy value is decreased;

adding all the file volumes with the adding state to obtain a first occupation value, and adding all the file volumes with the subtracting state to obtain a second occupation value;

and subtracting the second occupancy value from the first occupancy value to obtain the theoretical occupancy value.

2. The method according to claim 1, wherein when the file operation request is a file upload operation request, the obtaining the file operation request, executing the file operation request by using an online deduplication method, and generating corresponding log information includes:

Acquiring a file uploading operation request and a first file, and carrying out striping slicing on the first file to obtain a plurality of first objects;

calculating first fingerprint information of each first object, and sequentially matching the first fingerprint information with a fingerprint information base;

when the matching is successful, the reference count of the first target object corresponding to the first fingerprint information is increased by one; wherein a first target object is stored in the storage system;

when the matching is unsuccessful, storing a first object corresponding to the first fingerprint information into the storage system;

acquiring a first file volume of the first file, generating uploading log information by using the first file volume, and determining the uploading log information as the log information.

3. The method according to claim 1, wherein when the file operation request is a file update operation request, the obtaining the file operation request, executing the file operation request by using an online deduplication method, and generating corresponding log information includes:

acquiring a file updating operation request and a second file, and determining an updating file specified by the file updating operation request;

Acquiring the volume of the update file, and generating deletion log information by using the volume of the update file;

storing the second file into the storage system, acquiring a second file volume of the second file, and generating first log information by using the second file volume; wherein the first log information is upload log information;

the first log information and the deletion log information are used to form the log information.

4. A file deduplication rate calculation apparatus, comprising:

the log updating module is used for acquiring a file operation request, executing the file operation request by using an online deduplication method, generating corresponding log information and adding the log information into a log;

the first calculation module is used for reading the log and calculating a theoretical occupation value of the storage system;

the second calculation module is used for acquiring the actual occupation value of the storage system and calculating the deduplication rate by using the theoretical occupation value and the actual occupation value;

the first computing module, comprising:

the determining unit is used for reading each log information in the log and determining the file volume in each log information and the state corresponding to the file volume; wherein the state is an addition state or a subtraction state; the adding state is used for indicating that the theoretical occupancy value is increased, and the subtracting state is used for indicating that the theoretical occupancy value is decreased;

The first calculation unit is used for adding all the file volumes with the adding states to obtain a first occupation value, and adding all the file volumes with the subtracting states to obtain a second occupation value;

and the second calculation unit is used for subtracting the second occupation value from the first occupation value to obtain the theoretical occupation value.

5. The apparatus of claim 4, wherein the log update module comprises:

the device comprises a first object acquisition unit, a second object acquisition unit and a third object acquisition unit, wherein the first object acquisition unit is used for acquiring a file uploading operation request and a first file, and striping and slicing the first file to obtain a plurality of first objects;

the matching unit is used for calculating first fingerprint information of each first object and sequentially matching the first fingerprint information with a fingerprint information base;

the counting modification unit is used for adding one to the reference count of the first target object corresponding to the first fingerprint information when the matching is successful; wherein a first target object is stored in the storage system;

the storage unit is used for storing the first object corresponding to the first fingerprint information into the storage system when the matching is unsuccessful;

And the log generating unit is used for acquiring a first file volume of the first file, generating uploading log information by using the first file volume, and determining the uploading log information as the log information.

6. The apparatus of claim 4, wherein the log update module comprises:

the determining unit is used for acquiring a file updating operation request and a second file and determining an updating file specified by the file updating operation request;

the first generation unit is used for acquiring the volume of the update file and generating deletion log information by using the volume of the update file;

the second generation unit is used for storing the second file into the storage system, acquiring a second file volume of the second file, and generating first log information by using the second file volume; wherein the first log information is uploaded log information;

a generating unit configured to construct the log information using the first log information and the deletion log information.

7. A file deduplication rate computing apparatus comprising a memory and a processor, wherein:

the memory is used for storing a computer program;

The processor is configured to execute the computer program to implement the file deduplication rate calculation method according to any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the file deduplication rate calculation method according to any one of claims 1 through 3.

Images