CN112463021A

CN112463021A - Data processing method and device and chip

Info

Publication number: CN112463021A
Application number: CN201911209074.5A
Authority: CN
Inventors: 卜琪; 陈杰; 陈庆; 赵泓博
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2019-09-06
Filing date: 2019-11-30
Publication date: 2021-03-09

Abstract

The present application discloses a data processing method, device and chip, which belong to the technical field of data processing. The method includes: the processor sends a processing instruction to the accelerator, where the processing instruction includes: a segmentation instruction and first storage information; the accelerator obtains data to be processed based on the first storage information; The accelerator performs segmentation processing on the data to be processed based on the segmentation instruction to obtain multiple data blocks; the data processing device performs target processing on the multiple data blocks to obtain deduplicated data. The present application solves the problem of high CPU load, reduces the processor load, and is used for data processing.

Description

Data processing method and device and chip

The present application claims priority from chinese patent application No. 201910844519.0 entitled "data deduplication acceleration method, apparatus, and system" filed on 2019, 09/06, which is incorporated by reference in its entirety in the present disclosure.

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method, an apparatus, and a chip.

Background

With the development of technology, more and more data are stored in the devices and transmitted among the devices. In the related art, a data deduplication (data deduplication) method is generally adopted to reduce duplicate data in the device.

In the related art, the device generally includes a Central Processing Unit (CPU) and a memory, data is generally stored in the memory, and the CPU can read and write the data in the memory to implement deduplication processing on the data.

However, the load of the CPU in the related art is high.

Disclosure of Invention

The application provides a data processing method, a device and a chip, which can solve the problem of high load of a CPU, and the technical scheme is as follows:

in a first aspect, a data processing method is provided for a data processing apparatus comprising a processor and an accelerator, the method comprising: the processor sends a processing indication to the accelerator, the processing indication comprising: a segment indication, and first storage information; the accelerator acquires data to be processed based on the first storage information; the accelerator carries out segmentation processing on the data to be processed based on the segmentation indication to obtain a plurality of data blocks; and the data processing device performs target processing on the plurality of data blocks to obtain data subjected to deduplication processing.

In the data processing method provided by the application, the processor can send a processing instruction to the accelerator when the data to be processed needs to be subjected to deduplication processing. Also, the accelerator may perform at least some of the stages in the deduplication process, regardless of the implementation in which the process dictates. Therefore, the processing required to be executed by the processor is reduced, the load of the processor is reduced, and the data processing efficiency of the processor is improved.

Optionally, the data processing apparatus performs target processing on the plurality of data blocks to obtain data after deduplication processing, and includes: the processor calculating a corresponding signature for each of the plurality of data blocks; the processor queries a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is the data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed. The processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing. When the processor queries the dictionary based on each data block, the processor may query whether the dictionary has the data block recorded therein.

Optionally, before the processor calculates a corresponding signature for each of the plurality of data blocks, the method further comprises: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator finishes segmented processing on the data to be processed. Illustratively, the processing response may include second storage information for the plurality of data blocks. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the processing instructions further include: a signature indication; the data processing device performs target processing on the plurality of data blocks to obtain data after deduplication processing, and the data processing device comprises: the accelerator calculates a corresponding signature for each of the plurality of data chunks based on the signature indications; the processor queries a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is the data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed. The processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing. When the processor queries the dictionary based on each data block, the processor may query whether the dictionary has the data block recorded therein.

Optionally, before the processor queries a dictionary from the plurality of data blocks, the method further comprises: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator completes segmented processing of the data to be processed and completes calculation of a corresponding signature for each data block. For example, the processing response may include a plurality of data blocks and signed second storage information corresponding to the plurality of data blocks. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the accelerator comprises: the device comprises a controller, a receiving cache, a sending cache and a processing unit; the processor sends a processing indication to the accelerator, including: the processor sends the processing indication to the controller; the accelerator acquires data to be processed based on the first storage information, and the method comprises the following steps: the controller controls the receiving cache to acquire the data to be processed based on the first storage information; the accelerator performs segmentation processing on the data to be processed based on the segmentation indication, and the segmentation processing comprises the following steps: the controller controls the processing unit to acquire the data to be processed in the receiving cache and performs the segmentation processing on the data to be processed based on the segmentation indication; the method further comprises the following steps: the processing unit sends the data blocks and the signature corresponding to each data block to the sending cache; the processing unit sends a finish instruction to the controller; the controller controls the sending cache to store the plurality of data blocks and the signature corresponding to each data block based on the finishing indication; the processor receiving a processing response from the accelerator, comprising: the processor receives the processing response from the controller.

Optionally, the processing instructions further include: a signature indication and a query indication; the data processing device performs target processing on the plurality of data blocks to obtain data after deduplication processing, and the data processing device comprises: the accelerator calculates a corresponding signature for each of the plurality of data chunks based on the signature indications; the accelerator queries a dictionary based on the query indication and the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is as follows: and all data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed. The processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing. When the accelerator queries the dictionary based on each data block, the accelerator may query whether the dictionary has the data block recorded therein.

Optionally, before the processor replaces the first data block with the first signature, the method further comprises: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator finishes processing the data to be processed. The processing comprises the following steps: the segmentation process described above, as well as computing a corresponding signature for each data block based on the previous indication, and querying the dictionary based on the plurality of data blocks. For example, the processing response may include a plurality of data blocks, signatures corresponding to the plurality of data blocks, and second storage information of whether the result of each data block is recorded in the dictionary. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the processing instructions further include: a signature indication, a query indication, and an update indication; the data processing device performs target processing on the plurality of data blocks to obtain data after deduplication processing, and the data processing device comprises: the accelerator calculates a corresponding signature for each of the plurality of data chunks based on the signature indications; the accelerator queries a dictionary based on the query indication and the plurality of data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the accelerator replaces the first data block in the data to be processed with the first signature according to the updating indication; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; and the processor takes the data processed by the accelerator as the data after the deduplication processing, and the data processed by the accelerator are all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed. When the accelerator queries the dictionary based on each data block, the accelerator may query whether the dictionary has the data block recorded therein.

Optionally, before the processor uses all data blocks in the data to be processed that are not replaced by the signature and all replaced signatures as the data after the deduplication processing, the method further includes: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator finishes processing the data to be processed. The processing comprises the following steps: the segmentation processing includes calculating a corresponding signature for each data block based on the previous indication, querying a dictionary based on a plurality of data blocks, and replacing a first data block in the data to be processed with the first signature by the accelerator according to the update indication when the first data block in the plurality of data blocks and the first signature of the first data block are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the corresponding relation between the signature of the second data block and the second data block in the dictionary. For example, the processing response may include all data blocks in the data to be processed that are not replaced by the signature and all second stored information of the replaced signature. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the accelerator comprises: a plurality of processing channels, the accelerator performing a segmentation process on the data to be processed based on the segmentation indication, including: the accelerator determining any idle processing channel of the plurality of processing channels; and the accelerator performs the segmented processing on the data to be processed through any idle processing channel.

Optionally, the accelerator comprises: a plurality of processing channels, the accelerator performing a segmentation process on the data to be processed based on the segmentation indication, including: the accelerator divides the data to be processed into at least two continuous data segments; and the accelerator performs the segmentation processing on the at least two data segments in a one-to-one correspondence manner through at least two idle processing channels in the plurality of processing channels. In the multi-channel fast processing mode, the accelerator performs segmented processing on the same data to be processed through a plurality of processing channels, so that the speed of processing the data to be processed by the accelerator is high. When the data to be processed is the data to be transmitted, the data to be processed is processed quickly, so that the data to be processed is transmitted quickly, and the transmission delay of the data to be processed is reduced.

Optionally, the processing instructions further include: a first data indication indicating whether the to-be-processed data includes a first data in the data set to which the to-be-processed data belongs, wherein before the accelerator performs the segmentation processing on the to-be-processed data based on the segmentation indication, the method further includes: when the data to be processed meets a target condition, the accelerator updates the data to be processed; wherein the target conditions include: the first data indication is used for indicating that the data to be processed does not contain the first data, and auxiliary data in the data set is subjected to the segmentation processing before the segmentation processing is carried out on the data to be processed, and residual data segments except data blocks exist in the auxiliary data; the updated data to be processed comprises: the data to be processed before updating, a first remaining data segment in the auxiliary data, and all data located after the first remaining data segment in the auxiliary data.

Optionally, the accelerator comprises: a plurality of processing channels, the auxiliary data comprising n remaining data segments, n > 1; the accelerator includes: a plurality of processing channels, the accelerator performing a segmentation process on the data to be processed based on the segmentation indication, including: the accelerator divides the updated data to be processed into n continuous first data segments, wherein the jth first data segment comprises: j is more than or equal to 1 and less than n in the jth residual data segment in the auxiliary data and the data block between the jth residual data segment and the j +1 th residual data segment; the nth first data segment includes: the nth residual data segment in the auxiliary data and the data to be processed; and the accelerator performs the segmentation processing on the n first data segments in a one-to-one correspondence manner through n idle processing channels in the plurality of processing channels.

Optionally, the accelerator performs a segmentation process on the to-be-processed data based on the segmentation indication, including: when the data to be processed does not meet the target condition, the accelerator divides the data to be processed into at least two continuous second data segments; and the accelerator performs the segmentation processing on the at least two second data segments in a one-to-one correspondence manner through at least two idle processing channels in the plurality of processing channels.

It can be seen that, in the multi-channel compression processing mode, the accelerator processes the data to be processed through the multiple processing channels, so that the processing efficiency of the data to be processed is high. When the data to be processed is the data to be transmitted, the data to be processed is processed quickly, so that the data to be processed is transmitted quickly, and the transmission delay of the data to be processed is reduced. And when the data to be processed meets the target condition, the accelerator can also combine part of auxiliary data with the data to be processed to update the data to be processed, and then process the updated data to be processed, so that the number of the residual data segments obtained by processing can be reduced, and the purpose of compressing the residual data segments in the data to be processed is achieved.

Optionally, the processing indication and the processing response are each further configured to indicate: and the task number of the deduplication processing. Therefore, whether the deduplication processing of each data to be processed is finished or not can be accurately determined between the accelerator and the processor through the task number of the deduplication processing.

Optionally, before the accelerator performs the segmentation processing on the data to be processed based on the segmentation indication, the method further includes: the processor sends a mode setting instruction to the accelerator, wherein the mode setting instruction is used for indicating that: a target processing mode of the plurality of processing modes; the accelerator performs segmentation processing on the data to be processed based on the segmentation indication, and the segmentation processing comprises the following steps: and the accelerator performs the segmentation processing on the data to be processed by adopting the target processing mode based on the segmentation indication. It can be seen that the accelerator can select one processing mode from multiple processing modes (such as the above three processing modes) according to the instruction of the processor, and perform target processing on the data to be processed.

Optionally, before the processor sends the mode setting instruction to the accelerator, the method further includes: the processor acquires a first indication and a second indication aiming at the data to be processed, wherein the first indication is used for indicating whether the data to be processed needs priority processing, and the second indication is used for indicating whether the data to be processed needs quick processing; when the first indication is used for indicating that the to-be-processed data does not need priority processing, if the number of idle processing channels in the accelerator is equal to 1, the processor determines that the target processing mode is a first processing mode, and the processing unit comprises a plurality of processing channels; when the first indication is used for indicating that the to-be-processed data does not need to be processed preferentially, if the number of idle processing channels in the accelerator is greater than 1 and the second indication is used for indicating that the to-be-processed data needs to be processed rapidly, the processor determines that the target processing mode is a second processing mode; when the first indication is used for indicating that the to-be-processed data does not need to be processed preferentially, if the number of idle processing channels in the accelerator is greater than 1 and the second indication is used for indicating that the to-be-processed data does not need to be processed rapidly, the processor determines that the target processing mode is a third processing mode; when the first indication is used for indicating that the to-be-processed data needs to be processed preferentially, if the second indication is used for indicating that the to-be-processed data needs to be processed rapidly, the processor determines that the target processing mode is the second processing mode; when the first indication is used for indicating that the to-be-processed data needs to be processed preferentially, if the second indication is used for indicating that the to-be-processed data does not need to be processed rapidly, the processor determines that the target processing mode is the third processing mode; wherein A1 is more than A2, A1 is more than A3, B1 is more than or equal to B3, and B2 is more than or equal to B3; ax represents the number of processing channels used for executing the target processing by adopting an x-th processing mode, x is more than or equal to 1 and less than or equal to 3, and Bx represents: and obtaining the number of the residual data segments in the data to be processed after the target processing by adopting the x-th processing mode.

In a second aspect, there is provided a data processing method for use in an accelerator in a data processing apparatus, the data processing apparatus further comprising a processor, the method comprising: the accelerator receives a processing instruction sent by the processor, and the processing instruction comprises: a segment indication, and first storage information; the accelerator acquires data to be processed based on the first storage information; and the accelerator carries out segmentation processing on the data to be processed based on the segmentation indication to obtain a plurality of data blocks.

Optionally, the method further comprises: and the accelerator sends a processing response to the processor, wherein the processing response is used for indicating that the accelerator finishes the segmented processing of the data to be processed.

Optionally, the processing instructions further include: a signature indication; the method further comprises the following steps: the accelerator calculates a corresponding signature for each of the plurality of data chunks based on the signature indications.

Optionally, the method further comprises: the accelerator sends a processing response to the processor, the processing response being used for instructing the accelerator to complete the segmentation processing of the data to be processed and the calculation of the corresponding signature for each data block.

Optionally, the accelerator comprises: the device comprises a controller, a receiving cache, a sending cache and a processing unit; the accelerator receives a processing instruction sent by the processor, and comprises the following steps: the controller receives the processing instruction sent by the processor; the accelerator acquires data to be processed based on the first storage information, and the method comprises the following steps: the controller controls the receiving cache to acquire the data to be processed based on the first storage information; the accelerator performs segmentation processing on the data to be processed based on the segmentation indication, and the segmentation processing comprises the following steps: the controller controls the processing unit to acquire the data to be processed in the receiving cache and performs the segmentation processing on the data to be processed based on the segmentation indication; the method further comprises the following steps: the processing unit sends the data blocks and the signature corresponding to each data block to the sending cache; the processing unit sends a finish instruction to the controller; the controller controls the sending cache to store the plurality of data blocks and the signature corresponding to each data block based on the finishing indication; the accelerator sends a processing response to the processor, including: the controller sends the processing response to the processor.

Optionally, the method further comprises: and the accelerator sends a processing response to the processor, wherein the processing response is used for indicating the accelerator to finish processing the data to be processed. The processing comprises the following steps: the segmentation process described above, as well as computing a corresponding signature for each data block based on the previous indication, and querying the dictionary based on the plurality of data blocks.

Optionally, the processing instructions further include: a signature indication, a query indication, and an update indication; the method further comprises the following steps: the accelerator calculates a corresponding signature for each of the plurality of data chunks based on the signature indications; the accelerator queries a dictionary based on the query indication and the plurality of data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the accelerator replaces the first data block in the data to be processed with the first signature according to the updating indication; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the corresponding relation between the signature of the second data block and the second data block into the dictionary, and the data after the deduplication processing is all the data blocks which are not replaced by the signature and all the replaced signatures in the data to be processed.

Optionally, the method further comprises: the accelerator sends a processing response to the processor, wherein the processing response is used for indicating that the accelerator finishes processing the plurality of data blocks. The processing comprises the following steps: the segmentation processing includes calculating a corresponding signature for each data block based on the previous indication, querying a dictionary based on a plurality of data blocks, and replacing a first data block in the data to be processed with the first signature by the accelerator according to the update indication when the first data block in the plurality of data blocks and the first signature of the first data block are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the corresponding relation between the signature of the second data block and the second data block in the dictionary.

Optionally, before the accelerator performs the segmentation processing on the data to be processed based on the segmentation indication, the method further includes: the accelerator receives a mode setting instruction sent by the processor, wherein the mode setting instruction is used for indicating that: a target processing mode of the plurality of processing modes; the accelerator performs segmentation processing on the data to be processed based on the segmentation indication, and the segmentation processing comprises the following steps: and the accelerator performs the segmentation processing on the data to be processed by adopting the target processing mode based on the segmentation indication. It can be seen that the accelerator can select one processing mode from multiple processing modes (such as the above three processing modes) according to the instruction of the processor, and perform target processing on the data to be processed.

In a third aspect, a data processing method is provided, for a processor in a data processing apparatus, the data processing apparatus further including an accelerator, the method including: the processor sends a processing instruction comprising a segmentation instruction and first storage information to the accelerator, so that the accelerator acquires data to be processed based on the first storage information and performs segmentation processing on the data to be processed based on the segmentation instruction to obtain a plurality of data blocks.

Optionally, the method further comprises: the processor calculating a corresponding signature for each of the plurality of data blocks; the processor queries a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing. When the processor queries the dictionary based on each data block, the processor may query whether the dictionary has the data block recorded therein.

Optionally, before the processor calculates a corresponding signature for each of the plurality of data blocks, the method further comprises: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator finishes segmented processing on the data to be processed.

Optionally, the processing instructions further include: a signature indication; the method further comprises the following steps: the processor queries a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing.

Optionally, before the processor queries a dictionary from the plurality of data blocks, the method further comprises: the processor receives a processing response from the accelerator indicating that the accelerator completed processing the plurality of data blocks.

Optionally, the accelerator comprises: the device comprises a controller, a receiving cache, a sending cache and a processing unit; the processor sends a processing indication to the accelerator, including: the processor sends the processing indication to the controller; the processor receiving a processing response from the accelerator, comprising: the processor receives the processing response from the controller.

Optionally, the processing instructions further include: a signature indication and a query indication; the method further comprises the following steps: replacing, by the processor, a first data block of the plurality of data blocks and a first signature of the first data block with the first signature when the first data block and the first signature are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the processor stores the corresponding relation between the signature of the second data block and the second data block in the dictionary; the processor can also determine all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed as the data after the deduplication processing.

Optionally, before the processor replaces the first data block with the first signature, the method further comprises: the processor receives a processing response from the accelerator indicating that the accelerator completed processing the plurality of data blocks.

Optionally, the processing instructions further include: a signature indication, a query indication, and an update indication; the method further comprises the following steps: and the processor takes the data processed by the accelerator as the data after the deduplication processing, and the data processed by the accelerator are all the data blocks which are not replaced by the signatures and all the replaced signatures in the data to be processed.

Optionally, before the processor uses all data blocks in the data to be processed that are not replaced by the signature and all replaced signatures as the data after the deduplication processing, the method further includes: the processor receives a processing response from the accelerator, wherein the processing response is used for indicating that the accelerator finishes processing the data to be processed.

Optionally, the method further comprises: the processor sends a mode setting instruction to the accelerator, wherein the mode setting instruction is used for indicating that: a target processing mode among the plurality of processing modes.

In a fourth aspect, a data processing apparatus is provided, the data processing apparatus comprising: a processor for executing the method performed by the processor in the data processing method provided by the first aspect, and an accelerator for executing the method performed by the accelerator in the data processing method provided by the first aspect.

In a fifth aspect, an accelerator for executing the data processing method provided in the second aspect is provided.

In a sixth aspect, a processor is provided, which is configured to execute the data processing method provided in the third aspect.

In a seventh aspect, there is provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method of the second or third aspect.

In an eighth aspect, a computer-readable storage medium is provided, in which a computer program is stored, which, when executed by a processor, implements the data processing method of the second or third aspect.

In a ninth aspect, there is provided a chip, characterized in that the chip comprises programmable logic circuits and/or program instructions, which when run, are adapted to implement the data processing method according to the first, second or third aspect.

Drawings

Fig. 1 is a schematic diagram of a data deduplication process provided in an embodiment of the present application;

fig. 2 is a schematic view of an application scenario of the data processing method according to the embodiment of the present application;

fig. 3 is a schematic view of an application scenario of the data processing method according to the embodiment of the present application;

fig. 4 is a schematic structural diagram of a data processing apparatus provided in the related art;

fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 6 is a flowchart of a data processing method according to an embodiment of the present application;

fig. 7 is a schematic diagram of a first processing manner provided in the embodiment of the present application;

fig. 8 is a schematic diagram of a second processing manner provided in the embodiment of the present application;

FIG. 9 is a schematic diagram illustrating a relationship between a data set and data to be processed according to an embodiment of the present application;

fig. 10 is a schematic diagram of a third processing manner provided in the embodiment of the present application;

fig. 11 is a flowchart illustrating a processor determining a target processing manner in a plurality of processing manners according to an embodiment of the present application.

Detailed Description

In order to make the principles, technical solutions and advantages of the present application clearer, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

With the development of technology, in some scenarios, more and more data needs to be stored by the device, and in other scenarios, more and more data needs to be forwarded by the device. Generally, a data deduplication processing method can be adopted to perform data storage on the data required by the device so as to reduce the storage space required by the data storage, or perform data forwarding on the data required by the device so as to improve the data transmission speed.

The data processing method provided by the embodiment of the application is also used for carrying out deduplication processing on data. Before describing the embodiments of the present application, a description will be given of a data deduplication process.

Data deduplication, also known as data deduplication, is a technique for eliminating duplicate data. The data deduplication technology detects whether a processed data block exists in the data to be processed, and replaces the data block with a signature (signature) smaller than the data block. When the data to be processed is used, the signature of the data block is restored to the data block.

The data processing apparatus may be configured to perform the deduplication processing. The device that needs to perform deduplication processing on data may be referred to as a data processing device, and the deduplication processing performed by the data processing device on the data to be processed may include four stages, which are respectively: a segmentation phase, a signature phase, a query phase and an update phase.

(1) In the segmentation stage, the data processing apparatus generally segments the data to be processed by using a segmentation algorithm to obtain a plurality of data blocks (chunk) in the data to be processed. It should be noted that the data to be processed may only include the data block, or may include not only the data block but also the remaining data segment other than the data block. When the data processing device segments the data to be processed, any one of a plurality of segmentation modes can be adopted for segmentation.

For example, the data processing apparatus may segment the data to be processed according to a fixed data length, where the length of each obtained data block is the fixed data length, and when the length of the data to be processed is not an integer multiple of the fixed data length, the data to be processed further includes a remaining data segment, and the length of the remaining data segment is smaller than the fixed data length. For example, assuming that the length of the data to be processed is 100 bytes and the fixed data length is 10 bytes, the data processing apparatus may divide the data to be processed into 10 data blocks, each of which has a length of 10 bytes. If the length of the data to be processed is 102 bytes and the fixed data length is 10 bytes, the data processing apparatus may divide the data to be processed into 10 data blocks and a remaining data segment, where the length of the data block is 10 bytes and the length of the remaining data segment is 2 bytes. This method of segmentation may be referred to as a fixed-size chunking (FSC) -based method, and the length of each data block obtained by this method is equal.

For another example, the data processing apparatus may segment the data to be processed according to a fingerprinting algorithm. For example, the data processing apparatus may perform a fingerprint operation on the data to be processed with a sliding window of a fixed size. For example, the data processing apparatus may first obtain a portion of data of the data to be processed in the sliding window, calculate a fingerprint of the portion of data (e.g., calculate a fingerprint of the portion of data by using a hash algorithm), and determine whether the sliding window reaches a segment point in the data to be processed based on the fingerprint of the portion of data. And if the sliding window currently reaches the segmentation point, taking the first data in the sliding window as the segmentation point. Then, the data processing device slides the sliding window to replace the part of the data needing to calculate the fingerprint, and repeatedly executes the fingerprint of the data in the calculating sliding window and the process of determining the segmentation points until all the segmentation points in the data to be processed are found. Finally, the data processing device segments the data to be processed based on all the segmentation points in the data to be processed. This segmentation method may belong to a content-defined chunking (CDC) method, and the lengths of the respective data blocks obtained by this method may be equal or different. It should be noted that, in the embodiment of the present application, the CDC method includes segmenting the data to be processed according to the fingerprint algorithm as an example. Optionally, the CDC method may also include other methods that do not require computing a fingerprint to be able to segment the data. For example, a curve of American Standard Code for Information Interchange (ASCII) values of each character in the data to be processed is drawn, and the data to be processed is segmented using a maximum value/minimum value in the curve as a segmentation point.

(2) In the signature phase, the data processing apparatus needs to calculate a corresponding signature for each data block obtained in the segmentation phase (the signature corresponding to the data block may also be referred to as a signature of the data block). In the embodiment of the present application, the signature of the data block may be calculated in various ways, for example, the signature of the data block is calculated by using a hash algorithm, or the position of the data block in the dictionary is used as the signature of the data block (where the meaning of the dictionary is explained in the following query stage).

For example, as shown in fig. 1, it is assumed that the data processing apparatus obtains a data block 1, a data block 2, a data block 3, a data block 4, and a data block 5 after performing segmentation processing on data to be processed. In the signature phase, the data processing apparatus needs to calculate a signature S1 for data block 1, a signature S2 for data block 2, a signature S3 for data block 3, a signature S4 for data block 4, and a signature S5 for data block 5.

(3) In the query phase, the data processing apparatus needs to query the dictionary on a per data block basis. The dictionary includes a correspondence of the data block to the signature. After querying the dictionary, the data processing device can know whether the dictionary includes each data block obtained by the segmentation processing.

For example, assuming that the correspondence between the data blocks in the dictionary and the signatures is shown in table 1, it can be seen that the dictionary includes data blocks 4 and 6. The data processing apparatus knows that the dictionary includes the data block 4 after referring to the dictionary, and the data blocks 1, 2, 3, and 5 are not recorded in the dictionary.

TABLE 1

Data block	Signature
		Data block 4	S4
Data block 6	S6

(4) In the updating stage, when a first data block in the multiple data blocks and a first signature of the first data block are in the dictionary, the data processing device needs to replace the first data block with the first signature to obtain data after deduplication processing; when a second data block of the plurality of data blocks is not in the dictionary, the data processing apparatus needs to store the correspondence relationship between the signature of the second data block and the second data block in the dictionary.

The first data chunk may be understood as a duplicate data chunk of the plurality of data chunks. The second data chunk may be understood as a non-duplicate data chunk of the plurality of data chunks. If there may be multiple duplicate data blocks in the multiple data blocks, the data processing apparatus needs to replace each duplicate data block with a corresponding signature in the update phase.

For example, taking the example of the query phase as an example, the dictionary includes the data block 4 and its corresponding signature S4, and the data block 4 may be considered as the first data block, and the signature S4 may be the first signature. With continued reference to fig. 1, the data processing apparatus may replace data block 4 with a signature S4 corresponding to data block 4, while other data blocks (e.g., data blocks 1, 2, 3, 5) are unchanged. It should be noted that the length of the signature of the data block is shorter than the length of the data block, and after the data block is replaced with the signature, the length of the entire data to be processed becomes smaller. If the data to be processed is data that needs to be stored, the amount of data stored can be reduced, and if the data to be processed is data that needs to be forwarded, the amount of data forwarded can be reduced. In the embodiment of the present application, the plurality of data chunks may include only one duplicate data chunk, that is, only one first data chunk exists; the plurality of data blocks may also include a plurality of duplicate data blocks, that is, there are a plurality of first data blocks, and at this time, the data processing apparatus needs to replace each of the plurality of first data blocks with a corresponding signature.

Further, the data blocks 1, 2, 3, and 5 are not recorded in the dictionary, and the data blocks 1, 2, 3, and 5 may be considered as the second data blocks. In the update phase, the data processing apparatus may further add, to the dictionary, the correspondence of the data block 1 to the signature S1, the correspondence of the data block 2 to the signature S2, the correspondence of the data block 3 to the signature S3, and the correspondence of the data block 5 to the signature S5. In this embodiment of the present application, a plurality of data blocks includes a plurality of second data blocks as an example, optionally, the plurality of data blocks may include only one second data block, and this is not limited in this embodiment of the present application.

TABLE 2

Data block	Signature
		Data block 4	S4
Data block 6	S6
		Data block 1	S1
Data block
	2	S2
Data block
	3	S3
Data block 5			S5

After the data to be processed is processed in the four stages, the data after the deduplication processing can be recovered, and the data after the deduplication processing may include: and all data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed. After the data after the deduplication processing is obtained, if data to be processed (that is, data that is not deduplicated) needs to be obtained, the data after the deduplication processing needs to be decoded.

On one hand, when the data to be processed is data that needs to be forwarded by the data processing apparatus, if the data processing apparatus forwards the data after the deduplication processing to another apparatus and the another apparatus needs to obtain the data to be processed, the another apparatus may also decode the data after the deduplication processing to obtain the data to be processed.

On the other hand, when the data to be processed is the data that needs to be stored, if the stored data to be processed needs to be read, the data processing apparatus may first read the data after the deduplication processing. Then, the data processing apparatus needs to decode the data after the deduplication processing to obtain the data to be processed.

For example, the process of decoding the data after the deduplication processing may include: and querying the dictionary to find a data block corresponding to the signature in the data subjected to the deduplication processing, and replacing the signature in the data subjected to the deduplication processing with the corresponding data block to obtain the data to be processed. For example, taking the above example of the update phase as an example, the data processing apparatus may query the dictionary to determine the data block 4 corresponding to the signature S4, and replace the signature S4 in the deduplicated data with the data block 4, thereby obtaining the above data to be processed. Further, in the process of decoding the data after the deduplication processing, if it is found that the dictionary does not include the signature of a certain data block, the corresponding relationship between the data block and the signature thereof may be added to the dictionary (this process may refer to the process of adding the corresponding relationship between the second data block and the second signature in the update stage).

The process of the deduplication processing and decoding will be explained below by taking a specific scenario as an example.

Exemplarily, fig. 2 is a schematic view of an application scenario of the data processing method provided in the embodiment of the present application, and as shown in fig. 2, the application scenario includes: the system comprises a sending end A, a sending end B, a receiving end C, a receiving end D, a forwarding device E and a forwarding device F. The transmitting end A and the transmitting end B are both connected to a forwarding device E, the forwarding device E is connected to a forwarding device F, and the forwarding device F is connected to a receiving end C and a receiving end D. The forwarding device may be a router, a switch, a gateway, or other devices for forwarding data. The sending end and the receiving end can be devices such as a mobile phone, a computer, a server, a camera and the like. In this embodiment, for example, the forwarding device E includes the data processing device provided in this embodiment.

Assume that the sender a first sends data 1 to receiver C, and then sender B needs to send data 2 to receiver D.

On one hand, when the sending end a sends the data 1 to the receiving end C, the sending end a may send the data 1 to the forwarding apparatus E, and the forwarding apparatus E may perform the processing of the segmentation stage, the signature stage, the query stage, and the update stage on the data 1. For example, the forwarding device E may segment the data 1 into the data blocks 1, 2, 3, 4, and 5 in the segmentation stage, and since the data 1 is the first data forwarded by the forwarding device E, a dictionary in the forwarding device E does not record a signature corresponding to any data block before processing the data 1, and in the query stage, the forwarding device E may record a correspondence relationship between the data block 1 and its signature, a correspondence relationship between the data block 2 and its signature, a correspondence relationship between the data block 3 and its signature, a correspondence relationship between the data block 4 and its signature, and a correspondence relationship between the data block 5 and its signature in the dictionary. After the forwarding device performs the processing of the segmentation phase, the signature phase, the query phase and the update phase on the data 1, the obtained data after the deduplication processing still includes the data blocks 1, 2, 3, 4 and 5. Thereafter, the forwarding device may transmit the data after the deduplication process to the forwarding device F.

After receiving the data after the deduplication processing, the forwarding device F may query the dictionary to find a data block corresponding to the signature in the data after the deduplication processing, and replace the signature in the data after the deduplication processing with the corresponding data block, thereby obtaining the data 1. It should be noted that, since the data after the deduplication processing is the first data received by the forwarding device F, at this time, no corresponding relationship between any data block and a signature is recorded in the local dictionary, and the data after the deduplication processing does not include a signature, so that the data after the deduplication processing obtained by the forwarding device F after a series of processing is the same as the data 1 received by the forwarding device F. Finally, the forwarding device F may send data 1 to the receiving end C.

In the process of querying the local dictionary, if the dictionary does not include the correspondence between a certain data block and a signature in the data after the deduplication processing, the forwarding device F may add the correspondence between the data block and the signature in the local dictionary. For example, in the process of sending data 1 to the receiving end C by the sending end a, the forwarding device may add, to the dictionary, the correspondence between the data block 1 and the signature thereof, the correspondence between the data block 2 and the signature thereof, the correspondence between the data block 3 and the signature thereof, the correspondence between the data block 4 and the signature thereof, and the correspondence between the data block 5 and the signature thereof.

On the other hand, when the sending end B sends the data 2 to the receiving end D, the sending end B may send the data 2 to the forwarding apparatus E, and the forwarding apparatus E may perform the processing of the segmentation stage, the signature stage, the query stage, and the update stage on the data 2. Illustratively, the forwarding device E may segment the data 2 into data blocks 1, 2, 6, 4 and 5 in the segmentation stage, and since the data 2 is the second data forwarded by the forwarding device E, a dictionary in the forwarding device E already records the correspondence between each data block (e.g. data blocks 1, 2, 3, 4 and 5) in the data 1 and its signature before processing the data 2. In the query phase, the forwarding device E may replace the data block 1 in the data 2 with the signature S1 of the data block 1, replace the data block 2 in the data 2 with the signature S2 of the data block 2, replace the data block 4 in the data 2 with the signature S4 of the data block 4, replace the data block 5 in the data 2 with the signature S5 of the data block 5, and re-record the corresponding relationship between the data block 6 and the signature thereof in the dictionary. After the forwarding device performs the processing of the segmentation stage, the signature stage, the query stage and the update stage on the data 2, the obtained deduplicated data includes a signature S1, a signature S2, a signature S3, a data block 4 and a signature S5. Thereafter, the forwarding device may transmit the data after the deduplication process to the forwarding device F.

After receiving the data subjected to the deduplication processing, the forwarding device F may query a local dictionary to find a data block corresponding to the signature in the data subjected to the deduplication processing, and replace the signature in the data subjected to the deduplication processing with the corresponding data block, thereby obtaining the data 2. For example, data 2 including data blocks 1, 2, 3, 4, and 5 is obtained by replacing the signature S1 in the deduplicated data with data block 1, replacing the signature S2 in the deduplicated data with data block 2, replacing the signature S3 in the deduplicated data with data block 3, and replacing the signature S5 in the deduplicated data with data block 5. Finally, the forwarding device F may send the data 2 to the receiving end D.

In the process of querying the local dictionary, if the queried local dictionary does not include the correspondence between a certain data block in the deduplicated data and the signature thereof, the forwarding device F may add the correspondence between the data block and the signature thereof to the local dictionary. For example, in the process of sending the data 2 from the sending end B to the receiving end D, since the correspondence between the signature and the data block 4 in the data after the deduplication processing is already recorded in the dictionary local to the forwarding device, the forwarding device does not need to add any correspondence to the dictionary in the process.

The process of the deduplication processing and decoding will be explained below by taking another specific scenario as an example.

Exemplarily, fig. 3 is a schematic view of an application scenario of the data processing method provided in the embodiment of the present application, and as shown in fig. 3, the application scenario includes: a first device Y1, a second device Y2 and a storage device Y3. The first device Y1 and the second device Y2 are both connected to the storage device Y3. The first device Y1, the second device Y2 and the storage device Y3 may be any devices, such as a mobile phone, a computer, a server, a camera, and the like. In the embodiment of the present application, the storage device Y3 includes the data processing device provided in the embodiment of the present application as an example.

Assume that the first device Y1 first stores data 1 in the storage device Y3, and thereafter, the second device Y2 stores data 2 in the storage device Y3.

On the one hand, when the first device Y1 stores data 1 in the storage device Y3, the first device Y1 may transmit data 1 to the storage device Y3, and the storage device Y3 may perform a re-processing on the data 1. The process of the deduplication processing may refer to the deduplication processing process of the forwarding device E on the data 1, which is not described herein again in this embodiment of the application. When the storage device Y3 stores data 1, the data actually stored is the data after the deduplication processing (the same as data 1), and includes data block 1, data block 2, data block 3, data block 4, and data block 5. When the first device Y1 needs to read the data 1, the storage device Y3 may refer to the above-mentioned forwarding device F to obtain the data 1 based on the deduplicated data, and send the data 1 to the first device Y1.

On the other hand, when the second device Y2 stores data 2 in the storage device Y3, the second device Y2 may transmit the data 2 to the storage device Y3, and the storage device Y3 may perform deduplication processing on the data 2. The process of deduplication processing may refer to the deduplication processing process of the forwarding device E on the data 2, which is not described herein again in this embodiment of the application. Thus, when the storage device Y3 stores data 2, the data actually stored is not data 2, but the data after the deduplication processing includes the signature S1, the signature S2, the signature S3, the data block 4, and the signature S5. When the second device Y2 needs to read the data 2, the storage device Y3 may refer to the above-mentioned forwarding device F to obtain the data 2 based on the deduplicated data, and send the data 2 to the second device Y2.

In the embodiment of the present application, the first device Y1 and the second device Y2 are taken as examples of different devices, alternatively, the first device Y1 and the second device Y2 may be the same device, and the embodiment of the present application does not limit this. In addition, in the embodiment of the present application, taking the first device Y1 reading data 1 and the second device Y2 reading data as an example, optionally, the first device Y1 may also read data 2, and the process of the first device Y1 reading data 2 may refer to the process of the second device Y2 reading data 2; the second device Y2 may also read data 1, and the process of the second device Y2 reading data 1 may refer to the process of the first device Y1 reading data 1.

It should be noted that, as shown in fig. 4, a data processing apparatus according to the related art includes: a processor, a memory, and a storage medium. Wherein the storage medium and the memory are different storage components in the data processing apparatus. The data to be processed is usually located in a memory, and the processor can read and write the memory to realize the deduplication processing of the data to be processed. Moreover, the dictionary used in the deduplication processing process can be stored in a storage medium, and the data to be processed in the storage medium is read into the memory by the processor, so that the processor can be used in the query stage in the deduplication processing.

It can be seen that the deduplication processing in the related art is performed by a processor in the data processing apparatus. The data processing process is complicated, and the workload is large, so that the load of the processor is high. Moreover, when the data to be processed is data to be forwarded, if the load of the processor is high, the efficiency of the processor for performing deduplication processing on the data is reduced, so that the forwarding efficiency of the data to be processed is affected.

The embodiment of the application provides a data processing device, which not only comprises a processor but also comprises an accelerator, and the accelerator can assist the processor to perform deduplication processing on data to be processed so as to reduce the load of the processor.

Fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, and as shown in fig. 5, the data processing apparatus includes: a processor, an accelerator, a memory, and a storage medium.

The processor may be any device having a processing function, such as a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP. The data to be processed may be stored in a memory. The data to be processed may be data transmitted in a network read by the processor from the network card, and stored in the memory. Alternatively, the data to be processed may be data stored in a storage medium read by the processor and stored in the memory. The data to be processed in the memory may also be data from other sources, which is not limited in this embodiment of the application. The storage medium may be a hard disk, a Random Access Memory (RAM), a Secure Digital (SD) card, or the like.

The dictionary used in the query phase may be stored in a storage medium. Alternatively, the dictionary may be directly stored in the memory. When the memory stores the dictionary, the data processing apparatus may not include a storage medium, which is not limited in the embodiment of the present application.

With continued reference to fig. 5, the accelerator may include: the system comprises a controller, a receiving buffer, a sending buffer and a processing unit, wherein the processing unit comprises at least one processing channel (channel), and fig. 5 exemplifies that the processing unit comprises four processing channels. The controller is connected to the processor, the receive buffer, the transmit buffer, and the processing unit (e.g., each processing channel in the processing unit), and the receive buffer and the transmit buffer are connected to the memory and the processing unit (e.g., each processing channel in the processing unit).

In the process of performing deduplication processing on data to be processed by the accelerator auxiliary processor, the controller may control the receiving cache to acquire the data to be processed from the memory according to an instruction issued by the processor, and control the processing unit to perform at least part of deduplication processing on the data to be processed acquired by the receiving cache. After the processing unit finishes processing the data to be processed, the processing unit may send the processed data to the sending buffer, and notify the controller that at least part of the processing has been completed. Then, the controller may control the sending buffer to send the data processed by the processing unit to the memory, and notify the processor to obtain the data processed by the processing unit in the memory. The processor may then process the data processed by the processing unit in the accelerator to obtain deduplicated data (which is obtained by performing deduplication processing on the data to be processed). At least part of the processing performed by the accelerator may include at least one stage of the deduplication processing described above, which is not limited in this embodiment.

It should be noted that, the controller, the receiving buffer, the sending buffer, and the processing unit may all be hardware structures, and optionally, when the accelerator is implemented in a hardware manner, the structure of each hardware in the accelerator may also be different from that of the accelerator in fig. 5, for example, the receiving buffer and the sending buffer in fig. 5 are integrated into one buffer, or the receiving buffer and the sending buffer in fig. 5 are integrated into the processing unit, or the accelerator is implemented by a Field Programmable Gate Array (FPGA), or the accelerator is implemented by an NP.

Alternatively, while the accelerator may be implemented in software + hardware, some of the structure in the accelerator (such as the controller described above) may be implemented in logic running on a processor or other hardware, and another part of the structure in the accelerator may be implemented in hardware.

In addition, all structures in the data processing device can be arranged on the same circuit board to form a module (which can be called as a chip) with rich functions; alternatively, the structures in the data processing apparatus may also be disposed on different circuit boards in a distributed manner, which is not limited in this embodiment of the application.

The structure and function of the accelerator are explained above in general, and the function of the accelerator will be further explained below by taking the data processing method provided in the embodiment of the present application as an example.

Fig. 6 is a flowchart of a data processing method provided in an embodiment of the present application, where the method may be used in a data processing apparatus (such as the data processing apparatus shown in fig. 5) provided in an embodiment of the present application. As shown in fig. 6, the data processing method may include:

step 601, the processor sends a processing instruction to the accelerator.

There are various implementation manners for the processing instruction, and four of them are explained below as an example.

In a first implementation, processing the indication includes: a segment indication, and first storage information. Wherein the segment indication is used to instruct the accelerator to perform segment processing on the data to be processed (i.e., to perform processing of the segmentation stage in data deduplication). The first storage information is used for indicating the position of the data to be processed. Illustratively, the first stored information may include: the storage address of the first data and the storage address of the last data in the data to be processed, or the storage address of the first data in the data to be processed and the length of the data to be processed. Optionally, the data to be processed may be located in a memory, and thus the storage address in the first storage information may be a storage address in the memory. Of course, the data to be processed may not be located in the memory, and this is not limited in this embodiment of the application.

In a second implementation, processing the indication includes: a segment indication, a signature indication, and first stored information. The signature indication is used to instruct the accelerator to calculate a corresponding signature for each data block in the data to be processed (i.e., to perform the processing of the signature phase in the data deduplication). For the explanation of the segment indication and the first storage information, reference may be made to the relevant explanation in the first implementation manner, and details are not described herein in this embodiment of the application.

In a third implementation, processing the indication includes: a segment indication, a signature indication, a query indication, and first stored information. The query indication is used to instruct the accelerator to query the dictionary based on the plurality of data blocks obtained by the segmentation process (i.e., to perform the process of the query phase in the data deduplication). The explanation of the segment indication and the first storage information may refer to a relevant explanation in the first implementation manner, and the explanation of the signature indication may refer to a relevant explanation in the second implementation manner, which is not described herein again in this embodiment of the application.

In a fourth implementation, processing the indication includes: a segment indication, a signature indication, a query indication, an update indication, and first stored information. Wherein the update indication is to instruct the accelerator to replace a first data block of the plurality of data blocks with a first signature when the first data block and the first signature of the first data block are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, storing the corresponding relation between the signature of the second data block and the second data block into the dictionary (namely, performing the processing of the update stage in the data deduplication). The explanation of the segment indication and the first storage information may refer to a relevant explanation in the first implementation manner, the explanation of the signature indication may refer to a relevant explanation in the second implementation manner, and the explanation of the query indication may refer to a relevant explanation in the third implementation manner, which is not described herein again in this embodiment of the application.

It can be seen that in the above four implementations, the processor may send a processing instruction to the accelerator, so that the accelerator performs at least part of the deduplication processing, so as to facilitate the subsequent accelerator auxiliary processor to perform the above data deduplication process.

Illustratively, when the internal structure of the accelerator is as shown in fig. 5, the processor may send the above-described processing indication to the controller in the accelerator in step 601.

When the processor sends the processing instruction to the accelerator, the processor may call an encapsulation interface of the accelerator to generate a command that can be sent to the accelerator, and then fill all information that the processing instruction needs to be instructed in the command to obtain the processing instruction, and then send the processing instruction to the accelerator.

Step 602, the accelerator acquires the data to be processed based on the first storage information in the processing instruction.

The processing instruction includes first storage information, and the accelerator needs to acquire the data to be processed based on the first storage information before processing the data to be processed based on the instruction in the processing instruction. For example, when the data to be processed is located in the memory, the accelerator may obtain the data to be processed from the memory based on the first storage information.

Illustratively, when the internal structure of the accelerator is as shown in fig. 5, the controller may send the first storage information to the receive cache in step 602. The receiving cache may obtain the data to be processed in the memory based on the first storage information, and cache the obtained data to be processed in the receiving cache.

Step 603, the accelerator processes the data to be processed based on the processing instruction.

Note that, when the implementation of the processing instruction described above is different, the processing executed by the accelerator is also different.

When the processing instruction is implemented in the first implementation manner, the processing executed by the accelerator includes: and (5) carrying out segmentation treatment in the deduplication treatment. At this time, the processing the data to be processed by the accelerator based on the processing instruction includes: and the accelerator performs segmentation processing on the data to be processed based on the segmentation indication in the processing indication to obtain a plurality of data blocks. For the segmentation processing process, reference may be made to the foregoing explanation of the segmentation stage, and details are not described herein in this embodiment of the application.

When the processing instruction is implemented in the second implementation manner, the processing executed by the accelerator includes: and the process of a segmentation stage and a signature stage in the deduplication process. At this time, the processing the data to be processed by the accelerator based on the processing instruction includes: the accelerator performs segmentation processing on the data to be processed based on the segmentation indication in the processing indication to obtain a plurality of data blocks, and calculates a corresponding signature for each data block based on the signature indication in the processing indication. The segmentation processing process may refer to the explanation of the segmentation stage, and the process of calculating the corresponding signature for each data block may refer to the explanation of the signature stage, which is not described herein again in this embodiment of the present application.

When the processing instruction is implemented in the third implementation manner, the processing executed by the accelerator includes: the de-duplication process includes the steps of segmenting, signing and inquiring. At this time, the processing the data to be processed by the accelerator based on the processing instruction includes: the accelerator performs segmentation processing on the data to be processed based on the segmentation indications, calculates a corresponding signature for each data block based on the signature indications, and queries the dictionary based on each data block. The segmentation processing process may refer to the explanation of the segmentation stage, the process of calculating a corresponding signature for each data block may refer to the explanation of the signature stage, and the process of querying a dictionary based on each data block may refer to the explanation of the query stage, which is not described herein again in this embodiment of the present application.

When the processing instruction is implemented in the fourth implementation manner, the processing executed by the accelerator includes: the de-duplication process comprises a segmentation stage, a signature stage, a query stage and an update stage. At this time, the processing the data to be processed by the accelerator based on the processing instruction includes: the accelerator performs segmentation processing on data to be processed based on the segmentation indication, calculates a corresponding signature for each data block based on the signature indication, queries a dictionary based on each data block, and replaces a first data block of the plurality of data blocks with a first signature when the first data block and the first signature of the first data block are in the dictionary; and when a second data block in the plurality of data blocks is not in the dictionary, storing the corresponding relation between the signature of the second data block and the second data block into the dictionary. The process of the segmentation process can refer to the aforementioned explanation of the segmentation stage. The process of calculating a corresponding signature for each data block may refer to the aforementioned explanation of the signature phase. The process of querying the dictionary on a per data block basis may refer to the foregoing explanation of the query phase. When a first data block of the multiple data blocks and a first signature of the first data block are in the dictionary, the first data block is replaced with the first signature, and when a second data block of the multiple data blocks is not in the dictionary, a corresponding relationship between the signature of the second data block and the second data block is stored in the dictionary.

It should be noted that, in the first implementation manner and the second implementation manner, the processing in the query phase is performed by the processor, in this case, a storage address of the dictionary may be stored in the processor, or the dictionary may be stored in the processor, so that the processor can query the dictionary in the query phase. In the third implementation manner and the fourth implementation manner, the processing in the query stage is performed by the accelerator, in this case, a storage address of the dictionary may be stored in the accelerator, or the dictionary may be stored in the accelerator, so that the accelerator can query the dictionary in the query stage.

For example, when the internal structure of the accelerator is as shown in fig. 5, in step 603, the controller may control the processing unit to acquire the data to be processed on the receiving buffer and process the data to be processed based on the processing instruction.

Further, after processing the data to be processed, the processing unit may send the processed data to the sending buffer, and send an end instruction to the controller. After receiving the finishing instruction, the controller may determine that the current controller has finished processing the data to be processed, and the data processed by the processing unit has been stored in the sending cache, and at this time, the controller may control the sending cache to store the data processed by the processing unit. Optionally, the controller may control the sending buffer to store the data processed by the processing unit in the memory, so that the processor can obtain the data processed by the processing unit in the accelerator in a subsequent step.

The data obtained by the processing unit is also the data obtained by the accelerator. When the implementation of the processing instruction is different, the accelerator processes the resulting data differently. Illustratively, when the processing indication comprises the above-described segmentation indication, the data processed by the accelerator comprises a plurality of data blocks. When the processing indication includes the segment indication and the signature indication, the data processed by the accelerator includes a plurality of data blocks and a signature of each data block. When the processing indication comprises a segmentation indication, a signature indication and a query indication, the data processed by the accelerator comprises: a plurality of data blocks, a signature for each data block, and whether the dictionary contains the query results for each data block. When the processing indication comprises a segmentation indication, a signature indication query indication and an update indication, the data processed by the accelerator comprises: all data blocks in the plurality of data blocks that have not been replaced with a signature, and all signatures that have been replaced.

Step 604, the accelerator sends a processing response to the processor.

After the accelerator finishes the processing of step 603 and obtains the data, the accelerator may send a processing response to the processor, where the processing response is used to indicate that the accelerator has finished processing the data to be processed in step 603.

Optionally, the process response may also be used to indicate second stored information of the data processed by the accelerator. The processor may then retrieve the data processed by the accelerator based on the second stored information indicated by the processing response. For example, when the accelerator processes different data, the second storage information is implemented differently.

For example, when the data processed by the accelerator includes the plurality of data blocks, the second storage information may include: storage information of a plurality of data blocks.

When the data processed by the accelerator includes a plurality of data blocks and a signature of each data block, the second method may include: a plurality of data blocks and a signature for each data block. For example, the second may include a plurality of sub-storage information in one-to-one correspondence with the plurality of data blocks, each sub-storage information including a corresponding data block and storage information of a signature corresponding to the data block. As yet another example, the second may include: and the two pieces of sub-storage information, wherein one piece of sub-storage information is storage information of a plurality of data blocks, and the other piece of sub-storage information is signed storage information corresponding to the plurality of data blocks. The storage information indicated by the processing response may have various implementation manners, and details are not described herein in this embodiment of the application.

When the data processed by the accelerator includes a plurality of data blocks, the signature of each data block, and whether the dictionary contains the query result of each data block, the second method may include: a plurality of data blocks, a signature for each data block, and stored information of the query result. For example, the second may include a plurality of pieces of sub-storage information in one-to-one correspondence with a plurality of data blocks, each piece of sub-storage information including a corresponding data block, a signature corresponding to the data block, and storage information of whether the dictionary includes a query result for the data block. As yet another example, the second may include: and the three pieces of sub-storage information comprise one piece of sub-storage information, another piece of sub-storage information and another piece of sub-storage information, wherein one piece of sub-storage information is storage information of a plurality of data blocks, the other piece of sub-storage information is storage information of signatures corresponding to the plurality of data blocks, and the other piece of sub-storage information is storage information of whether the dictionary contains a query result of each data block. The second may have a plurality of implementation manners, which are not described herein again.

When the data processed by the accelerator includes all data blocks in the plurality of data blocks that are not replaced by the signature and all signatures that have been replaced, the second method may include: all data blocks in the plurality of data blocks that have not been replaced with a signature, and all stored information of signatures that have been replaced.

Optionally, the processing response may contain the second stored information.

Or, the processing response does not include the second storage information, but the processor stores the second storage information in advance, and the processor is further provided with a register, and before the accelerator sends the processing response to the processor, the value of the register is used for indicating that the position indicated by the second storage information does not store the data processed by the accelerator. The processing response may be used to instruct the processor to update the value of the register so that the value of the register is used to indicate that the location indicated by the second storage information stores the data processed by the accelerator. At this time, the processor can know that the data processed by the accelerator is the data stored in the location indicated by the second storage information.

Optionally, the processing instruction issued by the processor in step 601 may further include the second storage information, which is not limited in this embodiment of the application.

Illustratively, when the internal structure of the accelerator is as shown in fig. 5, the controller in the accelerator may send the above-described processing response to the processor in step 604.

It can be seen that, after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator completes processing the data to be processed, but waits for the processing response sent by the accelerator after completing processing the data to be processed, so as to further reduce the load of the processor. Of course, the accelerator may not send the processing response to the processor in step 604, but the processor may continuously (e.g., periodically) detect whether the accelerator has performed the processing of the data to be processed, which is not limited in this embodiment of the present invention.

In step 605, the processor obtains the data processed by the accelerator based on the second storage information indicated by the processing response.

After receiving the processing response, the processor may obtain the second storage information based on the processing response, and obtain the data processed by the accelerator at the position indicated by the second storage information, so that the data processed by the accelerator can be processed in the subsequent step to obtain the deduplicated data. For the explanation of the data obtained by the accelerator, reference may be made to step 604, and details of the embodiment of the present application are not described herein.

And step 606, processing the data obtained by the accelerator by the processor to obtain data subjected to deduplication processing.

It should be noted that, when the processing instructions are different, the processing that the processor needs to execute in step 606 is also different.

When the processing instruction is implemented in the first implementation manner, the processing executed by the accelerator includes: and (4) processing in a segmentation stage in the deduplication processing. At this time, the processor needs to perform the processing of the tag phase, the query phase, and the update phase in the deduplication processing. For example, the processor needs to calculate a corresponding signature for each data block based on the signature indication in the processing indication; the processor then needs to query the dictionary on a per data block basis; and finally, when a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor replaces the first data block in the data to be processed with the first signature according to the updating indication to obtain the data after the duplication elimination processing. The processor is further configured to store a signature of a second data block of the plurality of data blocks in the dictionary when the second data block is not in the dictionary. The processor may further use all of the data blocks in the plurality of data blocks that are not replaced with the signature and all of the replaced signatures as the deduplicated data.

When the processing instruction is implemented in the second implementation manner, the processing executed by the accelerator includes: and the process of a segmentation stage and a signature stage in the deduplication process. At this time, the processor needs to perform the processing of the inquiry stage and the update stage in the deduplication processing. Illustratively, the processor queries whether a signature of each data block is recorded in the dictionary; and when a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor replaces the first data block in the data to be processed with the first signature according to the updating indication to obtain the data after the deduplication processing. The processor is further configured to store a signature of a second data block of the plurality of data blocks in the dictionary when the second data block is not in the dictionary. The processor may further use all of the data blocks in the plurality of data blocks that are not replaced with the signature and all of the replaced signatures as the deduplicated data.

When the processing instruction is implemented in the third implementation manner, the processing executed by the accelerator includes: the de-duplication process includes the steps of segmenting, signing and inquiring. At this time, the processor needs to perform the processing of the update stage in the deduplication processing. For example, when a first data block of the multiple data blocks and a first signature of the first data block are in the dictionary, the processor needs to replace the first data block in the data to be processed with the first signature according to the update instruction, so as to obtain the data after the deduplication processing. The processor is further configured to store a signature of a second data block of the plurality of data blocks in the dictionary when the second data block is not in the dictionary. The processor may further use all of the data blocks in the plurality of data blocks that are not replaced with the signature and all of the replaced signatures as the deduplicated data.

When the processing instruction is implemented in the fourth implementation manner, the processing executed by the accelerator includes: the de-duplication process comprises a segmentation stage, a signature stage, a query stage and an update stage. At this time, four stages in the deduplication processing are all executed by the accelerator, and the processing, by the processor, of the data processed by the accelerator may include: the processor takes all the data blocks which are not replaced by the signatures and all replaced signatures in the plurality of data blocks as the data after the deduplication processing.

The tag phase, the query phase, and the update phase may refer to the foregoing explanations of the tag phase, the query phase, and the update phase, which are not described herein in detail in this embodiment of the present application.

For example, in the data processing method provided in the embodiment of the present application, after the processor sends the processing instruction to the accelerator, the accelerator may obtain data to be processed based on the first storage information, and perform segmentation processing on the data to be processed based on the segmentation instruction to obtain a plurality of data blocks; then, the data processing apparatus can perform target processing on the plurality of data blocks to obtain data after deduplication processing.

When the processing instructions are different in implementation mode, the target processing is different in implementation mode.

Illustratively, when the processing directive includes a segmentation directive, the target processing is implemented by a processor, and the target processing includes the processing of the tagging phase, the querying phase, and the updating phase described above. The target process may further include: the processor takes all the data blocks which are not replaced by the signatures and all replaced signatures in the plurality of data blocks as the data after the deduplication processing.

When the processing indication includes a segment indication and a tag indication, the target processing is implemented by the accelerator and the processor, and the target processing includes processing of a tag phase performed by the accelerator, and processing of a query phase and an update phase performed by the processor. The target process may further include: the processor takes all the data blocks which are not replaced by the signatures and all replaced signatures in the plurality of data blocks as the data after the deduplication processing.

When the processing indication includes a section indication, a tag indication, and a query indication, the target processing is implemented by the accelerator and the processor, and the target processing includes processing of a tag phase and a query phase performed by the accelerator, and processing of an update phase performed by the processor. The target process may further include: the processor takes all the data blocks which are not replaced by the signatures and all replaced signatures in the plurality of data blocks as the data after the deduplication processing.

When the processing indication includes a segmentation indication, a tag indication, and a query indication, the target processing is implemented by the accelerator and the processor, and the target processing includes processing of a tag phase, a query phase, and an update phase performed by the accelerator. The target process may further include: the processor takes all the data blocks which are not replaced by the signatures and all replaced signatures in the plurality of data blocks as the data after the deduplication processing.

To sum up, in the data processing method provided in the embodiment of the present application, when the processor needs to perform deduplication processing on data to be processed, a processing instruction may be sent to the accelerator. Also, the accelerator may perform at least some of the stages in the deduplication process, regardless of the implementation in which the process dictates. Therefore, the processing required to be executed by the processor is reduced, the load of the processor is reduced, and the data processing efficiency of the processor is improved.

Optionally, in the process of performing deduplication processing on each piece of data to be processed, the processor may generate a task number of the deduplication processing. And the task numbers of the deduplication processing of different data to be processed are different. The processing indication and the processing response may each further indicate: task number of the deduplication process. Therefore, whether the deduplication processing of each data to be processed is finished or not can be accurately determined between the accelerator and the processor through the task number of the deduplication processing.

Further, the accelerator may include a processing unit, and the processing of the data to be processed by the accelerator in step 603 may be performed by the processing unit in the accelerator. For example, the processing unit may include a plurality of processing channels, and in this case, the accelerator may adopt any one of the following processing manners in the process of processing the data to be processed in step 603.

Illustratively, the several processing manners may include a first processing manner, a second processing manner, and a third processing manner. Wherein A1 is less than A2, A1 is less than A3; ax represents the number of processing channels used to perform processing using the x-th processing mode, and x is greater than or equal to 1 and less than or equal to 3. That is, the number of processing channels used for performing the processing in the first processing mode is smaller than the number of processing channels used for performing the processing in the second processing mode, and the number of processing channels used for performing the processing in the first processing mode is smaller than the number of processing channels used for performing the processing in the third processing mode. The number of processing channels employed by the first processing mode is the smallest.

B1 is more than or equal to B3, B2 is more than or equal to B3, and Bx represents: and the number of the residual data segments in the data obtained by the x-th processing mode is adopted. That is, the number of the remaining data segments in the data obtained by the first processing method is greater than or equal to the number of the remaining data segments in the data obtained by the third processing method, and the number of the remaining data segments in the data obtained by the second processing method is greater than or equal to the number of the remaining data segments in the data obtained by the third processing method. The third processing mode can effectively process the data to be processed so as to minimize the number of residual data segments in the obtained data.

These three processing manners will be exemplified below, respectively, and it should be noted that these three processing manners are not limited to the following examples.

1. The first processing mode: single channel processing mode.

When the accelerator processes the data to be processed in a single-channel processing manner, the accelerator may first determine any idle processing channel in the multiple processing channels in step 603, and then process the data to be processed through the any idle processing channel. The idle processing channel is a processing channel that does not process data in the plurality of processing channels.

In the single-channel processing mode, each piece of data to be processed is processed through one processing channel. In a single channel processing mode, the accelerator may process the obtained data, including: and when the data to be processed comprises the residual data segment except the data block, the data processed by the accelerator also comprises the residual data segment.

For example, as shown in fig. 7, it is assumed that the accelerator needs to process data to be processed 1, data to be processed 2, and data to be processed 3, respectively, and the processing unit of the accelerator includes: the accelerator can process the data to be processed 1 through the processing channel 1, process the data to be processed 2 through the processing channel 2, and process the data to be processed 3 through the channel processing channel 3. The processed data output by the processing channel 1 includes: data block 1.1, data block 1.2, data block 1.3 and remaining data segment 1. The processed data output by the processing channel 2 includes: data block 2.1, data block 2.2 and remaining data segment 2. The processed data output by the processing channel 3 includes: data block 3.1, data block 3.2, data block 3.3 and remaining data segment 3.

2. The second processing mode is as follows: and (4) a multi-channel rapid processing mode.

When the accelerator processes the data to be processed in a multi-channel fast processing manner, the accelerator may first divide the data to be processed into at least two consecutive data segments in step 603; then, the accelerator can process the at least two data segments in a one-to-one correspondence through at least two idle processing channels of the plurality of processing channels. The idle processing channel is a processing channel that does not process data in the plurality of processing channels.

The step of the accelerator dividing the data to be processed into at least two consecutive data segments may be performed by a receiving buffer in the accelerator, or the receiving buffer may be performed under the control of the controller.

The number of the data segments in the at least two data segments may be a default number, and when the accelerator processes each data to be processed, the accelerator divides the data to be processed into the data segments of the default number. Or, the number of the data segments in the at least two data segments may be a random number, and when the accelerator processes each to-be-processed data, the accelerator may temporarily determine the number, and further divide the to-be-processed data into at least two data segments based on the number.

In the multi-channel fast processing mode, each data to be processed is processed through a plurality of processing channels. The data obtained by the multi-channel fast processing mode can be the same as the data obtained by the single-channel processing mode. For example, in the multi-channel fast processing mode, the processing of the obtained data may include: and when the data to be processed comprises the residual data segment except the data block, the processed data also comprises the residual data segment.

For example, as shown in fig. 8, it is assumed that the accelerator needs to process data to be processed, and the processing unit of the accelerator includes: processing channel 1, processing channel 2, and processing channel 3, the accelerator may first divide the data to be processed into data segment 1, data segment 2, and data segment 3. The accelerator may then process data segment 1 via processing channel 1, data segment 2 via processing channel 2, and data segment 3 via channel processing channel 3. The processed data output by the processing channel 1 includes: data block 1.11, data block 1.22, data block 1.33 and remaining data segment 1.1. The processed data output by the processing channel 2 includes: data block 2.11, data block 2.22 and remaining data segment 2.2. The processed data output by the processing channel 3 includes: data block 3.11, data block 3.22, data block 3.33 and remaining data section 3.3.

In the multi-channel fast processing mode, the accelerator processes the same data to be processed through a plurality of processing channels, so that the accelerator has a higher speed for processing the data to be processed. When the data to be processed is the data to be transmitted, the data to be processed is processed quickly, so that the data to be processed is transmitted quickly, and the transmission delay of the data to be processed is reduced.

3. The third processing mode: and (4) a multi-channel compression processing mode.

It should be noted that the data to be processed may belong to a data set, that is, the data to be processed is data determined by the processor from the data set. The data set may be some data to be stored, or a data stream to be forwarded, etc. The segment indication in the processing indication issued by the processor in step 601 includes: the first data indication is used for indicating whether the data to be processed contains the first data in the data set to which the data to be processed belongs. The characteristic data indication may be a start marker called a new partition. For example, as shown in fig. 9, the processor may take out the to-be-processed data 1 and the to-be-processed data 2 from the data set, where the to-be-processed data 1 includes the first data in the data set, and the to-be-processed data 2 does not include the first data in the data set, so that the segment indication of the to-be-processed data 1 issued to the accelerator by the processor may carry a first data indication for indicating that the to-be-processed data 1 includes the first data in the data set, and the segment indication of the to-be-processed data 2 issued to the accelerator by the processor may carry a first data indication for indicating that the to-be-processed data 2 does not include the first data in the data set.

When the accelerator processes the data to be processed in a multi-channel compression processing manner, the accelerator may first determine whether the data to be processed meets a target condition in step 603. Wherein the target conditions include: the first data indication in the processing indication indicates that the data to be processed does not include the first data, and the auxiliary data in the data set is processed before the data to be processed (such as the processing required by the accelerator in step 603), and the remaining data segment except the data block exists in the auxiliary data. For example, if when the data to be processed 2 is processed, the first data indication in the processing indication of the data to be processed 2 is used to indicate that the data to be processed 2 does not include the first data, and before the data to be processed 2 is processed, the accelerator further processes the data to be processed 1 (auxiliary data), and there is a remaining data segment in the data to be processed 2, the accelerator may determine that the data to be processed 2 satisfies the target condition.

On one hand, when the data to be processed meets the target condition, the accelerator may update the data to be processed first, and the updated data to be processed includes: the data to be processed before updating, the first remaining data segment in the auxiliary data, and all data in the auxiliary data that is located after the first remaining data segment. After obtaining the updated to-be-processed data, the accelerator may divide the updated to-be-processed data into n consecutive first data segments, and process the n first data segments in a one-to-one correspondence manner through n idle processing channels of the multiple processing channels. Wherein the auxiliary data comprises n remaining data segments, n > 1; the jth first data segment includes: j is more than or equal to 1 and less than n in the jth residual data segment in the auxiliary data and the data block between the jth residual data segment and the jth +1 residual data segment; the nth first data segment includes: the nth remaining data segment in the auxiliary data and the data to be processed. When the updated data to be processed has the residual data segment, the data obtained by processing comprises: and updating the data block before the first residual data segment in the data to be processed. When the updated data to be processed does not have the residual data segment, the data obtained by processing comprises the following steps: all data blocks in the data to be processed.

On the other hand, when the data to be processed does not meet the target condition, the accelerator divides the data to be processed into at least two continuous second data segments, and then processes the at least two second data segments in a one-to-one correspondence manner through at least two idle processing channels in the plurality of processing channels. When the remaining data segments exist in the data to be processed, the data obtained by processing comprises the following steps: a data block preceding a first remaining data segment in the data to be processed; when the remaining data segment does not exist in the data to be processed, the data obtained by processing comprises the following steps: all data blocks in the data to be processed. It can be seen that, when the data to be processed does not meet the target condition, the processing procedure of the accelerator for the data to be processed by adopting the multi-channel compression processing mode is similar to the processing procedure for the data to be processed by adopting the multi-channel rapid mode, but the output processed data is different.

The multi-channel compression processing mode will be explained below by taking the data to be processed 1 and the data to be processed 2 in fig. 9 as an example.

When the accelerator processes the data to be processed 1, the data to be processed 1 contains the first data in the data set to which the accelerator belongs, and therefore the data to be processed 1 does not meet the target condition. At this time, as shown in fig. 10, the accelerator may divide the data to be processed 1 into three consecutive second data segments (including the second data segment 1, the second data segment 2, and the second data segment 3), and then process the three second data segments through three idle processing channels in a one-to-one correspondence manner. For example, as shown in fig. 10, the second data segment 1 is processed by the idle processing channel 1, the second data segment 2 is processed by the idle processing channel 2, and the second data segment 3 is processed by the idle processing channel 3. As shown in fig. 10, the processed data output by the idle processing channel 1 includes: data block 1.11, data block 1.22, data block 1.33 and remaining data segment 1.1. The processed data output by the idle processing channel 2 includes: data block 2.11, data block 2.22 and remaining data segment 2.2. The processed data output by the idle processing channel 3 includes: data block 3.11, data block 3.22, data block 3.33 and remaining data section 3.3. However, as shown in fig. 10, the processed data output by the accelerator includes only: data block 1.11, data block 1.22, and data block 1.33.

When the accelerator processes the data to be processed 2, the first data indication in the processing indication of the data to be processed 2 is used to indicate that the data to be processed 2 does not include the first data, and before the data to be processed 2 is processed, the accelerator further processes the data to be processed 1 (auxiliary data), and a remaining data segment exists in the data to be processed 2, then the accelerator may determine that the data to be processed 2 satisfies the target condition. At this point, the accelerator may first update the data to be processed 2. For example, as shown in fig. 10, the accelerator updated pending data 2 includes: data to be processed 2, remaining data segment 1.1, data block 2.11, data block 2.22, remaining data segment 2.2, data block 3.11, data block 3.22, data block 3.33, and remaining data segment 3.3.

Since the to-be-processed data 1 includes 3 remaining data segments, after the accelerator obtains the updated to-be-processed data 2, the accelerator can divide the updated to-be-processed data 2 into 3 consecutive first data segments, and process the 3 first data segments in a one-to-one correspondence manner through 3 idle processing channels of the multiple processing channels. For example, the first data segment 1 is processed through the idle processing channel 1, the first data segment 2 is processed through the idle processing channel 2, and the first data segment 3 is processed through the idle processing channel 3. Wherein the first data segment 1 comprises: the 1 st remaining data segment (i.e. the remaining data segment 1.1) in the data to be processed 1, and the data block (such as the data block 2.11 and the data block 2.22) between the 1 st remaining data segment and the 2 nd remaining data segment; the first data segment 2 includes: the 2 nd remaining data segment (i.e. the remaining data segment 2.2) in the data 1 to be processed, and the data blocks (such as the data block 3.11, the data block 3.22, and the data block 3.33) between the 2 nd remaining data segment and the 3 rd remaining data segment. The first data segment 3 includes: the 3 rd remaining data segment (remaining data segment 3.3) and the data to be processed 2 in the data to be processed 1. With continued reference to fig. 10, the processed data output by the idle processing channel 1 includes: data block 11, data block 12, and data block 13; the processed data output by the idle processing channel 2 includes: data block 21, data block 22, data block 23, and data block 24; the processed data output by the idle processing channel 3 includes: data block 31, data block 32 and remaining data segment 33.

Since the remaining data segment 33 exists in the updated to-be-processed data 2, as shown in fig. 10, the processed data output by the accelerator includes: the data blocks (including data block 11, data block 22, data block 33, data block 21, data block 22, data block 23, data block 24, data block 31, and data block 32) before the first remaining data segment in the updated data to be processed 2.

The processing instruction includes the above-mentioned first data instruction, and optionally, the processing instruction may further include: a tail data indication indicating whether the data to be processed includes the last data (also called tail data) in the data set to which the data belongs. The tail data indication may also be referred to as an end marker of the segment.

It should be noted that, in the process of processing data by each processing channel in the accelerator, reference may be made to the process of processing the data to be processed by the accelerator in step 603, which is not described herein again in this embodiment of the present application.

It should be noted that the accelerator may process the data to be processed by default using any one of a plurality of processing methods (e.g., the three processing methods described above). Of course, the accelerator may also select one processing mode from multiple processing modes (such as the above three processing modes) according to the instruction of the processor, and process the data to be processed. At this time, before step 603, the processor may also send a mode setting instruction to the accelerator (e.g., before step 601, the processor may send the mode setting instruction to the accelerator). Wherein the mode setting instruction is used for indicating: and a target processing mode for processing the data to be processed. The target processing method is any one of the plurality of processing methods. In step 603, the accelerator may process the data to be processed in the target processing manner based on the processing instruction.

Further, the accelerator needs to determine a target processing manner among the plurality of processing manners before sending the manner setting instruction to the processor. For example, the processor may determine the target treatment based on some indication. These indications may be user-triggered indications or indications issued by other devices besides the data processing apparatus. Illustratively, the indications may include a first indication and a second indication. The first indication is used for indicating whether the data to be processed needs priority processing, and the second indication is used for indicating whether the data to be processed needs rapid processing.

As shown in fig. 11, the process of the processor determining the target processing manner based on the indications may include:

step 1001, the processor detects whether the first indication is used for indicating that the data to be processed needs priority processing. If the first indication is used to indicate that the data to be processed needs priority processing, step 1002 is executed; if the first indication is used to indicate that the pending data does not require priority processing, step 1003 is executed.

Step 1002, the processor detects whether the second indication is used for indicating that the data to be processed needs to be processed quickly. If the second indication is used to indicate that the data to be processed needs to be processed quickly, then step 1004 is executed; if the second indication indicates that the data to be processed does not require fast processing, step 1005 is executed.

Step 1003, the processor detects whether the number of idle processing channels in the accelerator is greater than 1. If the number of idle processing channels is equal to 1, go to step 1006; if the number of idle processing channels is greater than 1, go to step 1002.

Step 1004, the processor determines that the second processing mode is the target processing mode.

It can be seen that, when the first indication is used to indicate that the to-be-processed data needs to be processed preferentially, if the second indication is used to indicate that the to-be-processed data needs to be processed rapidly, the processor determines that the target processing manner is the second processing manner. When the first indication is used for indicating that the data to be processed does not need to be processed preferentially, if the number of idle processing channels in the processing unit in the accelerator is greater than 1, and the second indication is used for indicating that the data to be processed needs to be processed rapidly, the processor determines that the target processing mode is the second processing mode.

Step 1004, the processor determines that the third processing mode is the target processing mode.

It can be seen that, when the first indication is used to indicate that the to-be-processed data needs to be processed preferentially, if the second indication is used to indicate that the to-be-processed data does not need to be processed rapidly, the processor determines that the target processing mode is the third processing mode; when the first indication is used for indicating that the data to be processed does not need to be processed preferentially, if the number of idle processing channels in the processing unit in the accelerator is greater than 1, and the second indication is used for indicating that the data to be processed does not need to be processed rapidly, the processor determines that the target processing mode is a third processing mode.

Step 1006, the processor determines that the first processing mode is a target processing mode.

It can be seen that, when the first indication is used to indicate that the to-be-processed data does not need to be processed preferentially, if the number of idle processing channels in the accelerator is equal to 1, the processor determines that the target processing manner is the first processing manner.

In the above embodiments, the accelerator is taken as an example that the accelerator can directly process the whole data to be processed, and optionally, the accelerator may also divide the data to be processed into multiple parts of data, process each part of data, and combine the processed multiple parts of data to obtain the whole data processed by the accelerator. The process of processing each part of data by the accelerator may refer to the process of processing the whole data to be processed by the accelerator, which is not described herein in this embodiment of the present application.

Based on the foregoing embodiments of the data processing method, it can be seen that, in the data processing apparatus including the processor and the accelerator provided in the embodiments of the present application, the processor and the accelerator have the following functions:

the processor is configured to send a processing indication to the accelerator, the processing indication including: a segment indication, and first storage information; the accelerator is used for acquiring data to be processed based on the first storage information; the accelerator is used for carrying out segmentation processing on the data to be processed based on the segmentation indication to obtain a plurality of data blocks; and the data processing device is used for carrying out target processing on the plurality of data blocks to obtain data after the duplicate removal processing.

Optionally, the processor is configured to calculate a corresponding signature for each of the plurality of data blocks; the processor is used for inquiring a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block of the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor is configured to replace the first data block with the first signature; when a second data block of the plurality of data blocks is not in the dictionary, the processor is configured to store a corresponding relationship between a signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is the data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed.

Optionally, the processor is configured to receive a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete the segmented processing on the data to be processed. Illustratively, the processing response may include second storage information for the plurality of data blocks. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the processing instructions further include: a signature indication; the accelerator is configured to calculate, for each of the plurality of data chunks, a corresponding signature based on the signature indications; the processor is used for inquiring a dictionary according to the data blocks, wherein the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block of the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor is configured to replace the first data block with the first signature; when a second data block of the plurality of data blocks is not in the dictionary, the processor is configured to store a corresponding relationship between a signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is the data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed.

Optionally, the processor is configured to receive a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete the segmentation processing on the data to be processed, and to complete the calculation of the corresponding signature for each data block. For example, the processing response may include a plurality of data blocks and signed second storage information corresponding to the plurality of data blocks. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the accelerator comprises: the device comprises a controller, a receiving cache, a sending cache and a processing unit; the controller is used for sending the processing instruction to the processor; the controller is used for controlling the receiving cache to acquire the data to be processed based on the first storage information; the controller is used for controlling the processing unit to acquire the data to be processed in the receiving cache and performing the segmentation processing on the data to be processed based on the segmentation indication; the processing unit is used for sending the plurality of data blocks and the signature corresponding to each data block to the sending cache; the processing unit is used for sending a finish instruction to the controller; the controller is configured to control the sending cache to store the plurality of data blocks and the signature corresponding to each data block based on the finalization indication; the processor is configured to receive the processing response from the controller.

Optionally, the processing instructions further include: a signature indication and a query indication; the accelerator is configured to calculate, for each of the plurality of data chunks, a corresponding signature based on the signature indications; the accelerator is used for querying a dictionary based on the query indication and the data blocks, and the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block of the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor is configured to replace the first data block with the first signature; when a second data block of the plurality of data blocks is not in the dictionary, the processor is configured to store a corresponding relationship between a signature of the second data block and the second data block in the dictionary; the data after the deduplication processing is as follows: and all data blocks which are not replaced by the signatures and all replaced signatures in the data to be processed.

Optionally, the processor is configured to receive a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete processing of the data to be processed. The processing comprises the following steps: the segmentation process described above, as well as computing a corresponding signature for each data block based on the previous indication, and querying the dictionary based on the plurality of data blocks. For example, the processing response may include a plurality of data blocks, signatures corresponding to the plurality of data blocks, and second storage information of whether the result of each data block is recorded in the dictionary. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the processing instructions further include: a signature indication, a query indication, and an update indication; the accelerator is configured to calculate, for each of the plurality of data chunks, a corresponding signature based on the signature indications; the accelerator is used for querying a dictionary based on the query indication and the plurality of data blocks, and the dictionary comprises the corresponding relation between the data blocks and the signatures; when a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the accelerator is used for replacing the first data block in the data to be processed with the first signature according to the updating indication; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator is used for storing the corresponding relation between the signature of the second data block and the second data block into the dictionary; the processor is configured to use the data processed by the accelerator as the data after the deduplication processing, where the data processed by the accelerator is all the data blocks that are not replaced with the signature and all the replaced signatures in the data to be processed.

Optionally, the processor is configured to receive a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete processing of the data to be processed. The processing comprises the following steps: the segmentation processing includes calculating a corresponding signature for each data block based on the previous indication, querying a dictionary based on a plurality of data blocks, and replacing a first data block in the data to be processed with the first signature by the accelerator according to the update indication when the first data block in the plurality of data blocks and the first signature of the first data block are in the dictionary; when a second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the corresponding relation between the signature of the second data block and the second data block in the dictionary. For example, the processing response may include all data blocks in the data to be processed that are not replaced by the signature and all second stored information of the replaced signature. It can be seen that after the processor issues the processing instruction to the accelerator, the processor does not need to continuously detect whether the accelerator performs the finish processing, but waits for the processing response sent by the accelerator after performing the finish processing, thereby further reducing the load of the processor.

Optionally, the processor is configured to receive a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete processing of the data to be processed, and the processing response includes storage information of the data processed by the accelerator; and the processor is used for carrying out target processing on the data obtained by the processing of the accelerator based on the storage information to obtain the data after the deduplication processing.

Optionally, the accelerator comprises: the device comprises a controller, a receiving cache, a sending cache and a processing unit; the processor is used for sending the processing instruction to the controller; the controller is used for controlling the receiving cache to acquire the data to be processed based on the first storage information; the controller is used for controlling the processing unit to acquire the data to be processed in the receiving cache and performing the segmentation processing on the data to be processed based on the segmentation indication; the processing unit is used for sending the processed data to the sending cache; the processing unit is used for sending a finish instruction to the controller; the controller is used for controlling the sending cache to store the data obtained by the processing unit based on the finishing indication; the processor is configured to receive the processing response from the controller.

Optionally, the accelerator comprises: a plurality of processing channels, the accelerator to determine any idle processing channel of the plurality of processing channels; the accelerator is used for carrying out the segmented processing on the data to be processed through any idle processing channel.

Optionally, the accelerator comprises: the accelerator is used for dividing the data to be processed into at least two continuous data segments; the accelerator is configured to perform the segmentation processing on the at least two data segments in a one-to-one correspondence through at least two idle processing channels of the multiple processing channels.

Optionally, the segment indication comprises: the initial data indication is used for indicating whether the data to be processed contains initial data in the data set to which the data to be processed belongs, and the accelerator is used for updating the data to be processed when the data to be processed meets a target condition; wherein the target conditions include: the first data indication is used for indicating that the data to be processed does not contain the first data, and auxiliary data in the data set is subjected to the segmentation processing before the segmentation processing is carried out on the data to be processed, and residual data segments except data blocks exist in the auxiliary data; the updated data to be processed comprises: the data to be processed before updating, a first remaining data segment in the auxiliary data, and all data located after the first remaining data segment in the auxiliary data.

Optionally, the accelerator comprises a plurality of processing channels, the helper data comprises n remaining data segments, n > 1; the accelerator is configured to divide the updated to-be-processed data into n consecutive first data segments, where a jth first data segment includes: j is more than or equal to 1 and less than n in the jth residual data segment in the auxiliary data and the data block between the jth residual data segment and the j +1 th residual data segment; the nth first data segment includes: the nth residual data segment in the auxiliary data and the data to be processed; the accelerator is configured to perform the segmentation processing on the n first data segments in a one-to-one correspondence through n idle processing channels of the multiple processing channels.

Optionally, when the data to be processed does not satisfy the target condition, the accelerator is configured to divide the data to be processed into at least two consecutive second data segments; the accelerator is configured to perform the segmentation processing on the at least two second data segments in a one-to-one correspondence through at least two idle processing channels of the multiple processing channels.

Optionally, the accelerator and the CPU are both connected to a memory, the first storage information is storage information of the data to be processed in the memory, and the second storage information is storage information of the data obtained by processing by the accelerator in the memory.

Optionally, the processing indication and the processing response are each further configured to indicate: and the task number of the deduplication processing.

Optionally, the processor is configured to send, to the accelerator, a mode setting instruction before the accelerator performs the segmentation processing on the to-be-processed data based on the segmentation indication, where the mode setting instruction is configured to indicate: a target processing mode of the plurality of processing modes; the accelerator is used for processing the data to be processed by adopting the target processing mode based on the processing instruction.

Optionally, the processor is configured to:

acquiring a first indication and a second indication aiming at the data to be processed, wherein the first indication is used for indicating whether the data to be processed needs priority processing, and the second indication is used for indicating whether the data to be processed needs rapid processing;

when the first indication is used for indicating that the to-be-processed data does not need priority processing, if the number of idle processing channels in the accelerator is equal to 1, determining that the target processing mode is a first processing mode, wherein the processing unit comprises a plurality of processing channels;

when the first indication is used for indicating that the to-be-processed data does not need to be processed preferentially, if the number of idle processing channels in a processing unit in the accelerator is greater than 1 and the second indication is used for indicating that the to-be-processed data needs to be processed rapidly, determining that the target processing mode is a second processing mode;

when the first indication is used for indicating that the to-be-processed data does not need to be processed preferentially, if the number of idle processing channels in a processing unit in the accelerator is greater than 1 and the second indication is used for indicating that the to-be-processed data does not need to be processed rapidly, determining that the target processing mode is a third processing mode;

when the first indication is used for indicating that the data to be processed needs to be processed preferentially, if the second indication is used for indicating that the data to be processed needs to be processed rapidly, determining that the target processing mode is the second processing mode;

when the first indication is used for indicating that the to-be-processed data needs to be processed preferentially, if the second indication is used for indicating that the to-be-processed data does not need to be processed rapidly, determining that the target processing mode is the third processing mode;

wherein A1 is more than A2, A1 is more than A3, B1 is more than or equal to B3, and B2 is more than or equal to B3; ax represents the number of the processing channels used for executing the processing by the x-th processing mode, x is more than or equal to 1 and less than or equal to 3, and Bx represents: and processing the number of the residual data segments in the obtained data by adopting an x-th processing mode.

To sum up, in the data processing apparatus provided in the embodiment of the present application, when the processor needs to perform deduplication processing on data to be processed, the processor may send a processing instruction to the accelerator. Also, the accelerator may perform at least some of the stages in the deduplication process, regardless of the implementation in which the process dictates. Therefore, the processing required to be executed by the processor is reduced, the load of the processor is reduced, and the data processing efficiency of the processor is improved.

The accelerator or processor in the data processing apparatus may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product comprising one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., solid state disk), among others.

The present application also provides a chip, where the chip includes a programmable logic circuit and/or a program instruction, and when the chip runs, the chip is configured to implement the data processing method provided in the present application, or a method executed by an accelerator in the data processing method, or a method executed by a processor in the data processing method.

It should be noted that, the method embodiment provided in the embodiment of the present invention can be mutually referred to a corresponding apparatus embodiment, and the embodiment of the present invention does not limit this. The sequence of the steps of the method embodiments provided by the embodiments of the present invention can be appropriately adjusted, and the steps can be correspondingly increased or decreased according to the situation, and any method that can be easily conceived by those skilled in the art within the technical scope disclosed by the present invention shall be covered by the protection scope of the present invention, and therefore, the detailed description thereof shall not be repeated.

The above description is only an example of the present application and should not be taken as limiting the present application, and any modifications, equivalents, improvements and the like that are made within the principles of the present application should be included in the scope of the present application.

Claims

1. a data processing method, is characterized in that, for comprising the data processing device of processor and accelerator, described method comprises:

The processor sends a processing instruction to the accelerator, where the processing instruction includes: a segmentation instruction and first storage information;

The accelerator obtains the data to be processed based on the first storage information;

The accelerator performs segmentation processing on the to-be-processed data based on the segmentation instruction to obtain a plurality of data blocks;

The data processing apparatus performs target processing on the plurality of data blocks to obtain deduplicated data.

2. The method according to claim 1, wherein the data processing device performs target processing on the plurality of data blocks to obtain deduplicated processed data, comprising:

The processor calculates a corresponding signature for each data block in the plurality of data blocks;

The processor queries a dictionary according to the plurality of data blocks, and the dictionary includes the correspondence between the data blocks and the signature;

When the first data block in the plurality of data blocks and the first signature of the first data block are in the dictionary, the processor replaces the first data block with the first signature;

When the second data block in the plurality of data blocks is not in the dictionary, the processor stores the correspondence between the signature of the second data block and the second data block in the dictionary;

The deduplicated data is all data blocks that are not replaced with signatures and all replaced signatures in the data to be processed.

3. The method according to claim 1, wherein the processing instruction further comprises: a signature instruction;

The data processing apparatus performs target processing on the plurality of data blocks to obtain deduplicated data, including:

The accelerator calculates a corresponding signature for each data block in the plurality of data blocks based on the signature indication;

4. The method according to claim 3, wherein before the processor queries a dictionary according to the plurality of data blocks, the method further comprises:

The processor receives a processing response from the accelerator, where the processing response is used to instruct the accelerator to complete segment processing of the data to be processed and to complete the calculation of a corresponding signature for each data block.

5. The method according to claim 4, wherein the accelerator comprises: a controller, a receive buffer, a transmit buffer, and a processing unit;

The processor sending a processing instruction to the accelerator includes: the processor sending the processing instruction to the controller;

The accelerator acquiring the data to be processed based on the first storage information includes: the controller controlling the receiving buffer to acquire the data to be processed based on the first storage information;

The accelerator performs segment processing on the to-be-processed data based on the segment instruction, including: the controller controls the processing unit to acquire the to-be-processed data in the receive buffer, and based on the segment instructing to perform the segment processing on the data to be processed;

The method also includes:

The processing unit sends the multiple data blocks and the signature corresponding to each of the data blocks to the sending buffer;

sending, by the processing unit, a completion instruction to the controller;

The controller, based on the completion instruction, controls the sending cache to store the multiple data blocks and the signature corresponding to each of the data blocks;

The processor receiving the processing response from the accelerator includes the processor receiving the processing response from the controller.

6. The method according to claim 1, wherein the processing instruction further comprises: a signature instruction and a query instruction;

The accelerator queries a dictionary based on the query instruction and the plurality of data blocks, where the dictionary includes a correspondence between data blocks and signatures;

The data after deduplication processing is: all data blocks that are not replaced with signatures and all replaced signatures in the data to be processed.

7. The method according to claim 1, wherein the processing instruction further comprises: a signature instruction, a query instruction and an update instruction;

The accelerator queries a dictionary based on the query instruction and the plurality of data blocks, where the dictionary includes the correspondence between the data blocks and the signature;

When the first data block in the plurality of data blocks and the first signature of the first data block are in the dictionary, the accelerator changes the first data block in the data to be processed according to the update instruction A data block is replaced with the first signature;

When the second data block in the plurality of data blocks is not in the dictionary, the accelerator stores the correspondence between the signature of the second data block and the second data block in the dictionary;

The processor uses the data processed by the accelerator as the data after the deduplication processing, and the data processed by the accelerator is all the data blocks that have not been replaced with signatures and all the replaced signatures in the data to be processed. .

8. The method according to any one of claims 1 to 7, wherein the accelerator comprises: a plurality of processing channels, and the accelerator performs segment processing on the data to be processed based on the segment indication, comprising: :

The accelerator determines any idle processing channel in the plurality of processing channels;

The accelerator performs the segment processing on the data to be processed through the any idle processing channel.

9. The method according to any one of claims 1 to 7, wherein the accelerator comprises: a plurality of processing channels, and the accelerator performs segment processing on the data to be processed based on the segment indication, comprising: :

The accelerator divides the data to be processed into at least two consecutive data segments;

The accelerator performs the segment processing on the at least two data segments in a one-to-one correspondence through at least two idle processing channels in the plurality of processing channels.

10. The method according to any one of claims 1 to 7, wherein the segment indication comprises: a first data indication, wherein the first data indication is used to indicate whether the data to be processed includes the data in the data set to which it belongs. For the first data, before the accelerator performs segmentation processing on the data to be processed based on the segmentation instruction, the method further includes:

When the data to be processed satisfies the target condition, the accelerator updates the data to be processed;

Wherein, the target condition includes: the first data indication is used to indicate that the data to be processed does not include the first data, and before the segmentation processing is performed on the data to be processed, The auxiliary data has undergone the segmentation process, and there are remaining data segments other than data blocks in the auxiliary data; the data to be processed after the update includes: the data to be processed before the update, the auxiliary data The first remaining data segment in the data, and all data in the auxiliary data after the first remaining data segment.

11. The method according to claim 10, wherein the accelerator comprises a plurality of processing channels, the auxiliary data comprises n remaining data segments, n>1; The data to be processed is segmented, including:

The accelerator divides the updated data to be processed into consecutive n first data segments, wherein the jth first data segment includes: the jth remaining data segment in the auxiliary data, and the jth remaining data segment in the auxiliary data, and the The data block between the j remaining data segments and the j+1-th remaining data segment, 1≤j<n; the n-th first data segment includes: the n-th remaining data segment in the auxiliary data and the to-be-to-be-data segment. Data processing;

The accelerator performs the segment processing on the n first data segments in a one-to-one correspondence through n idle processing channels in the plurality of processing channels.

12. The method according to claim 10 or 11, wherein the accelerator performs segment processing on the data to be processed based on the segment indication, comprising:

When the data to be processed does not meet the target condition, the accelerator divides the data to be processed into at least two consecutive second data segments;

The accelerator performs the segment processing on the at least two second data segments in a one-to-one correspondence through at least two idle processing channels in the plurality of processing channels.

13. A data processing device, comprising a processor and an accelerator;

The processor is configured to send a processing instruction to the accelerator, where the processing instruction includes: a segmentation instruction and first storage information;

The accelerator is configured to acquire data to be processed based on the first storage information;

The accelerator is configured to perform segmentation processing on the to-be-processed data based on the segmentation instruction to obtain a plurality of data blocks;

The data processing device is configured to perform target processing on the plurality of data blocks to obtain deduplicated processed data.

14. The data processing apparatus according to claim 13, characterized in that:

The processor is configured to calculate a corresponding signature for each data block in the plurality of data blocks;

The processor is configured to query a dictionary according to the plurality of data blocks, where the dictionary includes the correspondence between the data blocks and the signature;

When a first data block in the plurality of data blocks and a first signature of the first data block are in the dictionary, the processor is configured to replace the first data block with the first signature ;

When the second data block in the plurality of data blocks is not in the dictionary, the processor is configured to store the correspondence between the signature of the second data block and the second data block into the dictionary ;

15. The data processing apparatus according to claim 13, wherein the processing instruction further comprises: a signature instruction;

The accelerator is configured to calculate a corresponding signature for each data block in the plurality of data blocks based on the signature indication;

16. The data processing apparatus according to claim 15, wherein the processor is configured to receive a processing response from the accelerator, the processing response being used to instruct the accelerator to complete the processing of the plurality of data blocks deal with.

17. The data processing apparatus according to claim 13, wherein the processing instruction further comprises: a signature instruction and a query instruction;

The accelerator is configured to query a dictionary based on the query instruction and the plurality of data blocks, where the dictionary includes a correspondence between data blocks and signatures;

18. The data processing apparatus according to claims 13 to 17, wherein the accelerator comprises: a plurality of processing channels; the accelerator is used for:

determining any idle processing channel in the plurality of processing channels;

The segment processing is performed on the data to be processed through the any idle processing channel.

19. The data processing apparatus according to any one of claims 13 to 17, wherein the accelerator comprises: a plurality of processing channels; the accelerator is used for:

dividing the data to be processed into at least two consecutive data segments;

The segment processing is performed on the at least two data segments in a one-to-one correspondence through at least two idle processing channels in the plurality of processing channels.

20. The data processing apparatus according to any one of claims 13 to 17, wherein the segment indication comprises: a first data indication, wherein the first data indication is used to indicate whether the data to be processed includes the data to which it belongs The first data in the collection;

The accelerator is configured to update the data to be processed when the data to be processed meets a target condition;

21. A chip, characterized in that, the chip comprises programmable logic circuits and/or program instructions, which are used to implement the data processing method according to any one of claims 1 to 12 when the chip is running.