CN106095971B - A kind of method and server for searching data flow cut-point based on server - Google Patents

Abstract

The embodiment of the present invention provides a server-based method for searching data stream segmentation points. In the embodiment of the present invention, the data flow segmentation point is searched by judging whether at least part of the data in one of the M windows satisfies the predetermined condition, and when at least part of the data in a certain window does not meet the predetermined condition, skip N*U length, to obtain the next potential split point, and improve the efficiency of searching for the split point of the data stream.

Description

A method and server for searching data flow split point based on server

technical field

The invention relates to the field of information technology, in particular to a server-based method for searching data stream segmentation points and a server.

Background technique

The continuous growth of data volume makes providing sufficient data storage a serious challenge in the current storage field. One way to address this challenge is to take advantage of the redundant nature of the data that needs to be stored and use de-duplication technology to reduce the amount of stored data.

In the prior art, the data deduplication algorithm based on Content Defined Chunk (CDC) first needs to divide the data stream to be stored into many data blocks, and to divide the data stream into data blocks needs to find the appropriate data stream in the data stream. The data between the split points of two adjacent data streams constitutes a data block. Calculate the characteristic value of the data block to find out whether there is a data block with the same characteristic value. If the data block with the same characteristic value is found, it is considered that there is duplicate data. Specifically, the data deduplication technology based on content partitioning is to apply the sliding window technology (Sliding Window Technique) to find the split point of the block based on the content of the file, that is, to determine the data stream split point by calculating the Rabin fingerprint of the data in the window. Assuming that the segmentation point is found from the left to the right of the data stream, the fingerprint of the data in the sliding window is calculated each time, and the fingerprint value is moduloed based on the given integer K, and compared with the given remainder R; if they are equal, the window The right end of is the split point of the data stream, otherwise, the window will continue to slide one byte to the right, and the calculation and comparison will be performed cyclically until the end of the data stream is reached. In the process of data deduplication based on content partitioning, finding data stream segmentation points consumes a large amount of computing resources, which becomes a bottleneck in improving deduplication performance.

Contents of the invention

In the first aspect, the embodiment of the present invention provides a server-based method for finding data stream segmentation points, where rules are preset on the server, and the rules are: potential segmentation points

k Determine M points p _x , the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x and the predetermined condition corresponding to the window W _x [p _x -A _x ,p _x +B _x ] C _x , wherein, x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers; the method includes:

a) Determine the point p _iz and the window W _iz [p _iz -A _z ,p _iz +B _z ] corresponding to the point p _iz for the current potential segmentation point ki according to the rules, _i and z are integers, and 1 ≤z≤M;

b) judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data stream segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, perform step a);

c) When at least part of the data in each of the M windows W _ix [p _ix -A _x , p _ix +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the first aspect, in a first possible implementation manner, the rule further includes: at least two points pe and p _f satisfying the conditions of A _e =A _f , _Be =B _f , and C _{e =C f .}

In combination with the first possible implementation of the first aspect, in the second possible implementation, the rule further includes: the at least two points p _e and p _f , relative to the potential segmentation point k, at The data stream split point looks in the reverse direction.

In combination with the first possible implementation manner or the second possible implementation manner of the first aspect, in a third possible implementation manner, the rule further includes: the distance between the at least two points p _e and p _f The distance is 1 U.

In combination with the first aspect, or any one of the first to third possible implementations of the first aspect, in the fourth possible implementation, the window W _iz [p _iz -A _z ,p _iz +B _z ] is judged Whether at least part of the data in satisfies the predetermined condition C _z , specifically including:

A random function is used to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In combination with the fourth possible implementation of the first aspect, in the fifth possible implementation, the use of a random function to determine at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] Whether the predetermined condition C _z is satisfied is specifically using a hash function to determine whether at least part of the data in the window W _iz [p _iz -A _z , p _iz +B _z ] satisfies the predetermined condition C _z .

In combination with the first aspect, or any of the first to fifth possible implementations of the first aspect, in the sixth possible implementation, when the window W _iz [p _iz -A _z ,p _iz +B _z ] At least part of the data does not meet the predetermined condition C _z , from the point p _iz along the search direction of the data stream segmentation point to jump N data stream segmentation point minimum search unit U, to obtain the new potential segmentation point, according to According to the rule, the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is the same as the window W _iz [p _iz -A _z ,p _iz +B _z ] coincides with the right boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is located within the range of the window W _iz [p _iz -A _z ,p _iz +B _z ]; wherein, the point p _ic determined for the new potential segmentation point is according to the rule, for the new The M points determined by the potential segmentation points are the first points in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the first aspect, in the seventh possible implementation, a random function is used to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies The predetermined condition C _z specifically includes:

Select F bytes in the window W _iz [p _iz -A _z ,p _iz +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In the second aspect, the embodiment of the present invention provides a server-based method for searching for data flow segmentation points. Rules are preset on the server, and the rules are: determine M windows W _x [kA for a potential segmentation point k _x ,k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x ,k+B _x ], wherein x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers;

The methods include:

a) Determine the corresponding window W _iz [k _i -A _z , _ki +B _z ] for the current potential segmentation point ki according to the rules, _i and z are integers, and 1≤z≤M;

b) judging whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip the minimum search unit U of N data stream segmentation points, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), obtain a new potential segmentation point, and perform step a);

c) When at least part of the data in each window W _ix [k _i -A _x , _ki +B _x ] of the M windows of the current potential segmentation point _ki satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the second aspect, in the first possible implementation manner, the rule further includes: at least two windows W _ie [k _i -A _e ,k _i +B _e ] and W _if [k _i -A _f ,k _i +B _f ], satisfying the conditions: |A _e +B _e |=|A _f +B _f |, C _e ＝C _f .

With reference to the first possible implementation manner of the second aspect, in the second possible implementation manner, the rule further includes: A _e and A _f are positive integers.

In combination with the first possible implementation or the second possible implementation of the second aspect, in a third possible implementation, the rule further includes: A _e -1=A _f , _Be +1= B _f .

In combination with the second aspect, or any of the first to third possible implementations of the second aspect, in the fourth possible implementation, it is determined that in the window W _iz [k _i -A _z , _ki +B _z ] Whether at least part of the data meets the predetermined condition C _z , specifically including:

A random function is used to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In combination with the fourth possible implementation of the second aspect, in the fifth possible implementation, the use of a random function to determine at least part of the data in the window W _iz [k _i -A _z , k _i +B _z ] Whether the predetermined condition C _z is satisfied is specifically using a hash function to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In combination with the second aspect, or any of the first to fifth possible implementations of the second aspect, in the sixth possible implementation, when the window W _iz [k _i -A _z , _ki +B _z ] At least part of the data does not satisfy the predetermined condition C _z , jumping from the current potential segmentation point _ki along the data stream segmentation point search direction for N minimum search units U of the data stream segmentation point to obtain the new potential segmentation point , according to the rule, the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is the same as the window W _iz [k _i -A _z ,k _i +B _z ] coincides with the right boundary or the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is located in the window W _iz [k _i - A _z , _ki + B _z ] range; wherein, the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is according to the rule, as The M windows determined by the new potential segmentation point are the first windows in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the second aspect, in the seventh possible implementation, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies The predetermined condition C _z specifically includes:

Select F bytes in the window W _iz [k _i -A _z , _ki +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In a third aspect, an embodiment of the present invention provides a server for finding a data flow split point, the server includes a central processing unit and a main memory, the central processing unit communicates with the main memory, and on the server There are preset rules, which are: determine M points p _x for potential segmentation point k, the window W _x [p _x -A _x ,p _{x +} B _x ] and the window W _x [p _x -A _x ,p _x +B _x ] corresponds to the predetermined condition C _x , wherein x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers;

The main memory is used to store executable instructions, and the central processing unit executes the executable instructions to perform the following steps:

a) Determine the point p _iz and the window W _iz [p _iz -A _z ,p _iz +B _z ] corresponding to the point p _iz for the current potential segmentation point ki according to the rules, _i and z are integers, and 1 ≤z≤M;

b) judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data stream segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, perform step a);

c) When at least part of the data in each of the M windows W _ix [p _ix - A _x , p _ix + B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the third aspect, in the first possible implementation manner, the rule further includes: at least two points pe and p _f satisfy the conditions of A _e =A _f , _Be =B _f , and C _{e =C f .}

With reference to the first possible implementation of the third aspect, in the second possible implementation, the rule further includes: the at least two points p _e and p _f , relative to the potential segmentation point k, at The data stream split point looks in the reverse direction.

With reference to the first possible implementation manner or the second possible implementation manner of the third aspect, in the third possible implementation manner, the rule further includes: the distance between the at least two points p _e and p _f The distance is 1 U.

With reference to the third aspect, or any of the first to third possible implementation manners, in a fourth possible implementation manner, the central processing unit is specifically configured to

A random function is used to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In combination with the fourth possible implementation of the third aspect, in the fifth possible implementation, the central processing unit is specifically configured to use a hash function to determine the window W _iz [p _iz -A _z ,p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

In combination with the third aspect, or any one of the first to fifth possible implementations, in the sixth possible implementation, when at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] If the predetermined condition C _z is not satisfied, the minimum search unit U of N data stream segmentation points is jumped from the point p _iz along the search direction of the data stream segmentation point to obtain the new potential segmentation point. According to the rule, The left boundary of the window W _ic [p _ic -A _c , p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is the same as the window W _iz [p _iz -A _z ,p _iz + The right boundary of B _z ] coincides or the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is located in the window Within the range of W _iz [p _iz -A _z ,p _iz +B _z ]; wherein, the point p _ic determined for the new potential segmentation point is the new potential segmentation point according to the rule The determined M points are the first points in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the third aspect, in the seventh possible implementation, the central processing unit uses a random function to determine whether in the window W _iz [p _iz -A _z ,p _iz +B _z ] Whether at least part of the data satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [p _iz -A _z ,p _iz +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z . In a fourth aspect, an embodiment of the present invention provides a server for finding a data stream split point, the server includes a central processing unit and a main memory, the central processing unit communicates with the main memory, and on the server A rule is preset, and the rule is: determine M windows W _x [kA _x , k+B _x ] and a predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ] for the potential segmentation point k, Wherein, x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers;

The main memory is used to store executable instructions, and the central processing unit executes the executable instructions to perform the following steps:

a) Determine the corresponding window W _iz [k _i -A _z , _ki +B _z ] for the current potential segmentation point ki according to the rules, _i and z are integers, and 1≤z≤M;

b) judging whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip the minimum search unit U of N data stream segmentation points, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), obtain a new potential segmentation point, and perform step a);

c) When at least part of the data in each window W _ix [k _i -A _x , _ki +B _x ] of the M windows of the current potential segmentation point _ki satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the fourth aspect, in the first possible implementation manner, the rule further includes: at least two windows W _ie [k _i -A _e ,k _i +B _e ] and W _if [k _i -A _f ,k _i +B _f ], satisfying the conditions: |A _e +B _e |=|A _f +B _f |, C _e ＝C _f .

With reference to the first possible implementation manner of the fourth aspect, in the second possible implementation manner, the rule further includes: A _e and A _f are positive integers.

In combination with the first possible implementation or the second possible implementation of the fourth aspect, in the third possible implementation, the rule further includes: A _e -1=A _f , _Be +1= B _f .

With reference to the fourth aspect, or any of the first to third possible implementation manners, in a fourth possible implementation manner, the central processing unit is specifically used to

A random function is used to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

With reference to the fourth possible implementation of the fourth aspect, in the fifth possible implementation, the central processing unit is specifically configured to use a hash function to determine the window W _iz [k _i -A _z , k _i +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

With reference to the fourth aspect, or any of the first to fifth possible implementations, in the sixth possible implementation, when at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] If the predetermined condition C _z is not satisfied, jump N minimum search units U of the data stream segmentation point from the current potential segmentation point _ki along the data stream segmentation point search direction to obtain the new potential segmentation point, according to the According to the above rules, the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is the same as the window W _iz [k _i -A _z , _ki +B _z ] or the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is located at the window W _iz [k _i -A _z , k _i +B _z ] range; wherein, the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is based on the rule, for the new The M windows determined by the potential splitting points are the first windows in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the fourth aspect, in the seventh possible implementation, the central processing unit _uses a _{random function} to _{judge the} Whether at least part of the data satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [k _i -A _z , _ki +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In the fifth aspect, the embodiment of the present invention provides a server for searching data stream segmentation points, and rules are preset on the server, and the rules are: determine M points p _x , point The window W _x [p _x -A _x ,p _x +B _x ] corresponding to p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], where x is 1 to M continuous natural numbers, M≥2, A _x and B _x are integers;

The server includes: a processing unit, configured to perform step a):

a) Determine the point p _iz and the window W _iz [p _iz -A _z ,p _iz +B _z ] corresponding to the point p _iz for the current potential segmentation point ki according to the rules, _i and z are integers, and 1 ≤z≤M;

A judging processing unit, configured to judge whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies a predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data flow segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, then the determination unit performing step a) for said new potential segmentation point;

When at least part of the data in each of the M windows W _ix [p _ix -A _x , p _ix +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential segmentation point k _i is the data stream split point.

With reference to the fifth aspect, in the first possible implementation manner, the rule further includes: at least two points pe and p _f satisfy the conditions of A _e =A _f , _Be =B _f , and C _{e =C f .}

With reference to the first possible implementation of the fifth aspect, in the second possible implementation, the rule further includes: the at least two points p _e and p _f , relative to the potential segmentation point k, at The data stream split point looks in the reverse direction.

With reference to the first possible implementation manner or the second possible implementation manner of the fifth aspect, in a third possible implementation manner, the rule further includes: between the at least two points p _e and p _f The distance is 1 U.

With reference to the fifth aspect, or any of the first to third possible implementation manners, in a fourth possible implementation manner, the determination processing unit specifically uses a random function to determine the window W _iz [p _iz -A _z ,p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

With reference to the fourth possible implementation of the fifth aspect, in the fifth possible implementation, the judgment processing unit is specifically configured to use a hash function to judge the window W _iz [p _iz -A _z ,p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

With reference to the fifth aspect, or any of the first to fifth possible implementation manners, in a sixth possible implementation manner, the judgment processing unit is configured to when the window W _iz [p _iz -A _z ,p _iz + At least part of the data in B _z ] does not satisfy the predetermined condition C _z , jump N minimum search units U of data flow segmentation points from the point p _iz along the data flow segmentation point search direction, and obtain the new potential segmentation point, the determination unit performs step a) for the new potential segmentation point, according to the rule, the window W _ic corresponding to the point p _ic determined for the new potential segmentation point [p _ic -A _c ,p _ic +B _c ] coincides with the right boundary of the window W _iz [p _iz -A _z ,p _iz +B _z ] or the window W _ic [p _ic -A _c ,p _ic +B _c ] is located within the window W _iz [p _iz -A _z ,p _iz +B _z ]; The window W _ic [p _ic -A _c ,p _ic +B _c ] is the first point in the sequence obtained according to the search direction of the data flow among the M points determined for the new potential segmentation point according to the rule.

With reference to the fourth possible implementation of the fifth aspect, in the seventh possible implementation, the determination processing unit is specifically configured to use a random function to determine the window W _iz [p _iz -A _z ,p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [p _iz -A _z ,p _iz +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In a sixth aspect, an embodiment of the present invention provides a server for searching data stream segmentation points, and a rule is preset on the server, and the rule is: determine M windows W _x [kA for a potential segmentation point k _x ,k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x ,k+B _x ], wherein x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers;

The server includes: a determining unit, configured to perform step a:

a) Determine the corresponding window W _iz [k _i -A _z ,k _i for the current potential segmentation point _ki according to the rules

+B _z ], i and z are integers, and 1≤z≤M;

A judging processing unit, configured to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies a predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip the minimum search unit U of N data stream segmentation points, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), obtain a new potential segmentation point, and perform step a);

c When at least part of the data in each of the M windows W _ix [k _i -A _x , _ki +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential segmentation Point _ki is the data flow splitting point.

With reference to the sixth aspect, in the first possible implementation manner, the rule further includes: at least two windows W _ie [k _i -A _e ,k _i +B _e ] and W _if [k _i -A _f ,k _i +B _f ], satisfying the conditions: |A _e +B _e |=|A _f +B _f |, C _e ＝C _f .

With reference to the first possible implementation manner of the sixth aspect, in the second possible implementation manner, the rule further includes: A _e and A _f are positive integers.

In combination with the first possible implementation manner or the second possible implementation manner of the sixth aspect, in the third possible implementation manner, the rule further includes: A _e -1=A _f , _Be +1= B _f .

With reference to the sixth aspect, or any of the first to third possible implementation manners, in a fourth possible implementation manner, the judgment processing unit is specifically configured to

A random function is used to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

With reference to the fourth possible implementation of the sixth aspect, in the fifth possible implementation, the judgment processing unit is specifically configured to use a hash function to judge the window W _iz [k _i -A _z , k _i +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

With reference to the sixth aspect, or any of the first to fifth possible implementation manners, in the sixth possible implementation manner, the judgment processing unit is configured to: when the window W _iz [k _i -A _z ,k _i + At least part of the data in B _z ] does not satisfy the predetermined condition C _z , jump N minimum search units U of the data stream segmentation point from the current potential segmentation point _ki along the data stream segmentation point search direction, and obtain the new The potential segmentation point of , the determination unit performs step a) for the new potential segmentation point, according to the rule, the window W _ic [k _i -A _c ,k _i + The left boundary of B _c ] coincides with the right boundary of the window W _iz [k _i -A _z , _ki +B _z ] or the window W _ic [k _i -A _c , _ki +B _c ] is located within the range of the window W _iz [ _ki -A _z , _ki +B _z ]; wherein, the window W determined for the new potential segmentation point _ic [k _i -A _c , _ki +B _c ] is the first window in the sequence obtained according to the search direction of the data flow among the M windows determined for the new potential segmentation point according to the rule.

In combination with the fourth possible implementation manner of the sixth aspect, in the seventh possible implementation _manner , the _judgment processing unit uses a _{random function} to _judge the Whether at least part of the data satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [k _i -A _z , _ki +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In the seventh aspect, the embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium is used to store executable instructions, and the server executes the executable instructions to find data flow segmentation points, and the server There are rules preset above, and the rules are: determine M points p _x for the potential segmentation point k, the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x , and the window W _x [p _x -A _x ,p _x +B _x ] corresponds to the predetermined condition C _x , wherein x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers;

When the server executes the executable instruction to perform the following steps:

a) Determine the point p _iz and the window W _iz [p _iz -A _z ,p _iz +B _z ] corresponding to the point p _iz for the current potential segmentation point ki according to the rules, _i and z are integers, and 1 ≤z≤M;

b) judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data stream segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, perform step a);

c) When at least part of the data in each of the M windows W _ix [p _ix -A _x , p _ix +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the seventh aspect, in the first possible implementation manner, the rule further includes: at least two points pe and p _f satisfy the conditions of A _e =A _f , _Be =B _f , and C _{e =C f .}

With reference to the first possible implementation manner of the seventh aspect, in the second possible implementation manner, the rule further includes: the at least two points _pe and p _f , relative to the potential segmentation point k, at The data stream split point looks in the reverse direction.

With reference to the first possible implementation manner or the second possible implementation manner of the seventh aspect, in a third possible implementation manner, the rule further includes: between the at least two points p _e and p _f The distance is 1 U.

With reference to the seventh aspect, or any of the first to third possible implementation manners of the seventh aspect, in the fourth possible implementation manner, the server determines that the window W _iz [p _iz -A _z , p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z , specifically including:

The server uses a random function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

With reference to the fourth possible implementation of the seventh aspect, in the fifth possible implementation, the server uses a random function to judge that at least part of the window W _iz [p _iz -A _z ,p _iz +B _z ] Whether the data satisfies the predetermined condition C _z , specifically including:

The server uses a hash function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In combination with the seventh aspect, or any of the first to fifth possible implementations of the seventh aspect, in the sixth possible implementation, when the window W _iz [p _iz -A _z ,p _iz +B _z ] At least part of the data does not satisfy the predetermined condition C _z , jumping from the point p _iz along the direction of searching for the data stream split point by N minimum search units U of the data stream split point to obtain the new potential split point, according to the According to the above rules, the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is the same as the window W _iz [p _iz -A _z , The right boundary of p _iz +B _z ] coincides or the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is located at Within the range of the window W _iz [p _iz -A _z ,p _iz +B _z ]; wherein, the point p _ic determined for the new potential segmentation point is according to the rule, for the new The M points determined by the potential split point are the points ranked first in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the seventh aspect, in the seventh possible implementation, a random function is used to judge whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies The predetermined condition C _z specifically includes:

Select F bytes in the window W _iz [p _iz -A _z ,p _iz +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

In an eighth aspect, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium is used to store executable instructions, and the server executes the executable instructions to find data flow segmentation points. There are rules preset above, and the rules are: determine M windows W _x [kA _x , k+B _x ] and predetermined conditions C _x corresponding to windows W _x [kA _x , k+B _x ] for potential segmentation point k , wherein, x is a continuous natural number from 1 to M, M≥2, A _x and B _x are integers; when the server executes the executable instruction, the following steps are performed:

a) Determine the corresponding window W _iz [k _i -A _z ,k _i for the current potential segmentation point _ki according to the rules

+B _z ], i and z are integers, and 1≤z≤M;

b) judging whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip the minimum search unit U of N data stream segmentation points, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), obtain a new potential segmentation point, and perform step a);

c) When at least part of the data in each window W _ix [k _i -A _x , _ki +B _x ] of the M windows of the current potential segmentation point _ki satisfies the predetermined condition C _x , then the current potential The split point _ki is the data stream split point.

With reference to the eighth aspect, in the first possible implementation manner, the rule further includes: at least two windows W _ie [k _i -A _e ,k _i +B _e ] and W _if [k _i -A _f ,k _i +B _f ], satisfying the conditions: |A _e +B _e |=|A _f +B _f |, C _e ＝C _f .

With reference to the first possible implementation manner of the eighth aspect, in the second possible implementation manner, the rule further includes: A _e and A _f are positive integers.

In combination with the first possible implementation manner or the second possible implementation manner of the eighth aspect, in the third possible implementation manner, the rule further includes: A _e -1=A _f , _Be +1= B _f .

With reference to the eighth aspect, or any of the first to third possible implementation manners of the eighth aspect, in the fourth possible implementation manner, the server determines that the window W _iz [k _i -A _z ,k _i +B _z ] whether at least part of the data satisfies the predetermined condition C _z , specifically including:

A random function is used to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

With reference to the fourth possible implementation of the eighth aspect, in the fifth possible implementation, the server uses a random function to judge that at least part of the window W _iz [k _i -A _z , k _i +B _z ] Whether the data satisfies the predetermined condition C _z , specifically, the server uses a hash function to determine whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In combination with the eighth aspect, or any of the first to fifth possible implementations of the eighth aspect, in the sixth possible implementation, when the window W _iz [k _i -A _z , _ki +B _z ] At least part of the data does not satisfy the predetermined condition C _z , jumping from the current potential segmentation point _ki along the data stream segmentation point search direction for N minimum search units U of the data stream segmentation point to obtain the new potential segmentation point , according to the rule, the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is the same as the window W _iz [k _i -A _z ,k _i +B _z ] coincides with the right boundary or the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is located in the window W _iz [k _i - A _z , _ki + B _z ] range; wherein, the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is according to the rule, as The M windows determined by the new potential segmentation point are the first windows in the sequence obtained according to the search direction of the data flow.

In combination with the fourth possible implementation of the eighth aspect, in the seventh possible implementation, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies The predetermined condition C _z specifically includes:

Select F bytes in the window W _iz [k _i -A _z , _ki +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

In the embodiment of the present invention, the data flow segmentation point is searched by judging whether at least part of the data in one of the M windows satisfies the predetermined condition, and when at least part of the data in a certain window does not meet the predetermined condition, skip N*U length, to obtain the next potential split point, and improve the efficiency of searching for the split point of the data stream.

Description of drawings

FIG. 1 is a schematic diagram of an application scenario according to an embodiment of the present invention;

Fig. 2 is a schematic diagram of a data flow splitting point;

Fig. 3 is a schematic diagram of finding data stream segmentation points;

Fig. 4 is a schematic diagram of the method of the embodiment of the present invention;

Fig. 5 and Fig. 6 are the schematic diagrams of the implementation manner of finding the data stream segmentation point;

FIG. 7 and FIG. 8 are schematic diagrams of implementations of searching for data stream segmentation points;

Fig. 9 and Fig. 10 are the schematic diagrams of the embodiment of finding data stream segmentation point;

Fig. 11 and Fig. 12 and Fig. 13 are the schematic diagrams of the embodiment of finding the data stream segmentation point;

Fig. 14 and Fig. 15 are the schematic diagrams of the embodiment of finding the split point of the data flow;

Figure 16 and Figure 17 are schematic diagrams for judging whether at least part of the data in the window satisfies a predetermined condition;

Figure 18 is a structural diagram of the deduplication server;

Figure 19 is a structural diagram of the deduplication server;

Figure 20 is a schematic diagram of the method of the embodiment of the present invention;

FIG. 21 and FIG. 22 are schematic diagrams of implementations of searching for data stream segmentation points;

FIG. 23 and FIG. 24 are schematic diagrams of implementations of searching for data stream segmentation points;

FIG. 25 and FIG. 26 are schematic diagrams of implementations of finding a data stream segmentation point;

Fig. 27, Fig. 28 and Fig. 29 are schematic diagrams of implementations of finding data stream segmentation points;

FIG. 30 and FIG. 31 are schematic diagrams of implementations of finding a data stream segmentation point;

FIG. 32 and FIG. 33 are schematic diagrams for judging whether at least part of the data in the window satisfies a predetermined condition.

34 and 35 are probability diagrams for judging whether a predetermined condition is satisfied.

specific embodiment

With the continuous advancement of storage technology, the amount of data generated is also increasing, and a large amount of data puts forward the highest requirements for storage capacity. The increase in storage capacity also increases the procurement cost of IT equipment. In order to alleviate the demand contradiction between data volume and storage capacity and save IT equipment procurement costs, data deduplication technology is introduced in the field of data storage.

A usage scenario in the embodiment of the present invention is a data backup scenario. Data backup is the process of backing up data to other storage media through the backup server to prevent data loss caused by various reasons. The data backup system architecture shown in Figure 1. The data backup system includes clients (101a, 101b...101n), backup server 102, data deduplication server (referred to as deduplication server or deduplication server) 103 and storage devices (104a, 104b...104n). Wherein the clients (101a, 101b...101n) can be application servers, workstations, etc.; the backup server 102 is used to back up the data generated by the client; the deduplication server 103 is used to perform the deduplication task of the backup data; the storage device (104a, 104b...104n) As a storage medium for storing deduplicated data, it may be a storage medium such as a disk array or a tape library. Clients (101a, 101b...101n), backup server 102, deduplication server 103, and storage devices (104a, 104b...104n) can be connected through switches, local area networks, the Internet, optical fibers, etc., and the above devices can be located at the same place or Can be located in different locations. The backup server 102, the deduplication server 103, and the storage devices (104a, 104b...104n) can be independent physical devices, or they can be physically integrated in a specific implementation, or the backup server 102 and the deduplication server 103 can be integrated into one, or The deduplication server 103 is integrated with the storage devices (104a, 104b...104n) and the like.

The deduplication server 103 performs deduplication operations on the data stream of the backup data, which generally includes the following steps:

1) Data stream split point search: find the data stream split point in the data stream according to a specific algorithm;

2) Divide the data block according to the found data flow segmentation point;

3) Calculate the characteristic value of the data block: calculate the characteristic value of the data block as the characteristic identifying the data block; add the calculated characteristic value to the characteristic list of the data block of the file corresponding to the data stream; generally use SHA-1 or MD5 algorithm to calculate the characteristic value of the data block;

4) Same data block detection: compare the calculated feature value of the data block with the existing feature value in the data block feature list to determine whether there is the same data block;

5) Delete duplicate data blocks: through the detection of the same data block, if it is found that there is a feature value identical to the data block in the feature list of the data block, it is not necessary to store the data block or determine the number of duplicate data blocks stored according to the backup strategy Whether to store the data block.

From the steps of the deduplication server 103 performing the deduplication operation on the data stream of the backup data, it can be seen that the data stream segmentation point search is a key step of the deduplication operation, which directly determines the performance of the deduplication.

In the embodiment of the present invention, the deduplication server 103 receives the backup file sent by the backup server 102, and performs deduplication processing on the file. Usually, the backup file to be processed is presented in the form of a data stream in the deduplication server 103. When the deduplication server 103 searches for a split point in the data stream, it usually needs to determine the minimum search unit of the data stream split point, as shown in FIG. The split point k ₁ is located between the minimum search units of two consecutive data stream split points whose serial numbers are ₁ and 2 respectively. A potential split point refers to a point that needs to be judged whether it can be used as a data stream split point; Stream split point, the search direction of the data stream split point is shown by the arrow in Figure 2, and the next potential split point to be searched is k ₇ , which is located between two consecutive data stream split point minimum search units whose sequence numbers are 7 and 8 respectively, When the potential splitting point k ₇ is a data stream splitting point, the data between two adjacent data stream splitting points k ₁ and k ₇ is 1 data block. The minimum search unit of the data stream segmentation point can be determined according to actual needs. Here, 1 byte (Byte) is taken as an example, that is, the minimum search unit size of the data stream segmentation points with serial numbers 1, 2, 7, and 8 is 1 word. Festival. The search direction of the data stream split point as shown in FIG. 2 usually means searching from the file head to the file tail, or from the file tail to the file head. In this embodiment, the search from the file head to the file tail is taken as an example.

In the data deduplication scenario, usually the smaller the data block, the higher the deduplication rate, and the easier it is to find the duplicate data block, but the greater the amount of metadata generated, and after the data block is small enough, the data deduplication The rate will not increase, but the amount of metadata will increase dramatically. Therefore, the size of the data block must be controlled. In practical applications, the minimum value of the data block is usually set, such as 4KB (4096 bytes). At the same time, considering the deduplication rate, the maximum value of the data block is also set, namely The data block size cannot exceed a maximum value, such as 12KB (12288 bytes). A specific implementation is shown in Figure 3, the deduplication server 103 searches for the data stream splitting point along the direction indicated by the arrow, k _a is the data stream splitting point currently found, and the search direction from k _a to the data stream splitting point Find the next potential split point. In order to meet the minimum data block requirements, the minimum data block size is usually skipped from the data stream split point along the data stream split point search direction, and the search starts from the end position of the smallest data block, that is, the smallest The end position of the data block is used as the next potential split point k _i . In the embodiment of the present invention, the minimum data block of 4KB can be skipped from point k _a along the search direction of the data stream segmentation point, that is, 4*1024=4096 bytes. Jump 4096 bytes from point k _a along the data stream split point search direction, and obtain point k _i at the end position of the 4096th byte as a potential split point, for example, k _i is located in two consecutive numbers of 4096 and 4097 The data stream split point is between the smallest search units. Still taking Figure 3 as an example, k _a is the currently found data stream split point, and the next data stream split point is searched along the direction shown in Figure 3. If the next data stream split point is still not found if the maximum value of the data block is exceeded, Then, the point k _z which reaches the maximum value of the data block in the search direction from k _a to the data stream split point is taken as the next data stream split point, and the forced split is performed.

An embodiment of the present invention provides a method for finding a data stream segmentation point based on a deduplication server, as shown in FIG. 4 , including:

Rules are preset on the deduplication server 103, and the rules are: determine M points p _x , the window W _x [p _x −A _x , p _x +B _x ] corresponding to the point p _x for potential segmentation point k and The predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], where x is a continuous natural number from 1 to M, M≥2, A _x and B _x are integers; where p _x The distance from the potential segmentation point k is d _x the minimum search unit of the data stream segmentation point. The minimum search unit of the data stream segmentation point is represented by U, and U=1 byte in this embodiment. In the implementation shown in FIG. 3 , regarding the value of M, in one implementation, the value of M*U is not greater than the preset maximum distance between two adjacent data stream segmentation points, that is, the preset The maximum length of the data block. Judging whether at least part of the data in the window W _z [p _z -A _z , p _z +B _z ] corresponding to the point p _z meets the predetermined condition C _z , where z is an integer, 1≤z≤M, (p _z -A _z ) and (p _z +B _z ) denote the two boundaries of the window W _z respectively. When it is judged that at least part of the data in the window W _z [p _z -A _z , p _z +B _z ] of any point p _z does not meet the predetermined condition C _z , then the window W _z [p _z -A _z , p _z +B _z ] corresponding point p _z jumps N bytes along the search direction of data stream segmentation point, N≤‖B _z ‖+max _x (‖A _x ‖+‖(kp _x )‖). Among them, ‖(kp _x )‖ represents the distance between any point in M points p _x and the potential segmentation point k, and max _x (‖A _x ‖+‖(kp _x )‖) represents the distance between M points p _x The maximum value of the sum of the distance between any point and the potential segmentation point k and the absolute value of A _x corresponding to the point; ‖B _z ‖ means B in W _z [p _z -A _z , p _z +B _z ] For the absolute value of _z , the principle of selecting the value of N will be specifically introduced in the following embodiments. When it is determined that at least part of the data in each of the M windows W _x [p _x -A _x , p _x +B _x ] satisfies the predetermined condition C _x , then the potential split point k is a data stream split point.

Specifically, for the current potential segmentation point k _i , according to the rules, the following steps are performed:

Step 401: Determine the point p _iz and the window W _iz [p _iz -A _z ,p _iz +B _z ] corresponding to the point p _iz for the current potential segmentation point ki according to the rule, _i and z are integers, and 1≤z≤M;

Step 402: judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data flow segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, go to step 401;

When at least part of the data in each of the M windows W _ix [p _ix -A _x , p _ix +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential segmentation point k _i is the data stream split point.

Further, the rule further includes: at least two points p _e and p _f satisfy the conditions of A _e =A _f , B _e =B _f , and C _e =C _f ;

The rule further includes: the at least two points _pe and p _f are in the reverse direction of the data flow split point search relative to the potential split point k.

The rule further includes: the distance between the at least two points _pe and p _f is 1 U.

Judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z , specifically includes:

A random function is used to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

The use of a random function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is specifically using a hash function to determine whether the window W _iz [p _iz -A _z , p _iz +B _z ] whether at least part of the data satisfies the predetermined condition C _z .

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data stream segmentation point is used to obtain the new potential segmentation point. According to the rules, the window W _ic corresponding to the point p _ic determined for the new potential segmentation point [p _ic -A _c ,p _ic +B _c ] is coincident with the right boundary of the window W _iz [p _iz -A _z ,p _iz +B _z ] or the point p _ic corresponding to the new potential segmentation point The left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] is within the range of the window W _iz [p _iz -A _z ,p _iz +B _z ]; wherein, is the new potential The point p _ic determined by the split point is the first point in the sequence obtained according to the search direction of the data flow among the M points determined for the new potential split point according to the rule.

In the embodiment of the present invention, the data flow segmentation point is searched by judging whether at least part of the data in one of the M windows satisfies the predetermined condition, and when at least part of the data in a certain window does not meet the predetermined condition, skip N*U length, where N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain the next potential segmentation point, which improves the efficiency of searching for data flow segmentation points.

In the process of data deduplication, in order to ensure uniform data block size, the average data block (also known as average block) size will be considered, that is, the average data block will be determined while meeting the minimum data block size and maximum data block size restrictions size to ensure that the obtained data blocks are uniform in size. The number M of points p _x and the probability that at least part of the data in the window W _x [p _x -A _x , p _x +B _x ] corresponding to point p _x satisfies the predetermined condition C _x , these two factors determine the finding of data stream segmentation The probability of the point (in P(n)). The former affects the length of the jump, the latter affects the probability of the jump, and the two together affect the average block size. Generally speaking, when the average block size is fixed and the number M of points p _x increases, at least part of the data in the window W _x [p _x -A _x , p _x +B _x ] corresponding to a single point p _x meets the predetermined conditions The probability of C _x also increases. For example, the preset rule on the deduplication server 103 is: determine 11 points p _x for the potential segmentation point k, x is a continuous natural number from 1 to 11, and any point p in the 11 points The probability that at least part of the data in the window W _x [p _x -A _x , p _x +B _x ] corresponding to _x satisfies the predetermined condition C _x is 1/2. Another set of rules preset on the deduplication server 103 is: 24 points p _x selected for the potential segmentation point k, x is a continuous natural number from 1 to 24, and any point p _x in the 24 points corresponds to The probability that at least part of the data in the window W _x [p _x -A _x , p _x +B _x ] satisfies the predetermined condition C _x is 3/4. The probability setting of at least part of the data in the specific window W _x [p _x -A _x , p _x +B _x ] meeting the predetermined condition C _x can be found in the judgment window W _x [p _x -A _x , p _x +B _x ] Whether at least part of the data satisfies the description of the predetermined condition C _x part. The number M of points p _x and the probability that at least part of the data in the window W _x [p _x -A _x , p _x +B _x ] corresponding to point p _x meet the predetermined condition C _x These two factors determine P(n), P (n) indicates: the probability that no data stream split point is found after searching n minimum search units of data stream split points from the data stream start position/previous data stream split point. The calculation process of determining P(n) by these two factors is actually a multi-step Fibonacci sequence, which will be described in detail later. After obtaining P(n), 1-P(n) is the distribution function of the data stream segmentation point, (1-P(n))-(1-P(n-1))=P(n-1)- P(n), that is, the probability of finding the data stream split point at the nth point, that is, the density function of the data stream split point, can be integrated according to the density function of the data stream split point Thus, the expected length of the split point of the data stream, that is, the average block size, is obtained, wherein 4*1024 (bytes) represents the minimum data block length, and 12*1024 (bytes) represents the maximum data block length.

On the basis of the data stream segmentation point search as shown in Figure 3, in the embodiment shown in Figure 5, a rule is preset on the deduplication server 103, and the rule is: determine 11 points for the potential segmentation point k p _x , window W _x [p _x -A _x ,p _x +B _x ] (referred to as window W _x ) corresponding to point p _x and reservation corresponding to window W _x [p _x -A _x ,p _x +B _x ] Condition C _x , where A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ = B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ . Among them, the distance between point p _x and potential segmentation point k is d _x bytes, specifically, the distance between point p ₁ and potential segmentation point k is 0 bytes, and the distance between point p ₂ and potential segmentation point k is 1 bytes, the distance between point p ₃ and potential segmentation point k is 2 bytes, the distance between point p ₄ and potential segmentation point k is 3 bytes, and the distance between point p ₅ and potential segmentation point k is 4 bytes node, the distance between point p ₆ and potential split point k is 5 bytes, the distance between point p ₇ and potential split point k is 6 bytes, the distance between point p ₈ and potential split point k is 7 bytes, The distance between point p ₉ and potential split point k is 8 bytes, the distance between point p ₁₀ and potential split point k is 9 bytes, the distance between point p ₁₁ and potential split point k is 10 bytes, and point p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ , p ₉ , p ₁₀ , and p ₁₁ are all located in the opposite direction of the data flow split point search relative to the potential split point k. k _a is the data stream splitting point. The search direction of the data stream splitting point shown in Figure 5 is from left to right. After skipping the smallest data block 4KB from the data stream splitting point k _a , the smallest data block 4KB end position is taken as the next potential The split point _ki is a point p _ix determined for the potential split point _ki . In this embodiment, according to the preset rule on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 5 , 11 points are determined for the potential segmentation point _ki , which are p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 , the windows corresponding to points p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 - 169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169, p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -169,p _i11 ]. The above windows are referred to as W _i1 , W _i2 , W _i3 , W _i4 , W _i5 , W _i6 , W _i7 , W _i8 , W _i9 , W _i10 and W _i11 for short respectively. Among them, the distance between point p _ix and potential segmentation point _ki is d _x bytes, specifically, the distance between p _i1 and _ki is 0 bytes, the distance between p _i2 and _ki is 1 byte, and the distance between p _i3 and _ki The distance between p _i4 and _ki is 3 bytes, the distance between p _i5 and _ki is 4 bytes, the distance between p _i6 and _ki is 5 bytes, the distance between p _i7 and _ki is 6 bytes, The distance between p _i8 and _ki is 7 bytes, the distance between p _i9 and _ki is 8 bytes, the distance between p _i10 and _ki is 9 bytes, the distance between p _i11 and _ki is 10 bytes, and p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are all located in the reverse direction of the data flow segmentation point search relative to the potential segmentation point _ki . Judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , judging W _i3 [ Whether at least part of the data in p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and judge whether W _i5 [p _i5 -169 ,p _i5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169,p _i6 ] meets the predetermined condition C ₆ , judging W _i7 [p _i7 -169,p _i7 ] Whether at least part of the data in W i8 [p i8 -169, p i8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _i8 [p _i8 -169, p _i8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _i9 [p _i9 -169, p _i9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _i10 [p _i10 -169, p _i10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [p _i11 -169, p _i11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined condition, as shown in Figure 6, at least part of the data in W _i5 [p _i5 -169, p _i5 ] does not meet the corresponding predetermined condition C ₅ , Then jump N bytes from the point p _i5 along the search direction of the data stream segmentation point, where N bytes are not greater than ‖B ₅ ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), In the embodiment shown in Fig. 6, jumping N bytes is no more than 179 bytes, and in this embodiment, N=11, obtains next potential segmentation point, for being different from potential segmentation point _ki , new here The potential split points of are denoted as k _j . According to the preset rules on the deduplication server 103 in the embodiment shown in FIG. 5 , there are 11 points determined for the potential segmentation point k _j , namely p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 , determine the points p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and The windows corresponding to p _j11 are respectively W _j1 [p _j1 -169,p _j1 ], W _j2 [p _j2 -169,p _j2 ], W _j3 [p _j3 -169,p _j3 ], W _j4 [p _j4 -169 ,p _j4 ], W _j5 [p _j5 -169,p _j5 ], W _j6 [p _j6 -169,p _j6 ], W _j7 [p _{j7 -169} ,p _j7 ], W _j8 [p _{j8 -169} ,p _j8 ], W _j9 [p _j9 -169,p _j9 ], W _j10 [p _j10 -169,p _j10 ], and W _j11 [p _j11 -169,p _j11 ]. Among them, the distance between p _jx and potential segmentation point k _j is d _x bytes, specifically, the distance between p _j1 and k _j is 0 bytes, the distance between p _j2 and k _j is 1 byte, and the distance between p _j3 and k _j 2 bytes, 3 bytes between p _j4 and k _j , 4 bytes between p _j5 and k _j , 5 bytes between p _j6 and k _j , 6 bytes between p _j7 and k _j , p The distance between _j8 and k _j is 7 bytes, the distance between p _j9 and k _j is 8 bytes, the distance between p _j10 and k _j is 9 bytes, the distance between p _j11 and k _j is 10 bytes, and p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 , and p _j11 are all located in the opposite direction of the data flow split point search relative to the potential split point k _j . In the embodiment shown in Figure 6, when the 11th window W _j11 [p _j11 -169,p _j11 ] determined for the potential segmentation point k _j , between the guaranteed potential segmentation point k _i and the potential segmentation point k _j range is within the judgment range, then in this embodiment, it must be ensured that the left boundary of window W _j11 [p _j11 -169,p _j11 ] coincides with the right boundary p _i5 of W _i5 [p _i5 -169,p _i5 ] Or within the range of W _i5 [p _i5 -169,p _i5 ], wherein the point p _j11 determined by the potential segmentation point k _j is the M points determined for the potential segmentation point k _j according to the rule The first point in the sequence obtained according to the search direction of the data flow. Therefore, within this limit, when at least part of the data in W _i5 [p _i5 -169, p _i5 ] does not satisfy the predetermined condition C ₅ , the jumping distance from p _i5 along the direction of data stream segmentation point search is not greater than ∥B ₅ ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), where M=11, 11*U is not greater than max _x (‖A _x ‖+‖(k _i -p _ix )‖ ), therefore, the jumping distance from p _i5 along the search direction of the data stream segmentation point is not greater than 179. Judging whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , judging W _j3 [ Whether at least part of the data in p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , and judge whether W _j5 [p _j5 -169 ,p _j5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging W _j7 [p _{j7 -169} ,p _j7 ] Whether at least part of the data in W j8 [p j8 -169,p j8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _j11 [p _j11 -169, p _j11 ] meets the predetermined condition _C11 . Of course, in the embodiment of the present invention, this rule is also followed when judging whether the potential split point k _a is a data stream split point, and the specific implementation will not be described again, and the description of judging the potential split point k _i can be referred to. When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data stream split point. When it is judged that the potential split point k _j is not a data stream split point, jump 11 bytes in the same way as k _i to obtain the next potential split point, and judge according to the preset rules on the deduplication server 103 and the above method Whether a potential split point is a stream split point. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the forced split point.

In the embodiment shown in FIG. 5 , according to the preset rules on the deduplication server 103, it starts from judging whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , when it is judged that W At least part of the data in _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , at least part of the data in the judgment W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , and the judgment W _i3 [p _i3 -169 ,p _i3 ] at least part of the data satisfies the predetermined condition C ₃ and at least part of the data in the judgment W _i4 [p _i4 -169,p _i4 ] satisfies the predetermined condition C ₄ , and in the judgment W _i5 [p _i5 -169,p _i5 ] at least When part of the data does not meet the predetermined condition _C5 , jump 10 bytes from the point p _i5 along the direction of data flow segmentation point search, and obtain a new potential segmentation point at the end position of the 10th byte, which is the same as other potential segmentation points The difference, expressed here as k _g , according to the preset rules on the deduplication server 103, determines 11 points p _gx for the potential segmentation point k _g , and x is a continuous natural number from 1 to 11, which are respectively p _g1 and p _g2 , p _g3 , p _g4 , p _g5 , p _g6 , p _g7 , p _g8 , p _g9 , p _g10 and p _g11 , determine the points p _g1 , p _g2 , p _g3 , p _g4 , p _g5 , p _g6 , p _g7 , p _g8 , p _g9 , p _g10 and p _g11 correspond to W _g1 [p _g1 -169,p _g1 ], W _g2 [p _g2 -169,p _g2 ], W _g3 [p _g3 -169,p _g3 ], W _g4 [p _g4 -169,p _g4 ], W _g5 [p _g5 -169,p _g5 ], W _g6 [p _g6 -169,p _g6 ], W _g7 [p _g7 -169,p _g7 ] , W _g8 [p _g8 -169, p _g8 ], W _g9 [p _g9 -169, p _g9 ], W _g10 [p _g10 -169, p _g10 ], and W _g11 [p _g11 -169, p _g11 ]. Among them, the distance between p _gx and potential segmentation point k _g is d _x bytes, specifically, the distance between p _g1 and k _g is 0 bytes, the distance between p _g2 and k _g is 1 byte, and the distance between p _g3 and k _g 2 bytes, 3 bytes between p _g4 and k _g , 4 bytes between p _g5 and k _g , 5 bytes between p _g6 and k _g , 6 bytes between p _g7 and k _g , p The distance between _g8 and k _g is 7 bytes, the distance between p _g9 and k _g is 8 bytes, the distance between p _g10 and k _g is 9 bytes, the distance between p _g11 and k _g is 10 bytes, and p _g2 , p _g3 , p _g4 , p _g5 , p _g6 , p _g7 , p _g8 , p _g9 , p _g10 and p _g11 are all located in the opposite direction of the data flow split point search relative to the potential split point k _g . Judging whether at least part of the data in W _g1 [p _g1 -169, p _g1 ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _g2 [p _g2 -169, p _g2 ] meets the predetermined condition C ₂ , judging W _g3 [ p _g3 -169,p _g3 ] whether at least part of the data satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _g4 [p _g4 -169,p _g4 ] meets the predetermined condition C ₄ , and judge W _g5 [p _g5 -169 ,p _g5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _g6 [p _g6 -169,p _g6 ] meets the predetermined condition C ₆ , judging W _g7 [p _g7 -169,p _g7 ] Whether at least part of the data in W g8 [p g8 -169,p g8 ] meets the predetermined condition C ₇ , whether at least part of the data in W _g8 [p _g8 -169,p _g8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _g9 [p _g9 -169,p _g9 ] Whether to meet the predetermined condition C ₉ , judging whether at least part of the data in W _g10 [p _g10 -169, p _g10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _g11 [p _g11 -169, p _g11 ] meets the predetermined condition _C11 . Therefore, the point p _g11 corresponding to the potential segmentation point k _g coincides with the point p _i5 corresponding to the potential segmentation point k _i , and the window W _g11 [p _g11 -169,p _g11 ] corresponding to the point p _g11 is the window corresponding to the point p _i5 W _i5 [p _i5 -169,p _i5 ] coincides, and C ₅ =C ₁₁ , therefore, for the potential segmentation point _ki , when it is judged that at least part of the data in W _i5 [p _i5 -169,p _i5 ] does not meet the predetermined When the condition C is ₅ , jumping 10 bytes from the point p _i5 along the direction of data stream split point search, the obtained potential split point k _g still does not meet the condition of being a data stream split point. Therefore, if jumping 10 bytes from point p _i5 along the search direction of the data flow split point, there will be double calculation, and jumping 11 bytes from point p _i5 along the search direction of the data stream split point can reduce double count and be more efficient . Therefore, the speed of finding data stream split points is increased. When the probability that at least part of the data in the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the midpoint p _x satisfies the predetermined condition C _x is 1/2, that is to say, 1/2 Probability to perform a jump, and each time a maximum of 179 bytes can be jumped.

In this embodiment, the predetermined rule is: determine 11 points p _x for the potential segmentation point k, the window W _x [p _x -A _x , p _x +B _x ] and the window W _x [p _x corresponding to the point p _x -A _x ,p _x +B _x ] corresponding to the predetermined condition C _x , x is a continuous natural number from 1 to 11, wherein, the window W _x corresponding to point p _x [p _x -A _x ,p _x +B _x ] The probability that at least some of the data satisfy the predetermined condition is 1/2, and P(n) can be calculated by these two factors. And A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ ＝C ₁₁ , wherein, the distance between p _x and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 0 bytes, and the distance between p ₂ and k is 1 byte Bytes, the distance between p ₃ and k is 2 bytes, the distance between p ₄ and k is 3 bytes, the distance between p ₅ and k is 4 bytes, and the distance between p ₆ and k is 5 bytes , the distance between p ₇ and k is 6 bytes, the distance between p ₈ and k is 7 bytes, the distance between p ₉ and k is 8 bytes, the distance between p ₁₀ and k is 9 bytes, p The distance between ₁₁ and k is 10 bytes, and p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ , p ₉ , p ₁₀ and p ₁₁ are all located in the data Stream split point lookup in reverse direction. Therefore, whether there are at least part of the data in each window corresponding to 11 consecutive points satisfying the predetermined condition _Cx determines whether the potential segmentation point k is a data stream segmentation point. After skipping the minimum block length of 4096 bytes from the start position of the data stream/the last data stream split point, go back 10 bytes in the opposite direction to the data stream split point and find the 4086th point. There is a data stream split point, so P(4086)=1, and so on, P(4087)=1, ... P(4095)=1. At the 4096th point, that is, at the minimum block size, with a probability of (1/2)^11, at least part of the data in each window corresponding to these 11 points satisfies the predetermined condition C _x , so (1 There is a data stream split point with a probability of /2)^11, and there is no data stream split point with a probability of 1-(1/2)^11, so P(11)=1-(1/2)^11.

As shown in Figure 34, at the nth point, P(n) can be deduced recursively in 12 cases.

Case 1: At least part of the data in the window corresponding to the nth point does not meet the predetermined condition with a probability of 1/2. At this time, the n-1 points in front of the nth point do not exist continuously with a probability of P(n-1) At least part of the data in each of the windows corresponding to the 11 points of , respectively satisfy the predetermined conditions, so P(n) includes 1/2*P(n-1). At least part of the data in the window corresponding to the nth point does not meet the predetermined conditions, and there are n-1 points in front of the nth point, and at least part of the data in each of the windows corresponding to 11 consecutive points meet the predetermined conditions respectively The case is independent of P(n).

Case 2: At least part of the data in the window corresponding to the nth point meets the predetermined condition with a probability of 1/2, and at least part of the data in the window corresponding to the n-1th point does not meet the predetermined condition with a probability of 1/2. At this time The n-2 points in front of the n-1th point do not exist with the probability of P(n-2), at least part of the data in each window corresponding to the continuous 11 points in the window respectively meet the predetermined conditions, so P(n) Contains 1/2*1/2*P(n-2). At least part of the data in the window corresponding to the nth point meets the predetermined condition, at least part of the data in the window corresponding to the n-1th point does not meet the predetermined condition, and there are continuous n-2 points in front of the n-1th point The fact that at least part of the data in each of the windows corresponding to the 11 points respectively satisfies the predetermined condition has nothing to do with P(n).

According to the above description, case 11: at least part of the data in the window corresponding to the nth to n-9 points meets the predetermined condition with a probability of (1/2)^10, and at least part of the data in the window corresponding to the n-10th point is The probability of 1/2 does not meet the predetermined conditions. At this time, the n-11 points in front of the n-10th point do not exist with the probability of P(n-11). In each window corresponding to 11 consecutive points, at least Part of the data respectively satisfies the predetermined conditions, so P(n) includes (1/2)^10*1/2*P(n-11). At least part of the data in the window corresponding to the n-9th point meets the predetermined condition, at least part of the data in the window corresponding to the n-10th point does not meet the predetermined condition, and n-11 points in front of the n-10th point It is independent of P(n) that at least part of the data in each of the windows corresponding to 11 consecutive points in each window satisfy the predetermined condition respectively.

Case 12: At least part of the data in the window corresponding to the nth to n-10th points meets the predetermined condition with a probability of (1/2)^11, and this case has nothing to do with P(n).

Therefore, P(n)=1/2*P(n-1)+(1/2)^2*P(n-2)+...+(1/2)^11*P(n-11) . Another preset rule: determine 24 points p _x for potential segmentation point k, the window W _x [p _x -A _x ,p _x +B _x ] and window W _x [p _x -A _x corresponding to point p _x ,p _x +B _x ] corresponding to the predetermined condition C _x , x is a continuous natural number from 1 to 24, wherein, at least part of the window W _x [p _x -A _x ,p _x +B _x ] corresponding to point p _x The probability that the data satisfies the predetermined condition C _x is 3/4, and P(n) can be calculated by these two factors. And A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =... =C ₂₂ =C ₂₃ =C ₂₄ , wherein, the distance between p _x and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 0 bytes, p ₂ and k The distance between p ₃ and k is 2 bytes, the distance between p ₄ and k is 3 bytes, the distance between p ₅ and k is 4 bytes, and the distance between p ₆ and k The distance between p 7 and k is 5 bytes, the distance between p ₇ and k is 6 bytes, the distance between p ₈ and k is 7 bytes, the distance between p ₉ and k is 8 bytes, ... the distance between p ₂₂ and k 21 bytes, the distance between p ₂₃ and k is 22 bytes, the distance between p ₂₄ and k is 23 bytes, and p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ , p ₉ . . . p ₂₂ , p ₂₃ and p ₂₄ are all located in the opposite direction of the data flow split point search relative to the potential split point k. Therefore, whether there are at least part of the data in each window corresponding to 24 consecutive points satisfying the predetermined condition C _x determines whether the potential segmentation point k is a data stream segmentation point, which can be calculated by the following formula:

P(4073)=1, P(4074)=1, ... P(,4095)=1, P(4096)=1-(3/4)^24,

P(n)＝1/4*P(n-1)+1/4*(3/4)*P(n-2)+......+1/4*(3/4)^23*P( n-24).

After calculation, P(5*1024)＝0.78, P(11*1024)＝0.17, P(12*1024)＝0.13, that is, after finding 12KB from the data stream start position/previous data stream split point, use 13 % probability that the data stream split point is still not found, forcing a split. Through this probability, the density function of the split point of the data stream is obtained, and the approximate average of the data stream split point is found when searching for 7.6 KB from the starting position of the data stream/the previous split point of the data stream through integration, that is, the average block length is about It is 7.6KB. Unlike at least part of the data in the window corresponding to consecutive 11 points that meets the predetermined condition with a probability of 1/2, the traditional CDC algorithm uses a window that meets the condition with a probability of 1/2^12 to achieve an average block length of 7.6 The effect of KB.

On the basis of the data stream segmentation point search shown in FIG. 3 , in the embodiment shown in FIG. 7 , a rule is preset on the deduplication server 103, and the rule is: determine 11 points for the potential segmentation point k p _x , the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], x respectively is a continuous natural number from 1 to 11, where the probability that at least part of the data in the window W _x [p _x -A _x ,p _x +B _x ] corresponding to point p _x meets the predetermined condition C _x is 1/2, and A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ , wherein, the distance between p _x and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 2 bytes, and the distance between p ₂ and k is 3 bytes, The distance between p ₃ and k is 4 bytes, the distance between p ₄ and k is 5 bytes, the distance between p ₅ and k is 6 bytes, the distance between p ₆ and k is 7 bytes, p ₇ The distance between p 8 and k is 8 bytes, the distance between p ₈ and k is 9 bytes, the distance between p ₉ and k is 10 bytes, the distance between p ₁₀ and k is 1 byte, and the distance between p ₁₁ and k The distance between them is 0 bytes, and p ₁ , p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ , p ₉ and p ₁₀ are all located at the data stream split point with respect to the potential split point k Find the opposite direction. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 7 is from left to right. After skipping the minimum data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is used as the next The potential segmentation point _ki is to determine the point p _ix for the potential segmentation point _ki . In this embodiment, according to the preset rules on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 7 , according to predetermined rules, 11 points are determined for the potential segmentation point _{ki, namely p i1 ,} p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 , the windows corresponding to points p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are respectively W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -169,p _i11 ]. Among them, the distance between point p _ix and potential segmentation point _ki is d _ix bytes. Specifically, the distance between p _i1 and _ki is 2 bytes, the distance between p _i2 and _ki is 3 bytes, and the distance between p _i3 and _ki The distance between p _i4 and _ki is 5 bytes, the distance between p _i5 and _ki is 6 bytes, the distance between p _i6 and _ki is 7 bytes, the distance between p _i7 and _ki is 8 bytes, The distance between p _i8 and _ki is 9 bytes, the distance between p _i9 and _ki is 10 bytes, the distance between p _i10 and _ki is 1 byte, the distance between p _i11 and _ki is 0 bytes, and p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , and p _i10 are all located in the opposite direction of the data stream segmentation point search relative to the potential segmentation point _ki . Judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , judging W _i3 [ Whether at least part of the data in p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and judge whether W _i5 [p _i5 -169 ,p _i5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169,p _i6 ] meets the predetermined condition C ₆ , judging W _i7 [p _i7 -169,p _i7 ] Whether at least part of the data in W i8 [p i8 -169, p i8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _i8 [p _i8 -169, p _i8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _i9 [p _i9 -169, p _i9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _i10 [p _i10 -169, p _i10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [p _i11 -169, p _i11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined condition, as shown in Figure 8, at least part of the data in W _i3 [p _i3 -169, p _i3 ] does not meet the predetermined condition C ₃ , point p _i3 jumps 11 bytes along the search direction of the data stream split point as an example for description. As shown in Figure 8, when it is judged that W ₃ does not meet the predetermined conditions, start from p ₃ and jump N bytes along the direction of data stream segmentation point search, where N bytes are not greater than ‖B ₃ ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), in the embodiment shown in Figure 6, skip N bytes, specifically not more than 179 bytes, in this embodiment, N= 11. At the end position of the 11th byte, obtain the next potential segmentation point. In order to distinguish it from the potential segmentation point ki, the new potential segmentation point is represented as _{k j here} , according to the preset value on the deduplication server 103 Rule, there are 11 points determined for the potential segmentation point k _j , which are p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 , Determine the windows corresponding to points p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 respectively as W _j1 [p _j1 -169,p _j1 ] , W _j2 [p _j2 -169,p _j2 ], W _j3 [p _j3 -169,p _j3 ], W _j4 [p _j4 -169,p _j4 ], W _j5 [p _j5 -169,p _j5 ], W _j6 [p _j6 -169,p _j6 ], W _j7 [p _{j7 -169} ,p _j7 ], W _j8 [p _{j8 -169} ,p _j8 ], W _j9 [p _j9 -169,p _j9 ], W _j10 [ p _j10 -169,p _j10 ] and W _j11 [p _j11 -169,p _j11 ]. Among them, the distance between p _jx and potential segmentation point k _j is d _x bytes, specifically, the distance between p _j1 and k _j is 2 bytes, the distance between p _j2 and k _j is 3 bytes, and the distance between p _j3 and k _j 4 bytes, 5 bytes between p _j4 and k _j , 6 bytes between p _j5 and k _j , 7 bytes between p _j6 and k _j , 8 bytes between p _j7 and k _j , p The distance between _j8 and k _j is 9 bytes, the distance between p _j9 and k _j is 10 bytes, the distance between p _j10 and k _j is 1 byte, the distance between p _j11 and k _j is 0 bytes, and p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , and p _j10 are all located in the opposite direction of the data flow split point search relative to the potential split point k _j . Judging whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , judging W _j3 [ Whether at least part of the data in p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , and judge whether W _j5 [p _j5 -169 ,p _j5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging W _j7 [p _{j7 -169} ,p _j7 ] Whether at least part of the data in W j8 [p j8 -169,p j8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _j11 [p _j11 -169, p _j11 ] meets the predetermined condition _C11 . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data flow split point. When it is judged that the potential segmentation point k _j is not a data stream segmentation point, jump 11 bytes in the same way as k _i to obtain the next potential segmentation point, and judge according to the preset rules on the deduplication server 103 and the above method Whether a potential split point is a stream split point. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the forced split point. Of course, the implementation of this method is limited by the maximum data block length and the size of the files constituting the data stream, so details will not be repeated here.

On the basis of the data stream segmentation point search shown in FIG. 3 , in the embodiment shown in FIG. 9 , a rule is preset on the deduplication server 103, and the rule is: determine 11 points for the potential segmentation point k p _x , the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x , and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], where A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ = _C11 . Among them, the distance between p _x and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 3 bytes, the distance between p ₂ and k is 2 bytes, and p ₃ The distance between p 4 and k is 1 byte, the distance between p ₄ and k is 0 byte, the distance between p ₅ and k is 1 byte, the distance between p ₆ and k is 2 bytes, and the distance between p ₇ and k The distance between p ₈ and k is 4 bytes, the distance between p ₉ and k is 5 bytes, the distance between p ₁₀ and k is 6 bytes, and the distance between p ₁₁ and k The distance is 7 bytes, and p ₅ , p ₆ , p ₇ , p ₈ , p ₉ , p ₁₀ and p ₁₁ are all located in the opposite direction of the data flow split point search relative to the potential split point k, p ₁ , p ₂ and p ₃ are located in the search direction of the data flow split point relative to the potential split point k. k _a is the data stream splitting point. The search direction of the data stream splitting point shown in Figure 9 is from left to right. After skipping the smallest data block 4KB from the data stream splitting point k _a , the smallest data block 4KB end position is taken as the next potential The split point _ki is a point p _ix determined for the potential split point _ki . In this embodiment, according to the preset rule on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 9 , 11 points are determined for the potential segmentation point _{ki, namely p i1 ,} p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 , the windows corresponding to points p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 - 169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169, p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -169,p _i11 ]. Among them, the distance between p _ix and potential segmentation point _ki is d _x bytes, specifically, the distance between p _i1 and _ki is 3 bytes, the distance between p _i2 and _ki is 2 bytes, and the distance between p _i3 and _ki 1 byte, the distance between p _i4 and _ki is 0 bytes, the distance between p _i5 and _ki is 1 byte, the distance between p _i6 and _ki is 2 bytes, the distance between p _i7 and _ki is 3 bytes, p The distance between _i8 and _ki is 4 bytes, the distance between p _i9 and _ki is 5 bytes, the distance between p _i10 and _ki is 6 bytes, the distance between p _i11 and _ki is 7 bytes, and p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are all located in the opposite direction of the data flow segmentation point search relative to the potential segmentation point _{ki, and p i1 ,} p _i2 and p _i3 are all located in the data stream relative to the potential segmentation point _ki Split point lookup direction. Judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , judging W _i3 [ Whether at least part of the data in p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and judge whether W _i5 [p _i5 -169 ,p _i5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169,p _i6 ] meets the predetermined condition C ₆ , judging W _i7 [p _i7 -169,p _i7 ] Whether at least part of the data in W i8 [p i8 -169, p i8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _i8 [p _i8 -169, p _i8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _i9 [p _i9 -169, p _i9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _i10 [p _i10 -169, p _i10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [p _i11 -169, p _i11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined conditions, as shown in Figure 10, at least part of the data in W _i7 [p _i7 -169, p _i7 ] does not meet the corresponding predetermined conditions, then from Point p _i7 jumps N bytes along the search direction of the data flow splitting point, where N bytes are not greater than ‖B ₄ ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), in Fig. In the embodiment shown in 10, N bytes are skipped, specifically no more than 179 bytes. In this embodiment, N=8 is specifically taken to obtain a new potential segmentation point, which is different from the potential segmentation point _ki , here the new potential segmentation point is denoted as _{k j} , according to the rules preset on the deduplication server 103 in the embodiment shown in FIG _. , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 , determine the points p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 correspond to W _j1 [p _j1 -169,p _j1 ], W _j2 [p _j2 -169,p _j2 ], W _j3 [p _j3 - 169,p _j3 ], W _j4 [p _j4 -169,p _j4 ], W _j5 [p _j5 -169,p _j5 ], W _j6 [p _j6 -169,p _j6 ], W _j7 [p _{j7 -169} , p _j7 ], W _j8 [p _{j8 -169} ,p _j8 ], W _j9 [p _j9 -169,p _j9 ], W _j10 [p _j10 -169,p _j10 ], and W _j11 [p _j11 -169,p _j11 ]. Among them, the distance between p _jx and potential segmentation point k _j is d _x bytes. Specifically, the distance between p _j1 and k _j is 3 bytes, the distance between p _j2 and k _j is 2 bytes, and the distance between p _j3 and k _j is 1 byte, the distance between p _j4 and k _j is 0 bytes, the distance between p _j5 and k _j is 1 byte, the distance between p _j6 and k _j is 2 bytes, the distance between p _j7 and k _j is 3 bytes, p The distance between _j8 and k _j is 4 bytes, the distance between p _j9 and k _j is 5 bytes, the distance between p _j10 and k _j is 6 bytes, the distance between p _j11 and k _j is 7 bytes, and p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 are all located in the opposite direction of the data flow segmentation point search relative to the potential segmentation point k _j , and p _j1 , p _j2 and p _j3 are all located in the data stream relative to the potential segmentation point k _j Split point lookup direction. Judging whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , judging W _j3 [ Whether at least part of the data in p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , and judge whether W _j5 [p _j5 -169 ,p _j5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging W _j7 [p _{j7 -169} ,p _j7 ] Whether at least part of the data in W j8 [p j8 -169,p j8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _j11 [p _j11 -169, p _j11 ] meets the predetermined condition _C11 . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data stream split point. When it is judged that the potential segmentation point k _j is not a data flow segmentation point, jump 8 bytes in the same way as k _i to obtain the next potential segmentation point, and judge according to the preset rules on the deduplication server 103 and the above method Whether a potential split point is a stream split point. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the mandatory split point.

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 11, a rule is preset on the deduplication server 103, and the rule is: determine 11 points for the potential segmentation point k p _x , the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x , and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], where A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =169, A ₁₁ =182, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ ≠C ₁₁ . Among them, the distance between p _x and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 0 bytes, the distance between p ₂ and k is 1 byte, and p ₃ The distance between p 4 and k is 2 bytes, the distance between p ₄ and k is 3 bytes, the distance between p ₅ and k is 4 bytes, the distance between p ₆ and k is 5 bytes, and the distance between p ₇ and k The distance between p ₈ and k is 7 bytes, the distance between p ₉ and k is 8 bytes, the distance between p ₁₀ and k is 1 byte, and the distance between p ₁₁ and k The distance is 3 bytes, and, p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ and p ₉ are all located in the opposite direction of the data flow split point search relative to the potential split point k, p ₁₀ and p ₁₁ is located in the search direction of the data stream split point relative to the potential split point k. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 11 is from left to right. After skipping the minimum data block 4KB from the data stream split point k _a , the minimum data block 4KB end position is taken as the next potential The split point _ki is a point p _ix determined for the potential split point _ki . In this embodiment, according to the preset rule on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 11 , 11 points are determined for the potential segmentation point k _i , which are p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 , the windows corresponding to points p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 - 169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169, p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -182,p _i11 ]. Among them, the distance between p _ix and potential segmentation point _ki is d _x bytes, specifically, the distance between p _i1 and _ki is 0 bytes, the distance between p _i2 and _ki is 1 byte, and the distance between p _i3 and _ki 2 bytes, 3 bytes between p _i4 and _ki , 4 bytes between p _i5 and _ki , 5 bytes between p _i6 and _ki , 6 bytes between p _i7 and _ki , p The distance between _i8 and _ki is 7 bytes, the distance between p _i9 and _ki is 8 bytes, the distance between p _i10 and _ki is 1 byte, the distance between p _i11 and _ki is 3 bytes, and p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 and p _i9 are all located in the opposite direction of the data stream segmentation point search relative to the potential segmentation point _ki , and p _i10 and p _i11 are located in the data stream relative to the potential segmentation point _ki Split point lookup direction. Judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , judging W _i3 [ Whether at least part of the data in p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and judge whether W _i5 [p _i5 -169 ,p _i5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169,p _i6 ] meets the predetermined condition C ₆ , judging W _i7 [p _i7 -169,p _i7 ] Whether at least part of the data in W i8 [p i8 -169, p i8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _i8 [p _i8 -169, p _i8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _i9 [p _i9 -169, p _i9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _i10 [p _i10 -169, p _i10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [p _i11 -169, p _i11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in the judging window W _i11 does not meet the predetermined condition C ₁₁ , then jump 1 byte from the potential segmentation point _ki along the data stream segmentation point search direction to obtain a new potential segmentation point, which is the potential segmentation point _ki difference, here denote the new potential segmentation point as k _j . When at least part of the data in any of the 10 windows W _i1 , W _i2 , W _i3 , W _i4 , W _i5 , W _i6 , W _i7 , W _i8 , W _i9 , and W _i10 does not meet the corresponding predetermined conditions, such as As shown in Figure 12, W _i4 [p _i4 -169, p _i4 ], jump N bytes from point p _i4 along the direction of data stream segmentation point search, where N bytes are not greater than ‖B ₄ ‖+max _x (‖A _x ‖+‖(k _i −p _ix )‖), in the embodiment shown in Figure 12, skip N bytes, specifically not more than 179, in this embodiment, specifically take N=9 , to obtain a new potential segmentation point, in order to distinguish it from the potential segmentation point ki, here the new potential segmentation point is expressed as _{k j ,} according to the rules preset on the deduplication server 103 in the embodiment shown in FIG. 11 , as The potential segmentation point k _j determines 11 points, which are p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 . The windows corresponding to _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 and p _j11 are respectively W _j1 [p _j1 -169,p _j1 ], W _j2 [p _j2 -169,p _j2 ], W _j3 [p _j3 -169,p _j3 ], W _j4 [p _j4 -169,p _j4 ], W _j5 [p _j5 -169,p _j5 ], W _j6 [p _j6 -169,p _j6 ], W _j7 [p _{j7 -169} ,p _j7 ], W _j8 [p _{j8 -169} ,p _j8 ], W _j9 [p _j9 -169,p _j9 ], W _j10 [p _j10 - 169,p _j10 ] and W _j11 [p _j8 -182,p _j8 ]. Among them, the distance between p _jx and potential segmentation point k _j is d _x bytes, specifically, the distance between p _j1 and k _j is 0 bytes, the distance between p _j2 and k _j is 1 byte, and the distance between p _j3 and k _j 2 bytes, 3 bytes between p _j4 and k _j , 4 bytes between p _j5 and k _j , 5 bytes between p _j6 and k _j , 6 bytes between p _j7 and k _j , p The distance between _j8 and k _j is 7 bytes, the distance between p _j9 and k _j is 8 bytes, the distance between p _j10 and k _j is 1 byte, the distance between p _j11 and k _j is 3 bytes, and p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 and p _j9 are all located in the opposite direction of the data stream segmentation point search relative to the potential segmentation point k _j , and p _j10 and p _j11 are located in the data stream relative to the potential segmentation point k _j Split point lookup direction. Judging whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , judging W _j3 [ Whether at least part of the data in p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , and judge whether W _j5 [p _j5 -169 ,p _j5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging W _j7 [p _{j7 -169} ,p _j7 ] Whether at least part of the data in W j8 [p j8 -169,p j8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _j11 [p _j11 -182, p _j11 ] meets the predetermined condition _C11 . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data flow split point. When it is judged that the potential split point k _j is not a data flow split point, the next potential split point is obtained in the same manner as k _i , and the next potential split point is judged according to the preset rules on the deduplication server 103 and the above method. Split point for data flow. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the forced split point.

On the basis of the data stream segmentation point search shown in FIG. 3 , in the embodiment shown in FIG. 13 , a rule is preset on the deduplication server 103: determine 11 points p _x , point The window W _x [p _x -A _x ,p _x +B _x ] corresponding to p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], x is 1 to 11 respectively Continuous natural numbers, where the probability that at least part of the data in the window W _x [p _x -A _x ,p _x +B _x ] corresponding to point p _x meets the predetermined condition is 1/2, and A ₁ =A ₂ =A ₃ =A ₄ =A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ , where p _x The distance between p 1 and potential segmentation point k is d _x bytes, specifically, the distance between p ₁ and potential segmentation point k is 0 bytes, the distance between p ₂ and k is 2 bytes, and the distance between p ₃ and k The distance between p ₄ and k is 6 bytes, the distance between p ₅ and k is 8 bytes, the distance between p ₆ and k is 10 bytes, and the distance between p ₇ and k is 12 bytes bytes, the distance between p ₈ and k is 14 bytes, the distance between p ₉ and k is 16 bytes, the distance between p ₁₀ and k is 18 bytes, and the distance between p ₁₁ and k is 20 bytes , and p ₂ , p ₃ , p ₄ , p ₅ , p ₆ , p ₇ , p ₈ , p ₉ , p ₁₀ and p ₁₁ are all located in the opposite direction of the data flow split point search relative to the potential split point k. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 13 is from left to right. After skipping the smallest data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is used as the next The potential segmentation point _ki is to determine the point p _ix for the potential segmentation point _ki . In this embodiment, according to the preset rules on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 13 , according to predetermined rules, 11 points are determined for the potential segmentation point _{ki, namely p i1 ,} p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 , the windows corresponding to points p _i1 , p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are respectively W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -169,p _i11 ]. Among them, the distance between p _ix and potential segmentation point _ki is d _x bytes, specifically, the distance between p _i1 and _ki is 0 bytes, the distance between p _i2 and _ki is 2 bytes, and the distance between p _i3 and _ki 4 bytes, 6 bytes between p _i4 and _ki , 8 bytes between p _i5 and _ki , 10 bytes between p _i6 and _ki , 12 bytes between p _i7 and _ki , p The distance between _i8 and _ki is 14 bytes, the distance between p _i9 and _ki is 16 bytes, the distance between p _i10 and _ki is 18 bytes, the distance between p _i11 and _ki is 20 bytes, and p _i2 , p _i3 , p _i4 , p _i5 , p _i6 , p _i7 , p _i8 , p _i9 , p _i10 and p _i11 are all located in the opposite direction of the data flow segmentation point search relative to the potential segmentation point _ki . Judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , judging W _i3 [ Whether at least part of the data in p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and judge whether W _i5 [p _i5 -169 ,p _i5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169,p _i6 ] meets the predetermined condition C ₆ , judging W _i7 [p _i7 -169,p _i7 ] Whether at least part of the data in W i8 [p i8 -169, p i8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _i8 [p _i8 -169, p _i8 ] meets the predetermined condition C ₈ , or whether at least part of the data in W _i9 [p _i9 -169, p _i9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _i10 [p _i10 -169, p _i10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [p _i11 -169, p _i11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined condition, as shown in Figure 14, at least part of the data in W _i4 [p _i4 -169,p _i4 ] does not meet the predetermined condition C ₄ , then select The next potential segmentation point, to be distinguished from the potential segmentation point ki, is denoted as k _j here, k _j is located to the right of _ki , and the distance between _{k j and ki} is 1 byte. As shown in Figure 14, according to the preset rules on the deduplication server 103, 11 points are determined for the potential segmentation point k _j , which are p _j1 , p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , _{p j8 , p j9 , p j10 and p j11 , determine the corresponding} The windows are respectively W _j1 [p _j1 -169,p _j1 ], W _j2 [p _j2 -169,p _j2 ], W _j3 [p _j3 -169,p _j3 ], W _j4 [p _j4 -169,p _j4 ] , W _j5 [p _j5 -169,p _j5 ], W _j6 [p _j6 -169,p _j6 ], W _j7 [p _{j7 -169} ,p _j7 ], W _j8 [p _{j8 -169} ,p _j8 ], W _j9 [p _j9 -169,p _j9 ], W _j10 [p _j10 -169,p _j10 ] and W _j11 [p _j11 -169,p _j11 ], where A ₁ ＝A ₂ ＝A ₃ ＝A ₄ ＝A ₅ =A ₆ =A ₇ =A ₈ =A ₉ =A ₁₀ =A ₁₁ =169, B ₁ =B ₂ =B ₃ =B ₄ =B ₅ =B ₆ =B ₇ =B ₈ =B ₉ =B ₁₀ =B ₁₁ =0, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ . Among them, the distance between p _jx and potential segmentation point k _j is d _x bytes, specifically, the distance between p _j1 and k _j is 0 bytes, the distance between p _j2 and k _j is 2 bytes, and the distance between p _j3 and k _j 4 bytes, 6 bytes between p _j4 and k _j , 8 bytes between p _j5 and k _j , 10 bytes between p _j6 and k _j , 12 bytes between p _j7 and k _j , p The distance between _j8 and k _j is 14 bytes, the distance between p _j9 and k _j is 16 bytes, the distance between p _j10 and k _j is 18 bytes, the distance between p _j11 and k _j is 20 bytes, and p _j2 , p _j3 , p _j4 , p _j5 , p _j6 , p _j7 , p _j8 , p _j9 , p _j10 , and p _j11 are all located in the opposite direction of the data flow split point search relative to the potential split point k _j . Judging whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , judging W _j3 [ Whether at least part of the data in p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , and judge whether W _j5 [p _j5 -169 ,p _j5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging W _j7 [p _{j7 -169} ,p _j7 ] Whether at least part of the data in W j8 [p j8 -169,p j8 ] satisfies the predetermined condition C ₇ , whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] Whether the predetermined condition C ₉ is met, judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _j11 [p _j11 -169, p _j11 ] meets the predetermined condition _C11 . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in the window W _i9 meets the predetermined condition _C9 , at least part of the data in the window _Wj10 meets the predetermined condition _C10 , and at least part of the data in the window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data flow splitting point. When judging that at least part of the data in any of the windows W _j1 , W _j2 , W _j3 , W _j4 , W _j5 , W _j6 , W _j7 , W _j8 , W _j9 , W _j10 and W _j11 does not meet the predetermined conditions, as As shown in Figure 15, when at least part of the data in W _j3 [p _j3 -169, p _j3 ] does not meet the predetermined condition C ₃ , point p _i4 is located on the left side of point p _j3 relative to the search direction of the data flow splitting point, from point p _i4 along Jump 21 bytes in the search direction of the data stream split point to obtain the next potential split point, which is expressed as k _l to distinguish it from potential split points ki and _{k j .} According to the preset rules on the deduplication server 103 in the embodiment shown in FIG. 13 , there are 11 points determined for the potential segmentation point k _l , which are p _l1 , p _l2 , p _l3 , p _l4 , p _l5 , and p _l6 , p _l7 , p _l8 , p _l9 , p _l10 and p _l11 , the points p _l1 , p _l2 , p _l3 , p _l4 , p _l5 , p _l6 , p _l7 , p _l8 , p _l9 , p _l10 and p _l11 correspond to The windows of W _l1 [p _l1 -169,p _l1 ], W _l2 [p _l2 -169,p _l2 ], W _l3 [p _l3 -169,p _l3 ], W _l4 [p _l4 -169,p _l4 ], W _l5 [p _l5 -169,p _l5 ], W _l6 [p _l6 -169,p _l6 ], W _l7 [p _l7 -169,p _l7 ], W _l8 [p _l8 -169,p _l8 ], W _l9 [p _l9 -169,p _l9 ], W _l10 [p _l10 -169,p _l10 ] and W _l11 [p _{l11 -169} ,p _l11 ], where the distance between p _lx and potential segmentation point k _l is d _x bytes, specifically, the distance between p _l1 and potential segmentation point k _l is 0 bytes, the distance between p _l2 and k _l is 2 bytes, the distance between p _l3 and k _l is 4 bytes, The distance between p _l4 and k _l is 6 bytes, the distance between p _l5 and k _l is 8 bytes, the distance between p _l6 and k _l is 10 bytes, and the distance between p _l7 and k _l is 12 characters section, the distance between p _l8 and k _l is 14 bytes, the distance between p _l9 and k _l is 16 bytes, the distance between p _l10 and k _l is 18 bytes, and the distance between p _l11 and k _l is 20 bytes bytes, and p _l2 , p _l3 , p _l4 , p _l5 , p _l6 , p _l7 , p _l8 , p _l9 , p _l10 and p _l11 are all located in the opposite direction of the data stream split point search relative to the potential split point k _l . Judging whether at least part of the data in W _l1 [p _l1 -169,p _l1 ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _l2 [p _l2 -169,p _l2 ] meets the predetermined condition C ₂ , judging W _l3 [ p _l3 -169,p _l3 ] whether at least part of the data meet the predetermined condition C ₃ , judge whether at least part of the data in W _l4 [p _l4 -169,p _l4 ] meet the predetermined condition C ₄ , judge W _l5 [p _l5 -169 ,p _l5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _l6 [p _l6 -169,p _l6 ] meets the predetermined condition C ₆ , judge whether W _l7 [p _l7 -169,p _l7 ] Whether at least part of the data in W l8 [p l8 -169,p l8 ] meets the predetermined condition C ₇ , whether at least part of the data in W _l8 [p _l8 -169,p _l8 ] meets the predetermined condition C ₈ , and whether at least part of the data in W _l9 [p _l9 -169,p _l9 ] is judged Whether to meet the predetermined condition C ₉ , judging whether at least part of the data in W _l10 [p _l10 -169,p _l10 ] meets the predetermined condition C ₁₀ and judging whether at least part of the data in W _l11 [p _{l11 -169} ,p _l11 ] meets the predetermined condition _C11 . When it is judged that at least part of the data in window W _l1 meets the predetermined condition C ₁ , at least part of the data in window W _l2 meets the predetermined condition C ₂ , at least part of the data in window W _l3 meets the predetermined condition C ₃ , and at least part of the data in window W _l4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _l5 meet the predetermined condition C ₅ , at least part of the data in window W _l6 meet the predetermined condition C ₆ , at least part of the data in window W _l7 meet the predetermined condition C ₇ , at least part of the data in window W _l8 When the predetermined condition C8 is met, at least part of the data in window _W19 meets the predetermined condition _C9 , at least part of the data in window _W110 meets the predetermined condition _C10 , and at least part of the data in window _{W111 meets} the predetermined condition _C11 , then the current potential segmentation Point k _l is the data flow splitting point. When at least some of the data in any of the windows W _l1 , W _l2 , W _l3 , W _l4 , W _l5 , W _l6 , W _l7 , W _l8 , W _l9 , W _l10 and W _l11 do not meet the predetermined conditions, select the following A potential segmentation point, to be distinguished from potential segmentation points _ki , _kj and _kl , is denoted as km, km is located to the right of _{kl ,} and the distance between _km and _kl is ₁ byte. According to the preset rules on the deduplication server 103 in the embodiment shown in Fig. 13, there are 11 points determined for the potential segmentation point k _m , which are respectively p _m1 , p _m2 , p _m3 , p _m4 , p _m5 , p _m6 , p _m7 , p _m8 , p _m9 , p _m10 and p _m11 , the points p _m1 , p _m2 , p _m3 , p _m4 , p _m5 , p _m6 , p _m7 , p _m8 , p _m9 , p _m10 and p _m11 correspond to The windows are respectively W _m1 [p _m1 -169,p _m1 ], W _m2 [p _m2 -169,p _m2 ], W _m3 [p _m3 -169,p _m3 ], W _m4 [p _m4 -169,p _m4 ], W _m5 [p _m5 -169, p _m5 ], W _m6 [p _m6 -169, p _m6 ], W _m7 [p _m7 -169, p _m7 ], W _m8 [p _m8 -169, p _m8 ], W _m9 [p _m9 -169,p _m9 ], W _m10 [p _m10 -169,p _m10 ] and W _m11 [p _m11 -169,p _m11 ], where the distance d between p _mx and the potential segmentation point k _{m x} bytes, specifically, the distance between p _m1 and potential segmentation point k _m is 0 bytes, the distance between p _m2 and k _m is 2 bytes, the distance between p _m3 and k _m is 4 bytes, The distance between p _m4 and k _m is 6 bytes, the distance between p _m5 and k _m is 8 bytes, the distance between p _m6 and k _m is 10 bytes, and the distance between p _m7 and k _m is 12 characters section, the distance between p _m8 and k _m is 14 bytes, the distance between p _m9 and k _m is 16 bytes, the distance between p _m10 and k _m is 18 bytes, and the distance between p _m11 and k _m is 20 bytes bytes, and p _m2 , p _m3 , p _m4 , p _m5 , p _m6 , p _m7 , p _m8 , p _m9 , p _m10 and p _m11 are all located in the reverse direction of the data flow split point search relative to the potential split point k _m . Judging whether at least part of the data in W _m1 [p _m1 -169, p _m1 ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _m2 [p _m2 -169, p _m2 ] meets the predetermined condition C ₂ , judging whether W _m3 [ p _m3 -169, p _m3 ] whether at least part of the data satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _m4 [p _m4 -169, p _m4 ] meet the predetermined condition C ₄ , and judge whether W _m5 [p _m5 -169 ,p _m5 ] whether at least part of the data satisfies the predetermined condition C ₅ , judging whether at least part of the data in W _m6 [ _pm6 -169, _pm6 ] meets the predetermined condition C ₆ , judging whether W _m7 [p _m7 -169,p _m7 ] Whether at least part of the data in W _m8 [p _{m8 -169} ,p _m8 ] meets the predetermined condition C ₈ , _whether at least part of the data in W _m8 [ _pm Whether to meet the predetermined condition C ₉ , judge whether at least part of the data in W _m10 [p _m10 -169, p _m10 ] meet the predetermined condition C ₁₀ and judge whether at least part of the data in W _m11 [p _m11 -169, p _m11 ] meet the predetermined condition _C11 . When it is judged that at least part of the data in window W _m1 meets the predetermined condition C ₁ , at least part of the data in window W _m2 meets the predetermined condition C ₂ , at least part of the data in window W _m3 meets the predetermined condition C ₃ , and at least part of the data in window W _m4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _m5 meets the predetermined condition C ₅ , at least part of the data in window W _m6 meets the predetermined condition C ₆ , at least part of the data in window W _m7 meets the predetermined condition C ₇ , and at least part of the data in window W _m8 When the predetermined condition C8 is met, at least part of the data in window _Wm9 meets the predetermined condition _C9 , at least part of the data in window _Wm10 meets the predetermined condition _C10 , and at least part of the data in window _{Wm11 meets} the predetermined condition _C11 , then the current potential segmentation The point k _m is the data flow splitting point. When at least part of the data in any window does not meet the predetermined condition, jumping is performed according to the scheme described above to obtain the next potential split point and determine whether it is a data flow split point.

The embodiment of the present invention provides a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z . In this embodiment, a random function is used to judge the window W _iz [ Whether at least part of the data in p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z , taking the implementation shown in Figure 5 as an example, according to the preset rules on the deduplication server 103, is a potential split k _i determines the point p _i1 and the window W _i1 [p _i1 -169, p _i1 ] corresponding to the point p _i1 , and judges whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , such as As shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169, p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , select 5 bytes , "■" in Figure 16 represents one selected byte, and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, the bits corresponding to 255 bytes It can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 For any one, the bits corresponding to 255 bytes are obtained according to the conversion relationship between a _m,n and V _am,n to obtain the matrix V _a , which can be expressed as: Select a large number of random numbers to form a matrix. Once the matrix composed of random data is formed, it remains unchanged. For example, select 255*8 random numbers from random numbers that obey a specific distribution (here, take the normal distribution as an example) to form a matrix R: Multiply the m-th row of the matrix V _a with the random number in the m-th row of the matrix R, and then sum to get a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _am,2 *h _m,2 +...+V _am,8 *h _m,8 . According to this method, S _a1 , S _a2 . . . to S _a255 are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _a1 , S _a2 . . . to S _a255 is counted. Since the matrix R obeys the normal distribution, S _am , like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of the normal distribution random number being greater than 0 is 1/2, in S _a1 , S _a2 ... to S _a255 In , the probability of each value greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K _whose value is greater than 0 from S _a1 , S _a2 . To meet the conditions. When K is an even number, it indicates that at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ ; when K is an odd number, it indicates that at least part of the data in W _i1 [p _i1 -169,p _i1 ] The predetermined condition C ₁ is not satisfied, where C ₁ means that the number K of _{S a1} , S _a2 . _{[ _ _ _ _ _ _ _ _ _} p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169, p _i10 ], and W _i11 [p _i11 -169,p _i11 ], each window has the same size, namely The size of the window is 169 bytes. At the same time, the method of judging whether at least part of the data in the window meets the predetermined condition is the same. For details, see the above-mentioned judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ describe. Therefore, as shown in Figure 16, Indicates one byte selected when judging whether at least part of the data in the window W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as b _m,1 ...b _m,8 , which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, the corresponding bits of 255 bytes It can be expressed as: When b _m,n =1, V _bm,n =1, when b _m,n =0, V _bm,n =-1, where b _m,n represents b _m,1 ...b _m,8 For any one, the bits corresponding to 255 bytes are obtained according to the conversion relationship between b _m,n and V _bm,n to obtain the matrix V _b , which can be expressed as: The method of judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition is the same as the method of judging whether at least part of the data in the window W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition, so use Matrix R: Multiply the m-th row of the matrix V _b with the random number in the m-th row of the matrix R, and then sum to get a value, specifically expressed as S _bm =V _bm,1 *h _m,1 +V _bm,2 *h _m,2 +...+V _bm,8 *h _m,8 . According to this method, obtain S _b1 , S _b2 . . . to S _b255 , and _count the number K of values in S _b1 , S _b2 . Since the matrix R obeys the normal distribution, S _bm , like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of the random number of the normal distribution being greater than 0 is 1/2, in S _b1 , S _b2 ... to S _b255 In , the probability of each value greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K whose value is greater than 0 from S _b1 , S _b2 ... to S _b255 is an even number, the probability that the random number of the binomial distribution is an even number is 1/2, so K is with the probability of 1/2 To meet the conditions. When K is an even number, it indicates that at least part of the data in W _i2 [p _i2 -169, p _i2 ] meets the predetermined condition C ₂ ; when K is an odd number, it indicates that at least part of the data in W _i2 [p _i2 -169, p _i2 ] The predetermined condition C ₂ is not met, where C ₂ means that the number K of _{S b1} , S _b2 . In the embodiment shown in FIG. 3 , at least part of the data in W _i2 [p _i2 −169, p _i2 ] satisfies the predetermined condition C ₂ .

Therefore, as shown in Figure 16, Indicates one byte selected when judging whether at least part of the data in the window W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the method of judging whether at least part of the data in the windows W _i1 [p _i1 -169,p _i1 ] and W _i2 [p _i2 -169,p _i2 ] meet the predetermined conditions, and judge W _i3 [p _i3 -169,p _i3 ] Whether at least the data satisfy the predetermined condition C ₃ . In the embodiment shown in FIG. 5 , at least part of the data in W _i3 [p _i3 −169, p _i3 ] satisfies a predetermined condition. As shown in Figure 16, Indicates one byte selected when judging whether at least part of the data in the window W _i4 [p _i4 -169,p _i4 ] satisfies the predetermined condition C ₄ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the method of judging whether at least part of the data in the windows W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ] and W _i3 [p _i3 -169,p _i3 ] meet the predetermined conditions, It is judged whether at least part of the data in W _i4 [p _i4 -169, p _i4 ] satisfies the predetermined condition C ₄ . In the embodiment shown in FIG. 5 , at least part of the data in W _i4 [p _i4 -169, p _i4 ] satisfies the predetermined condition C ₄ . As shown in Figure 16, Indicates one byte selected when judging whether at least part of the data in the window W _i5 [p _i5 -169,p _i5 ] satisfies the predetermined condition C ₅ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the judgment windows W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ] and W _i4 [p _i4 -169,p _i4 The method of whether at least part of the data in W _i5 [p _i5 −169, p _i5 ] satisfies the predetermined condition C ₅ is determined. In the embodiment shown in FIG. 5 , at least part of the data in W _i5 [p _i5 −169, p _i5 ] does not satisfy the predetermined condition C ₅ .

When at least part of the data in W _i5 [p _i5 -169,p _i5 ] does not meet the predetermined condition C ₅ , jump 11 bytes from point p _i5 along the direction of data flow splitting point search, at the end of the 11th byte The position obtains the next potential segmentation point k _j , as shown in Figure 6, according to the preset rules on the deduplication server 103, determine the point p _j1 for the potential segmentation point k _j , and the window W _j1 [p _j1 corresponding to the point p _j1 -169,p _j1 ], the method of judging whether at least part of the data in the window W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ is the same as judging at least part of the data in the window W _i1 [p _i1 -169,p _i1 ] Whether or not the predetermined condition C ₁ is met is the same, so as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], to judge at least part of the data in W _j1 [p _j1 -169,p _j1 ] Whether the predetermined condition C ₁ is met or not, 5 bytes are selected. "■" in FIG. 17 represents 1 byte selected, and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as a _m,1 '...a _m,8 ', indicating the 1st to 8th bits of the mth byte in 255 bytes, therefore, 255 bytes correspond to bits can be expressed as: When a _m,n '=1, V _am,n '=1, when a _m,n '=0, V _am,n '=-1, where a _m,n ' means a _m,1 '... For any one of a _m,8 ', the bits corresponding to 255 bytes are obtained according to the conversion relationship between a _m,n ' and V _am,n ' to obtain a matrix V _a ', which can be expressed as: Judging whether at least part of the data in the window W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition is the same as judging whether at least part of the data in the window W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition, so Using matrix R: Multiply the mth row of the matrix V _a ' with the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am '=V _am,1 '*h _m,1 +V _{am, 2} '*h _m,2 +...+V _am,8 '*h _m,8 . According to this method, S _a1 ′, S _a2 ′… to S _a255 ′ are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _a1 ′, S _a2 ′… to S _a255 ′ is counted. Since the matrix R obeys the normal distribution, S _am ', like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of a normal distribution random number being greater than 0 is 1/2. In S _a1 ', S _a2 '... To S _a255 ', the probability of each value being greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K whose value is greater than 0 from S _a1 ', S _a2 '... to S _a255 ' is an even number, and the probability that the random number of the binomial distribution is an even number is 1/2, so K is 1/2 The probability satisfies the condition. When K is an even number, it indicates that at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ ; when K is an odd number, it indicates that at least part of the data in W _j1 [p _j1 -169,p _j1 ] The predetermined condition C ₁ is not satisfied.

The method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ is the same as the method of judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , so, as shown in Figure 17, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as b _m,1 '...b _m,8 ', which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, 255 bytes correspond to bits can be expressed as: When b _m,n '=1, V _bm,n '=1, when b _m,n '=0, V _bm,n '=-1, where b _m,n ' means b _m,1 '... For any one of b _m,8 ', the bits corresponding to 255 bytes are obtained according to the conversion relationship between b _m,n ' and V _bm,n ' to obtain matrix V _b ', which can be expressed as: Whether at least part of the data in the window W ₂ [p ₂ -169,p ₂ ] and W ₂ [q ₂ -169,q ₂ ] meet the predetermined condition is the same, so the matrix R is still used: Multiply the mth row of the matrix V _b ' with the random number of the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _bm '=V _bm,1 '*h _m,1 +V _{bm, 2} '*h _m,2 +...+V _bm,8 '*h _m,8 . According to this method, S _b1 ′, S _b2 ′… to S _b255 ′ are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _b1 ′, S _b2 ′… to S _b255 ′ is counted. Since the matrix R obeys the normal distribution, S _bm ', like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of a normal distribution random number being greater than 0 is 1/2. In S _b1 ', S _b2 '... To S _b255 ', the probability of each value being greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K of S _b1 ', S _b2 '... to S _b255 ' whose value is greater than 0 is an even number, and the probability that the random number of the binomial distribution is an even number is 1/2, so K is 1/ The probability of 2 satisfies the condition. When K is an even number, it indicates that at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ ; when K is an odd number, it indicates that at least part of the data in W _j2 [p _j2 -169,p _j2 ] The predetermined condition C ₂ is not satisfied. Similarly, the method of judging whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ In the same way, judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , judge whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meet the predetermined condition C _5. Judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , judging Whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , judge whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] meet the predetermined condition C ₉ , and judge whether W _j10 [p Whether at least part of the data in _j10 -169,p _j10 ] satisfies the predetermined condition C ₁₀ and whether at least part of the data in W _j11 [p _j11 -169,p _j11 ] meets the predetermined condition C ₁₁ will not be repeated here.

Still taking the embodiment shown in FIG. 5 as an example, a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is provided, which is used in this embodiment The random function judges whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] meets the predetermined condition C _z , and determines the point for the potential split point k _i according to the preset rules on the deduplication server 103 p _i1 and the window W _i1 [p _i1 -169, p _i1 ] corresponding to p _i1 , judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , as shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169,p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ , select 5 bytes, in Figure 16 "■" indicates one byte selected, and the difference between two adjacent bytes selected "■" is 42 bytes. One of the implementation methods is to use the HASH function to calculate the selected 5 bytes. The value calculated by the HASH function is a fixed uniform distribution. If the value calculated by the HASH function is an even number, then judge W _i1 [p _i1 -169 ,p _i1 ] at least part of the data satisfies the predetermined condition C ₁ , that is, C ₁ indicates that the value calculated by using the HASH function in the above manner is an even number. Therefore, the probability of whether at least part of the data in W _i1 [p _i1 −169, p _i1 ] satisfies the predetermined condition is 1/2. In the embodiment shown in FIG. 5 , the Hash function is used to determine whether at least part of the data in W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C ₂ , and at least part of the data in W _i3 [p _i3 -169, p _i3 ] Whether the data meets the predetermined condition C ₃ , whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meet the predetermined condition C ₄ and whether at least part of the data in W _i5 [p _i5 -169,p _i5 ] meet the predetermined condition C _5. For specific implementation, refer to the manner C ₁ that describes whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies a predetermined condition by using a Hash function in the embodiment shown in FIG. 5 , and details are not repeated here.

When at least part of the data in W _i5 [p _i5 -169,p _i5 ] does not meet the predetermined condition C ₅ , jump 11 bytes from point p _i5 along the direction of data flow split point search, at the end of the 11th byte The position obtains the current potential segmentation point k _j , as shown in Figure 6, according to the preset rules on the deduplication server 103, determine the point p _{j1 for} the potential segmentation point k _j , and the window W _j1 [p _j1 - 169,p _j1 ], the method of judging whether at least part of the data in the window W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ is the same as judging whether at least part of the data in the window W _i1 [p _i1 -169,p _i1 ] The way to satisfy the predetermined condition C ₁ is the same, so as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], in order to judge whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] Satisfy the predetermined condition C ₁ , select 5 bytes, "■" in Fig. 17 represents 1 selected byte, and the difference between two adjacent selected bytes "■" is 42 bytes. Use the Hash function to calculate the 5 bytes selected from the window W _j1 [p _j1 -169,p _j1 ], if the obtained value is an even number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined Condition C ₁ . In Fig. 17, the method of judging whether at least part of the data in W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [p _j2 -169, p _j2 ] satisfies the predetermined condition C ₂ in the same way, so, as shown in Figure 17, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , and the adjacent two selected bytes There is a difference of 42 bytes between them. Use the Hash function to calculate the selected 5 bytes, and if the obtained value is an even number, then at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ . In Fig. 17, the method of judging whether at least part of the data in W _i3 [p _i3 -169, p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169, p _j3 ] satisfies the predetermined condition C ₃ in the same way, so, as shown in Figure 17, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ , and the adjacent two selected bytes " There is a difference of 42 bytes between ". Use the Hash function to calculate the selected 5 bytes, and the value obtained is an even number, then at least part of the data in W _j3 [p _j3 -169,p _j3 ] meets the predetermined condition C ₃ . Figure 17 In the method of judging whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ and whether at least part of the data in the window W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ way, therefore, as shown in Figure 17, Indicates one byte selected when judging whether at least part of the data in the window W _j4 [p _j4 -169,p _j4 ] satisfies the predetermined condition C ₄ , and two adjacent selected bytes There is a difference of 42 bytes between them. Use the Hash function to calculate the selected 5 bytes, and the value obtained is an even number, then at least part of the data in W _j4 [p _j4 -169,p _j4 ] satisfies the predetermined condition C ₄ . According to the above method, judge whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meet the predetermined condition C ₆ , Judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , judging whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , judging W _j9 [ p _j9 -169, p _j9 ] whether at least part of the data satisfies the predetermined condition C ₉ , judging whether at least part of the data in W _j10 [p _j10 -169, p _j10 ] meets the predetermined condition C ₁₀ and judging W _j11 [p _j11 -169 ,p _j11 ] whether at least part of the data satisfies the predetermined condition C ₁₁ , which will not be repeated here.

Taking the implementation shown in FIG. 5 as an example, a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is provided. In this embodiment, The random function judges whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] meets the predetermined condition C _z , and determines the point for the potential split point k _i according to the preset rules on the deduplication server 103 The window W _i1 [p _i1 -169, p _i1 ] corresponding to p _i1 and p _i1 , judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , as shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169,p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , select 5 bytes, and the serial number in Figure 16 is The byte "■" of 169, 127, 85, 43 and 1 represents one selected byte respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are respectively converted into a decimal value, expressed as a ₁ , a ₂ , a ₃ , a ₄ and a ₅ respectively. Because a byte is composed of 8 bits, each byte "■" is used as a value, and any a _r among a ₁ , a ₂ , a ₃ , a ₄ and a ₅ satisfies 0≤a _r ≤ 255. a ₁ , a ₂ , a ₃ , a ₄ and a ₅ form a 1*5 matrix. Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix R, which is expressed as:

According to the value of a ₁ and the column where it is located, find the corresponding value from the matrix R, such as a ₁ = 36, a ₁ is located in the first column, then look for the value corresponding to h _36,1 ; according to the value of a ₂ and where it is column, find the corresponding value from the matrix R, such as a ₂ = 48, a ₂ is located in the second column, then find the value corresponding to h _48,2 ; according to the value of a ₃ and the column where it is located, find the corresponding value from the matrix R value, such as a ₃ = 26, a ₃ is located in the third column, then look for the value corresponding to h _26,3 ; according to the value of a ₄ and the column where it is located, find the corresponding value from the matrix R, such as a ₄ = 26 , a ₄ is located in the fourth column, then search for the value corresponding to h _26,4 ; according to the value of a ₅ and the column where it is located, find the corresponding value from the matrix R, such as a ₅ = 88, a ₅ is located in the fifth column, Then find the value corresponding to h _88,5 . S ₁ =h _36,1 +h _48,2 +h _26,3 +h _26,4 +h _88,5 , because the matrix R obeys the binomial distribution, therefore, S ₁ also obeys the binomial distribution. When S ₁ is an even number, then at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , and when S ₁ is an odd number, then at least part of the data in W _i1 [p _i1 -169,p _i1 ] If the predetermined condition C ₁ is not met, the probability that S ₁ is an even number is 1/2, and C ₁ indicates that S ₁ is an even number calculated in the above manner. In the embodiment shown in FIG. 5 , at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ . As shown in Figure 16, Represents the 1 byte selected when judging whether at least part of the data in the window W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C _2. In FIG. Indicates that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 respectively converted into a decimal value, respectively expressed as b ₁ , b ₂ , b ₃ , b ₄ and b ₅ . Since 1 byte consists of 8 bits, each byte As a numerical value, any b _r among b ₁ , b ₂ , b ₃ , b ₄ and b ₅ satisfies 0≤b _r ≤255. b ₁ , b ₂ , b ₃ , b ₄ and b ₅ form a 1*5 matrix. In this embodiment, the method of judging whether at least part of the data in W _i1 and W _i2 meets the predetermined conditions is the same, so the matrix R is still used, and the corresponding value is searched from the matrix R according to the value _of b1 and the column where it is located, such as b ₁ = 66, b ₁ is located in the first column, then search for the value corresponding to h _66,1 ; according to the value of b ₂ and the column where it is located, find the corresponding value from the matrix R, such as b ₂ = 48, b ₂ is in the first column 2 columns, then search for the value corresponding to h _48,2 ; according to the value of b ₃ and the column where it is located, find the corresponding value from the matrix R, such as b ₃ =99, b ₃ is located in the third column, then search for h _99, The value corresponding to ₃ ; according to the value of b ₄ and the column where it is located, find the corresponding value from the matrix R, such as b ₄ = 26, b ₄ is located in the fourth column, then find the value corresponding to h ₂₆ , 4; according to b ₅ Find the corresponding value from the matrix R, such as b ₅ =90, and b ₅ is located in the fifth column, then find the value corresponding to h _90,5 . S ₂ =h _66,1 +h _48,2 +h _99,3 +h _26,4 +h _90,5 , because the matrix R obeys the binomial distribution, therefore, S ₂ also obeys the binomial distribution. When S ₂ is an even number, at least part of the data in W _i2 [p _i2 -169, p _i2 ] meets the predetermined condition C ₂ , and when S ₂ is an odd number, then at least part of the data in W _i2 [p _i2 -169, p _i2 ] If the predetermined condition C ₂ is not satisfied, the probability that S ₂ is an even number is 1/2. In the embodiment shown in FIG. 5 , at least part of the data in W _i2 [p _i2 −169, p _i2 ] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , and judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] satisfies the predetermined condition C _4. Judging whether at least part of the data in W _i5 [p _i5 -169, p _i5 ] meets the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169, p _i6 ] meets the predetermined condition C ₆ , judging W Whether at least part of the data in _i7 [p _i7 -169,p _i7 ] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [p _i8 -169,p _i8 ] meets the predetermined condition C ₈ , and judge W _i9 [p _i9 -169,p _i9 ] whether at least part of the data satisfies the predetermined condition C ₉ , judging whether at least part of the data in W _i10 [p _i10 -169,p _i10 ] meets the predetermined condition C ₁₀ and judging W _i11 [p _i11 -169,p _i11 ] whether at least part of the data satisfies the predetermined condition C ₁₁ . In the embodiment shown in FIG. 5 , at least part of the data in W _i5 [p _i5 -169, p _i5 ] does not meet the predetermined condition C ₅ , jumping 11 bytes from point p _i5 along the direction of data flow splitting point search, at The end position of the 11th byte obtains the current potential segmentation point k _j , as shown in FIG. 6 , according to the preset rules on the deduplication server 103, determine the point p _j1 and the corresponding point p _j1 for the potential segmentation point k _j Window W _j1 [p _j1 -169,p _j1 ], the method of judging whether at least part of the data in window W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ is the same as judging window W _i1 [p _i1 -169,p _i1 ] whether at least part of the data satisfies the predetermined condition C ₁ in the same way, so as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], to judge W _j1 [p _j1 -169,p Whether at least part of the data in _j1 ] satisfies the predetermined condition C ₁ , the bytes "■" with serial numbers 169, 127, 85, 43 and 1 in Figure 17 represent one selected byte respectively, and two adjacent selected bytes There is a difference of 42 bytes between them. Convert the bytes "■" with sequence numbers 169, 127, 85, 43, and 1 into a decimal value, which are respectively expressed as a ₁ ', a ₂ ', a ₃ ', a ₄ ', and a ₅ '. Because 1 byte consists of 8 bits, each byte "■" is used as a value, and any a _r 'in a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' is Satisfy 0≤a _r '≤255. a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' form a 1*5 matrix. The method of judging whether at least part of the data in the window W _j1 [p _j1 -169, p _j1 ] meets the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i1 [p _i1 -169, p _i1 ] meets the predetermined condition C ₁ In the same way, therefore, still using the matrix R, expressed as:

According to the value of a ₁ ' and the column where it is located, find the corresponding value from the matrix R, such as a ₁ '=16, and a ₁ ' is located in the first column, then find the value corresponding to h _16,1 ; according to the value of a ₂ ' value and the column where it is located, find the corresponding value from the matrix R, such as a ₂ '=98, a ₂ 'is located in the second column, then find the value corresponding to h _98,2 ; according to the value of a ₃ ' and the column where it is located , look up the corresponding value from the matrix R, such as a ₃ '=56, a ₃ 'is located in the third column, then look for the value corresponding to h _56,3 ; according to the value and column of a ₄ ', from the matrix R Find the corresponding value, such as a ₄ '=36, a ₄ ' is located in the fourth column, then find the value corresponding to h _36,4 ; according to the value of a ₅ ' and the column where it is located, find the corresponding value from the matrix R, For example, a ₅ '=99, and a ₅ 'is located in the fifth column, then search for the value corresponding to h _99,5 . S ₁ '=h _16,1 +h _98,2 +h _56,3 +h _36,4 +h _99,5 , because the matrix R obeys the binomial distribution, therefore, S ₁ ' also obeys the binomial distribution. When S ₁ 'is an even number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , and when S ₁ ' is an odd number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] Part of the data does not satisfy the predetermined condition C ₁ , and the probability that S ₁ ′ is an even number is 1/2.

The method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ is the same as the method of judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , so, as shown in Figure 17, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes, respectively Sequence numbers 170, 128, 86, 44 and 2 indicate that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 converted into a decimal value, respectively, expressed as b ₁ ', b ₂ ', b ₃ ', b ₄ ' and b ₅ '. Since 1 byte consists of 8 bits, each byte As a numerical value, b _r ' among b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ satisfies _0≤br ′≤255. b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ form a 1*5 matrix. Using the same matrix R as judging whether at least part of the data in the window W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , according to the value of b ₁ ' and the column where it is located, find the corresponding value from the matrix R, For example, b ₁ '=210, b ₁ 'is located in the first column, then find the value corresponding to h _210,1 ; according to the value of b ₂ ' and the column where it is located, find the corresponding value from the matrix R, such as b ₂ '= 156, b ₂ ' is located in the second column, then search for the value corresponding to h ₁₅₆ , 2; according to the value of b ₃ ' and the column where it is located, find the corresponding value from the matrix R, such as b ₃ '=144, b ₃ ' If it is in the third column, look for the value corresponding to h _144,3 ; look up the corresponding value from the matrix R according to the value of b ₄ ' and the column where it is located, such as b ₄ '=60, b ₄ 'is in the fourth column, Then look for the value corresponding to h _60,4 ; look up the corresponding value from the matrix R according to the value and column of b ₅ ', such as b ₅ '=90, b ₅ ' is in the fifth column, then look for h _{90, 5} corresponds to the value. S ₂ '=h _210,1 +h _156,2 +h _144,3 +h _60,4 +h _90,5 , same as S ₂ judgment conditions, when S ₂ ' is an even number, then W _j2 [p _j2 -169,p _j2 ] at least part of the data satisfies the predetermined condition C ₂ , when S ₂ ' is an odd number, then at least part of the data in W _j2 [p _j2 -169,p _j2 ] does not meet the predetermined condition C ₂ , S ₂ ' is The probability of an even number is 1/2.

Similarly, the method of judging whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ In the same way, judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , judge whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meet the predetermined condition C _5. Judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , judging Whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] satisfies the predetermined condition C ₈ , judge whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] meets the predetermined condition C ₉ , and judge whether W _j10 [p Whether at least part of the data in _j10 -169,p _j10 ] satisfies the predetermined condition C ₁₀ and whether at least part of the data in W _j11 [p _j11 -169,p _j11 ] meets the predetermined condition C ₁₁ will not be repeated here.

Taking the implementation shown in FIG. 5 as an example, a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is provided. In this embodiment, The random function judges whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] meets the predetermined condition C _z , and determines the point for the potential split point k _i according to the preset rules on the deduplication server 103 p _i1 and the window W _i1 [p _i1 -169, p _i1 ] corresponding to p _i1 , judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , as shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169,p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ , select 5 bytes, the serial number in Figure 16 The bytes "■" of 169, 127, 85, 43 and 1 respectively represent one selected byte, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are respectively converted into a decimal value, expressed as a ₁ , a ₂ , a ₃ , a ₄ and a ₅ respectively. Because a byte is composed of 8 bits, each byte "■" is used as a value, and any a _s in a ₁ , a ₂ , a ₃ , a ₄ and a ₅ satisfies 0≤a _s ≤ 255. a ₁ , a ₂ , a ₃ , a ₄ and a ₅ form a 1*5 matrix. Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix R, which is expressed as: Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix G, which is expressed as:

According to the value of a ₁ and the column where it is located, such as a ₁ = 36, and a ₁ is located in the first column, then look up the value corresponding to h _36,1 from the matrix R, and look up the value corresponding to g _36,1 from the matrix G ;According to the value of a ₂ and the column where it is located, such as a ₂ = 48, a ₂ is located in the second column, then look up the value corresponding to h _48,2 from the matrix R, and look up the value corresponding to g _48,2 from the matrix G ;According to the value of a ₃ and the column where it is located, such as a ₃ =26, a ₃ is located in the third column, then look up the value corresponding to h _26,3 from the matrix R, and look up the value corresponding to g _26,3 from the matrix G ;According to the value of a ₄ and the column where it is located, such as a ₄ =26, a ₄ is located in the fourth column, then look up the value corresponding to h _26,4 from the matrix R, and look up the value corresponding to g _26,4 from the matrix G ;According to the value of a ₅ and the column where it is located, as a ₅ =88, a ₅ is located in the fifth column, then look up the value corresponding to h _88,5 from the matrix R, and look up the value corresponding to g _88,5 from the matrix G . S _1h ＝h _36,1 +h _48,2 +h _26,3 +h _26,4 +h _88,5 , because matrix R obeys binomial distribution, therefore, S _1h also obeys binomial distribution; S _1g ＝g _36,1 +g _48,2 +g _26,3 +g _26,4 +g _88,5 , because matrix G obeys binomial distribution, so S _1g also obeys binomial distribution. When one of S _1h and S _1g is an even number, at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , and when both S _1h and S _1g are odd numbers, then W _i1 [p _i1 -169,p _i1 ] at least part of the data does not meet the predetermined condition C ₁ , and C ₁ indicates that one of S _1h and S _1g obtained by the above method is an even number. Because both S _1h and S _1g follow the binomial distribution, the probability that S _1h is even is 1/2, the probability that S _1g is even is 1/2, and the probability that one of S _1h and S _1g is even is 1 -1/4=3/4, therefore, the probability that at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ is 3/4. In the embodiment shown in FIG. 5 , at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ . _{[ _ _ _ _ _ _ _ _ _} p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169,p _i10 ], and W _i11 [p _i11 -169,p _i11 ], each window has the same size, namely The size of the window is 169 bytes. At the same time, the method of judging whether at least part of the data in the window meets the predetermined condition is the same. For details, see the above-mentioned judging whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ describe. Therefore, as shown in Figure 16, Represents the 1 byte selected when judging whether at least part of the data in the window W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C _2. In FIG. Indicates that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 respectively converted into a decimal value, respectively expressed as b ₁ , b ₂ , b ₃ , b ₄ and b ₅ . Since 1 byte consists of 8 bits, each byte As a numerical value, any b _s among b ₁ , b ₂ , b ₃ , b ₄ and b ₅ satisfies 0≤b _s ≤255. b ₁ , b ₂ , b ₃ , b ₄ and b ₅ form a 1*5 matrix. In this embodiment, the method of judging whether at least part of the data in each window satisfies the predetermined condition is the same, so the same matrices R and G are still used. According to the value of b ₁ and the column where it is located, such as b ₁ = 66, b ₁ is located in the first column, then look up the value corresponding to h _66,1 from the matrix R, and look up the value corresponding to g _66,1 from the matrix G; According to the value of b2 and the column where it is located, as b2=48, b2 is located in the _second column, then look _{up the value corresponding to h 48,2 from} matrix R, and look up the value corresponding to _{g 48,2} from matrix G; According to the value of b ₃ and the column where it is located, as b ₃ =99, b ₃ is located in the third column, then the value corresponding to h _99,3 is searched from matrix R, and the value corresponding to g _99,3 is searched from matrix G; According to the value of b ₄ and the column where it is located, as b ₄ =26, b ₄ is located in the fourth column, then look up the value corresponding to h _26,4 from the matrix R, and look up the value corresponding to g _26,4 from the matrix G; According to the value of b ₅ and the column where it is located, for example, b ₅ =90, and b ₅ is located in the fifth column, the value corresponding to h _90,5 is searched from the matrix R, and the value corresponding to g _90,5 is searched from the matrix G. S _2h =h _66,1 +h _48,2 +h _99,3 +h _26,4 +h _90,5 , because the matrix R obeys the binomial distribution, therefore, S _2h also obeys the binomial distribution. S _2g =g _66,1 +g _48,2 +g _99,3 +g _26,4 +g _90,5 , because the matrix G obeys the binomial distribution, therefore, S _2g also obeys the binomial distribution. When one of S _2h and S _2g is an even number, at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , and when both S _2h and S _2g are odd numbers, then W _i2 [p _i2 -169, p _i2 ] at least part of the data does not satisfy the predetermined condition C ₂ , and the probability that one of S _2h and S _2g is an even number is 3/4. In the embodiment shown in FIG. 5 , at least part of the data in W _i2 [p _i2 −169, p _i2 ] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , and judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] satisfies the predetermined condition C _4. Judging whether at least part of the data in W _i5 [p _i5 -169, p _i5 ] meets the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169, p _i6 ] meets the predetermined condition C ₆ , judging W Whether at least part of the data in _i7 [p _i7 -169,p _i7 ] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [p _i8 -169,p _i8 ] meets the predetermined condition C ₈ , and judge W _i9 [p _i9 -169,p _i9 ] whether at least part of the data satisfies the predetermined condition C ₉ , judging whether at least part of the data in W _i10 [p _i10 -169,p _i10 ] meets the predetermined condition C ₁₀ and judging W _i11 [p _i11 -169,p _i11 ] whether at least part of the data satisfies the predetermined condition C ₁₁ . In the embodiment shown in FIG. 5 , at least part of the data in W _i5 [p _i5 -169, p _i5 ] does not meet the predetermined condition C ₅ , jumping 11 bytes from point p _i5 along the direction of data flow splitting point search, at The end position of the 11th byte obtains the current potential segmentation point k _j , as shown in FIG. 6 , according to the preset rules on the deduplication server 103, determine the point p _j1 and the corresponding point p _j1 for the potential segmentation point k _j Window W _j1 [p _j1 -169,p _j1 ], the method of judging whether at least part of the data in window W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ is the same as judging window W _i1 [p _i1 -169,p _i1 ] whether at least part of the data satisfies the predetermined condition C ₁ in the same way, so as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], to judge W _j1 [p _j1 -169,p Whether at least part of the data in _j1 ] satisfies the predetermined condition C ₁ , the bytes "■" with serial numbers 169, 127, 85, 43 and 1 in Figure 17 represent one selected byte respectively, and two adjacent selected bytes There is a difference of 42 bytes between them. Convert the bytes "■" with sequence numbers 169, 127, 85, 43, and 1 into a decimal value, which are respectively expressed as a ₁ ', a ₂ ', a ₃ ', a ₄ ', and a ₅ '. Because 1 byte consists of 8 bits, each byte "■" is used as a value, and any a _s 'in a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' is Satisfy 0≤a _s '≤255. a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' form a 1*5 matrix. Using the same matrix R and G as judging whether at least part of the data in the window W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ is expressed as: and

According to the value of a ₁ ' and the column where it is located, such as a ₁ '=16, and a ₁ 'is located in the first column, then look up the value corresponding to h _16,1 from the matrix R, and find the corresponding value of g _16,1 from the matrix G value; according to the value of a ₂ ' and the column where it is located, such as a ₂ '=98, a ₂ 'is located in the second column, then look up the value corresponding to h ₉₈ and 2 from the matrix R, and look up g ₉₈ from the matrix G _{, the value corresponding to 2} ; according to the value of a ₃ ' and the column where it is located, such as a ₃ '=56, a ₃ 'is located in the third column, then look up the value corresponding to h _56,3 from the matrix R, and from the matrix G Find the value corresponding to g _56,3 ; according to the value of a ₄ ' and the column where it is located, such as a ₄ '=36, a ₄ ' is located in the fourth column, then find the value corresponding to h _36,4 from the matrix R, from Find the value corresponding to g _36,4 in the matrix G; according to the value of a ₅ ' and the column where it is located, such as a ₅ '=99, a ₅ ' is located in the fifth column, then find the value corresponding to h _99,5 from the matrix R value, find the value corresponding to g _99,5 from the matrix G. S _1h '=h _16,1 +h _98,2 +h _56,3 +h _36,4 +h _99,5 , because matrix R obeys binomial distribution, therefore, S _1h ' also obeys binomial distribution; S _1g '=g _16,1 +g _98,2 +g _56,3 +g _36,4 +g _99,5 , because the matrix G obeys the binomial distribution, so S _1g ' also obeys the binomial distribution. When one of S _1h ' and S _1g ' is an even number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , and when both S _1h ' and S _1g ' are odd numbers, then At least part of the data in W _j1 [p _j1 -169,p _j1 ] does not satisfy the predetermined condition C ₁ , and the probability that one of S _1h ' and S _1g ' is an even number is 3/4.

The method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ is the same as the method of judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , so, as shown in Figure 17, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. In FIG. 17, they are represented by sequence numbers 170, 128, 86, 44 and 2 respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 converted into a decimal value, respectively, expressed as b ₁ ', b ₂ ', b ₃ ', b ₄ ' and b ₅ '. Since 1 byte consists of 8 bits, each byte As a numerical value, any one of b _s ' among b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ satisfies 0≤b _s '≤255. b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ form a 1*5 matrix. Using the same matrix R and G as judging whether at least part of the data in the window W _i2 [p _i2 -169, p _i2 ] meets the predetermined condition C ₂ , according to the value of b ₁ ' and the column where it is located, such as b ₁ '=210, b ₁ ' is located in the first column, then look up the value corresponding to h _210,1 from the matrix R, and find the value corresponding to g _210,1 from the matrix G; according to the value of b ₂ ' and the column where it is located, such as b ₂ ' =156, b ₂ 'is in the second column, then look up the value corresponding to h _156,2 from the matrix R, and look up the value corresponding to g _156,2 from the matrix G; according to the value of b ₃ ' and the column where it is located, such as b ₃ '=144, b ₃ 'is located in the third column, then look up the value corresponding to h _144,3 from the matrix R, and look up the value corresponding to g _144,3 from the matrix G; according to the value of b ₄ ' and where column, such as b ₄ '=60, b ₄ 'is located in the fourth column, then look up the value corresponding to h _60,4 from the matrix R, and look up the value corresponding to g _60,4 from the matrix G; according to the value of b ₅ ' and the column where it is located, such as b ₅ '=90, and b ₅ 'is located in the fifth column, then look up the value corresponding to h _90,5 from the matrix R, and look up the value corresponding to g _90,5 from the matrix G. S _2h '=h _210,1 +h _156,2 +h _144,3 +h _60,4 +h _90,5 , S _2g '=g _210,1 +g _156,2 +g _144,3 +g _{60 ,4} +g _90,5 . When one of S _2h ' and S _2g ' is an even number, then at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , and when both S _2h ' and S _2g ' are odd numbers, then At least part of the data in W _j2 [p _j2 -169,p _j2 ] does not satisfy the predetermined condition C ₂ , and the probability that one of S _2h ' and S _2g ' is an even number is 3/4.

Similarly, the method of judging whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ In the same way, judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , judge whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meet the predetermined condition C _5. Judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , judging Whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] satisfies the predetermined condition C ₈ , judge whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] meets the predetermined condition C ₉ , and judge whether W _j10 [p Whether at least part of the data in _j10 -169,p _j10 ] satisfies the predetermined condition C ₁₀ and whether at least part of the data in W _j11 [p _j11 -169,p _j11 ] meets the predetermined condition C ₁₁ will not be repeated here.

Taking the implementation shown in FIG. 5 as an example, a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is provided. In this embodiment, The random function judges whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] meets the predetermined condition C _z , and determines the point for the potential split point k _i according to the preset rules on the deduplication server 103 p _i1 and the window W _i1 [p _i1 -169, p _i1 ] corresponding to p _i1 , judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , as shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169,p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , select 5 bytes, the sequence number in Figure 16 The bytes "■" of 169, 127, 85, 43 and 1 respectively represent one selected byte, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are regarded as 40 bits in turn, which are expressed as a ₁ , a ₂ , a ₃ , a ₄ ...a ₄₀ respectively. For any at of a ₁ , a ₂ , a ₃ , a ₄ ...a ₄₀ , when _at = 0, V _at = _-1 , when _at = 1, V _at = 1, according to _at and _Vat correspondence relationship generates V _a1 , V _a2 , V _a3 , V _a4 . . . V _a40 . Select 40 random numbers from the random numbers subject to the normal distribution, respectively denoted as: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ . S _a =V _a1 *h ₁ +V _a2 *h ₂ +V _a3 *h ₃ +V _a4 *h ₄ + . . . +V _a40 *h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey the normal distribution, therefore, S _a also obeys the normal distribution. When S _a is a positive number, at least part of the data in W _i1 [p _i1 -169,p _i1 ] meets the predetermined condition C ₁ , and when S _a is negative or 0, then in W _i1 [p _i1 -169,p _i1 ] At least part of the data does not satisfy the predetermined condition C ₁ , and the probability that S _a is a positive number is 1/2. In the embodiment shown in FIG. 5 , at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ . As shown in Figure 16, Represents the 1 byte selected when judging whether at least part of the data in the window W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C _2. In FIG. Indicates that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 They are regarded as 40 bits in turn, represented as b ₁ , b ₂ , b ₃ , b ₄ . . . b ₄₀ . For any b _t in b ₁ , b ₂ , b ₃ , b ₄ ... b ₄₀ , when b _t =0, V _bt =-1, when b _t =1, V _bt =1, according to b _t and The V _bt correspondence relationship generates V _b1 , V _b2 , V _b3 , V _b4 . . . V _b40 . The method of judging whether at least part of the data in the window W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i2 [p _i2 -169, p _i2 ] meets the predetermined condition C ₂ In the same way, therefore, using the same random numbers: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ , S _b = V _b1 *h ₁ +V _b2 *h ₂ +V _b3 *h ₃ +V _b4 *h ₄ +...+V _b40 *h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _b also obeys normal distribution. When S _b is a positive number, at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ , and when S _b is negative or 0, then in W _i2 [p _i2 -169,p _i2 ] At least part of the data does not satisfy the predetermined condition C ₂ , and the probability that S _b is a positive number is 1/2. In the embodiment shown in FIG. 5 , at least part of the data in W _i2 [p _i2 −169, p _i2 ] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ , and judge whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] satisfies the predetermined condition C _4. Judging whether at least part of the data in W _i5 [p _i5 -169, p _i5 ] meets the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [p _i6 -169, p _i6 ] meets the predetermined condition C ₆ , judging W Whether at least part of the data in _i7 [p _i7 -169,p _i7 ] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [p _i8 -169,p _i8 ] meets the predetermined condition C ₈ , and judge W _i9 [p _i9 -169,p _i9 ] whether at least part of the data satisfies the predetermined condition C ₉ , judging whether at least part of the data in W _i10 [p _i10 -169,p _i10 ] meets the predetermined condition C ₁₀ and judging W _i11 [p _i11 -169,p _i11 ] whether at least part of the data satisfies the predetermined condition C ₁₁ . In the embodiment shown in FIG. 5 , at least part of the data in W _i5 [p _i5 -169, p _i5 ] does not meet the predetermined condition C ₅ , jumping 11 bytes from point p _i5 along the direction of data flow splitting point search, at The end position of the 11th byte obtains the current potential segmentation point k _j , as shown in FIG. 6 , according to the preset rules on the deduplication server 103, determine the point p _j1 and the corresponding point p _j1 for the potential segmentation point k _j Window W _j1 [p _j1 -169,p _j1 ], the method of judging whether at least part of the data in window W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ is the same as judging window W _i1 [p _i1 -169,p _i1 ] whether at least part of the data satisfies the predetermined condition C ₁ is the same way, so as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], to judge W _j1 [p _j1 -169,p Whether at least part of the data in _j1 ] meets the predetermined condition C ₁ , in order to judge whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , select 5 bytes, and the serial numbers in Figure 17 are 169, The byte "■" of 127, 85, 43 and 1 represents one selected byte respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are regarded as 40 bits in sequence, which are represented as a ₁ ', a ₂ ', a ₃ ', a ₄ '...a ₄₀ ' respectively. For any a _t 'in a ₁ ', a ₂ ', a ₃ ', a ₄ '...a ₄₀ ', when a _t '=0, V _at '=-1, when a _t '=1, V _at '=1, V _a1 ', V _a2 ', V _a3 ', V _a4 '...V _a40 ' _are generated according to the correspondence between at ' and V _at '. The method of judging whether at least part of the data in the window W _j1 [p _j1 -169, p _j1 ] satisfies the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ In the same way, so the same random numbers are used: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ . S _a '=V _a1 '*h ₁ +V _a2 '*h ₂ +V _a3 '*h ₃ +V _a4 '*h ₄ +...+V _a40 '*h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _a ' also obeys normal distribution. When S _a 'is a positive number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] meets the predetermined condition C ₁ , and when S _a ' is negative or 0, then W _j1 [p _j1 -169,p _j1 ] at least part of the data does not meet the predetermined condition C ₁ , the probability that S _a ' is a positive number is 1/2.

The method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ is the same as the method of judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , so, as shown in Figure 17, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. In FIG. 17, they are represented by sequence numbers 170, 128, 86, 44 and 2 respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 It is regarded as 40 bits in turn, which are respectively expressed as b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′…b ₄₀ ′. Any b _t 'in b ₁ ', b ₂ ', b ₃ ', b ₄ '...b ₄₀ ', when b _t '=0, V _bt '=-1, when b _t '=1, V _bt '=1, V _b1 ', V _b2 ', V _b3 ', V _b4 '...V _b40 ' are generated according to the corresponding relationship between b _t ' and V _bt '. The method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] satisfies the predetermined condition C ₂ is the same as the method of judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] satisfies the predetermined condition C ₂ , therefore, using the same random numbers: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ , S _b '=V _b1 '*h ₁ +V _b2 '*h ₂ +V _b3 '*h ₃ +V _b4 '*h ₄ +...+V _b40 '*h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _b ' also obeys normal distribution. When S _b ' is a positive number, then at least part of the data in W _j2 [p _j2 -169,p _j2 ] meets the predetermined condition C ₂ , and when S _b ' is negative or 0, then W _j2 [p _j2 -169,p _j2 ] at least part of the data does not meet the predetermined condition C ₂ , the probability that S _b ' is a positive number is 1/2.

Similarly, the method of judging whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C ₃ In the same way, judge whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , judge whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meet the predetermined condition C _5. Judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , judging Whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , judge whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] meet the predetermined condition C ₉ , and judge whether W _j10 [p Whether at least part of the data in _j10 -169,p _j10 ] satisfies the predetermined condition C ₁₀ and whether at least part of the data in W _j11 [p _j11 -169,p _j11 ] meets the predetermined condition C ₁₁ will not be repeated here.

Still taking the embodiment shown in FIG. 5 as an example, a method for judging whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z is provided, which is used in this embodiment The random function judges whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] meets the predetermined condition C _z , and determines the point for the potential split point k _i according to the preset rules on the deduplication server 103 The window W _i1 [p _i1 -169, p _i1 ] corresponding to p _i1 and p _i1 , judge whether at least part of the data in W _i1 [p _i1 -169, p _i1 ] satisfies the predetermined condition C ₁ , as shown in Figure 16, W _i1 represents the window W _i1 [p _i1 -169,p _i1 ], in order to judge whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ , select 5 bytes, and the serial number in Figure 16 is The byte "■" of 169, 127, 85, 43 and 1 represents one selected byte respectively, and the difference between two adjacent selected bytes is 42 bytes. Convert the byte "■" with serial numbers 169, 127, 85, 43 and 1 into a decimal number in the range of 0-(2^40-1), using a uniformly distributed random number generator as 0-(2^ Each decimal number in 40-1) generates a specified value, and records the correspondence R between each decimal number in 0-(2^40-1) and the specified value. Once specified, the corresponding decimal number The specified value remains unchanged, and the specified value obeys the uniform distribution. If the specified value is even, then at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ . If the specified value is odd, then At least part of the data in W _i1 [p _i1 -169, p _i1 ] does not satisfy the predetermined condition C ₁ , and C ₁ indicates that the specified value obtained by the above method is an even number. Because the probability that a uniformly distributed random number is an even number is 1/2, therefore, the probability that at least part of the data in [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ is 1/2. In the embodiment shown in FIG. 5 , the same rule is used to determine whether at least part of the data in W _i2 [p _i2 -169, p _i2 ] satisfies the predetermined condition C ₂ , and to determine whether W _i3 [p _i3 -169, p _i3 ] whether at least part of the data in W i4 [p i4 -169,p i4 ] satisfies the predetermined condition C ₃ , whether at least part of the data in W _i4 [p _i4 -169,p _i4 ] meets the predetermined condition C ₄ , and whether at least part of the data in W _i5 [p _i5 -169,p _i5 ] Whether the data satisfies the predetermined condition C ₅ will not be repeated here.

When at least part of the data in W _i5 [p _i5 -169,p _i5 ] does not meet the predetermined condition C ₅ , jump 11 bytes from point p _i5 along the direction of data flow splitting point search, at the end position of the 11th byte Obtain the current potential segmentation point k _j , as shown in FIG. 6 , according to the preset rules on the deduplication server 103, determine the point p _j1 and the window W _j1 [p _j1 -169 corresponding to the point p _j1 ] for the potential segmentation point k _j ,p _j1 ], the method of judging whether at least part of the data in the window W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ is the same as judging whether at least part of the data in the window W _i1 [p _i1 -169,p _i1 ] satisfies The way of the predetermined condition C ₁ is the same, therefore, use the correspondence R between each decimal number in the same 0-(2^40-1) and the specified value, as shown in Figure 17, W _j1 represents the window W _j1 [p _j1 -169,p _j1 ], in order to judge whether at least part of the data in W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ , select 5 bytes, "■" in Figure 17 indicates the selected 1 bytes, and the difference between two adjacent selected bytes "■" is 42 bytes. Convert the byte "■" with serial numbers 169, 127, 85, 43 and 1 into a decimal number, and find the specified value corresponding to the decimal number in R. If the specified value is even, then W _j1 [p _j1 - 169,p _j1 ] at least part of the data satisfies the predetermined condition C ₁ , if the specified value is an odd number, then at least part of the data in W _j1 [p _j1 -169,p _j1 ] does not meet the predetermined condition C ₁ , because the random The probability that the number is even is 1/2, therefore, the probability that at least part of the data in W _j1 [p _j1 -169,p _j1 ] satisfies the predetermined condition C ₁ is 1/2. Similarly, the method of judging whether at least part of the data in W _i2 [p _i2 -169,p _i2 ] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [p _j2 -169,p _j2 ] meet the predetermined condition C ₂ The way of judging whether at least part of the data in W _i3 [p _i3 -169,p _i3 ] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [p _j3 -169,p _j3 ] satisfies the predetermined condition C The method of ₃ is the same, similarly, judging whether at least part of the data in W _j4 [p _j4 -169,p _j4 ] meets the predetermined condition C ₄ , judging whether at least part of the data in W _j5 [p _j5 -169,p _j5 ] meets the predetermined condition Condition C ₅ , judging whether at least part of the data in W _j6 [p _j6 -169,p _j6 ] meets the predetermined condition C ₆ , judging whether at least part of the data in W _j7 [p _{j7 -169} ,p _j7 ] meets the predetermined condition C ₇ , Judging whether at least part of the data in W _j8 [p _{j8 -169} ,p _j8 ] meets the predetermined condition C ₈ , judging whether at least part of the data in W _j9 [p _j9 -169,p _j9 ] meets the predetermined condition C ₉ , judging W _j10 [ Whether at least part of the data in p _j10 -169,p _j10 ] satisfies the predetermined condition C ₁₀ and whether at least part of the data in W _j11 [p _j11 -169,p _j11 ] meets the predetermined condition C ₁₁ will not be repeated here.

The deduplication server 103 in the embodiment of the present invention shown in Figure 1 refers to the device capable of implementing the technical solution described in the embodiment of the present invention, as shown in Figure 18, generally includes a central processing unit, a main memory, and an input and output interface . The central processing unit, the main memory, and the input and output interfaces communicate with each other, the main memory stores executable instructions, and the central processing unit executes the executable instructions stored in the main memory to perform specific functions, as shown in Figure 4 to Figure 4 of the embodiment of the present invention 17 to find data stream split points as described. Therefore, as shown in FIG. 19, according to the embodiment of the present invention shown in FIGS. point p _x , the window W _x [p _x -A _x ,p _x +B _x ] corresponding to the point p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], Wherein, x is a continuous natural number from 1 to M, M≥2, and A _x and B _x are integers; the deduplication server 103 includes a determination unit 1901 and a judgment processing unit 1902 . Wherein, the determination unit 1901 is configured to perform step a): a) determine a point p _iz and a window W _iz corresponding to the point p _iz for the current potential segmentation point k _i according to the rule [p _iz -A _z , p _iz +B _z ], i and z are integers, and 1≤z≤M; judging processing unit 1902, used for at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] Satisfy the predetermined condition C _z ;

When at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , jump N times from the point p _iz along the direction of searching the data stream splitting point The minimum search unit U of the data flow segmentation point, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖+‖(k _i -p _ix )‖), to obtain a new potential segmentation point, then the determination unit Execute step a) for the new potential segmentation point; when at least part of the data in each window W _ix [p _ix -A _x , p _ix +B _x ] of the M windows of the current potential segmentation point _ki If the predetermined condition C _x is met, the current potential split point _ki is a data stream split point.

Further, the rule further includes: at least two points p _e and p _f satisfy the conditions of A _e =A _f , B _e =B _f , and C _e =C _f . Further, the rule further includes: the at least two points _pe and p _f are in the reverse direction of the data flow split point search relative to the potential split point k.

Further, the rule further includes: the distance between the at least two points _pe and p _f is 1 U.

Further, the determination processing unit 1902 is specifically configured to use a random function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z . Specifically, the determination processing unit 1902 is specifically configured to use a hash function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z . Specifically, the determination processing unit 1902 is specifically configured to use a random function to determine whether at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [p _iz -A _z ,p _iz +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, which is specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] satisfies the predetermined condition C _z .

Further, the judgment processing unit 1902 is configured to, when at least part of the data in the window W _iz [p _iz -A _z ,p _iz +B _z ] does not satisfy the predetermined condition C _z , start from the point p _iz along the The data stream segmentation point search direction jumps N minimum search units U of data stream segmentation points to obtain the new potential segmentation point, and the determining unit 1901 performs step a) for the new potential segmentation point, according to the rule, the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] corresponding to the point p _ic determined for the new potential segmentation point is the same as the window W _iz [p _iz -A _z ,p _iz +B _z ] coincides or the left boundary of the window W _ic [p _ic -A _c ,p _ic +B _c ] determined for the new potential segmentation point is located at the window W _iz [p _iz -A _z ,p _iz +B _z ]; wherein, the window W _ic [p _ic -A _c ,p _ic +B _c ] determined for the new potential segmentation point is according to the rule, The M points determined for the new potential segmentation point are the points ranked first in the sequence obtained according to the search direction of the data flow.

According to the server-based method for _searching data _{stream segmentation points provided by} the embodiments of the present invention shown in FIGS. , p _ix +B _x ], where x is a continuous natural number from 1 to M, and M≥2, it can be judged in parallel whether at least part of the data in each of the M windows satisfies the predetermined condition C _x , or it can be judged sequentially in the windows Whether at least part of the data meets the predetermined condition can also be judged when at least part of the data in the window W _i1 [p _i1 -A ₁ , p _i1 +B ₁ ] meets the predetermined condition C ₁ , and then judge W _i2 [p _i2 -A ₂ , p When at least part of the data in _i2 +B ₂ ] satisfies the predetermined condition C ₂ , until it is determined that at least part of the data in W _im [p _im -A _m , p _im +B _m ] satisfies the predetermined condition C _m . The determination of other windows in the embodiment is the same as this, and will not be repeated here.

In addition, according to the embodiment of the present invention shown in FIG. 4 to FIG. 17 , in practical applications, rules are preset on the deduplication server 103, and the rules are: determine M points p _x , point The window W _x [p _x -A _x ,p _x +B _x ] corresponding to p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], x are 1 to M respectively Continuous natural numbers, M≥2, in this default rule, A ₁ , A ₂ , A ₃ ... A _m may not all be equal, B ₁ , B ₂ , B ₃ ... B _m may not all be equal, C ₁ , C ₂ , C ₃ . . . C _M may not all be the same. In the embodiment shown in FIG. 5 , in windows W _i1 [p _i1 -169,p _i1 ], W _i2 [p _i2 -169,p _i2 ], W _i3 [p _i3 -169,p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _{In i8} -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169,p _i10 ] and W _i11 [p _i11 -169,p _i11 ], each window has the same size, That is, the size of the window is 169 bytes. At the same time, the method of judging whether at least part of the data in the window meets the predetermined condition is the same. For details, see the above judgment of whether at least part of the data in W _i1 [p _i1 -169,p _i1 ] satisfies the predetermined condition C ₁ , but in the embodiment shown in Figure 11, W _i1 [p _i1 -169, p _i1 ], W _i2 [p _i2 -169, p _i2 ], W _i3 [p _i3 -169, p _i3 ], W _i4 [p _i4 -169,p _i4 ], W _i5 [p _i5 -169,p _i5 ], W _i6 [p _i6 -169,p _i6 ], W _i7 [p _i7 -169,p _i7 ], W _i8 [p _i8 -169,p _i8 ], W _i9 [p _i9 -169,p _i9 ], W _i10 [p _i10 -169,p _i10 ] and W _i11 [p _i11 -182,p _i11 ] window sizes can be different , and at the same time, the manner of judging whether at least part of the data in the window satisfies the predetermined condition may also be different. In all embodiments, according to the preset rules on the deduplication server 103, the method of judging whether at least part of the data in the window W _i1 satisfies the predetermined condition _C1 is the same as the method of judging whether at least part of the data in the window W _j1 satisfies the predetermined condition _C1 The method must be the same, the method of judging whether at least part of the data in W _i2 satisfies the predetermined condition _C2 must be the same as the method of judging whether at least part of the data in W _j2 satisfies the predetermined condition _C2 ...judging whether at least part of the data in the window W _iM satisfies the predetermined condition The way of C _M is necessarily the same as the way of judging whether at least part of the data in the window _WjM satisfies the predetermined condition C _M. No more details here, and according to the embodiments of the present invention shown in Figure 4 to Figure 17, although M=11 is taken as an example, according to actual needs, the value of M is not limited to 11, those skilled in the art The description in the embodiment of the invention determines the value of M.

According to the embodiment of the present invention shown in FIG. 4 to FIG. 17 , there are preset rules on the deduplication server 103, k _a , k _i , k _j , k _l and k _m are the search division along the direction of data flow segmentation point search. The potential segmentation points obtained at point k _a , k _i , k _j , k _l and k _m all follow this rule. The window W _x [p _x -A _x ,p _x +B _x ] in the embodiment of the present invention represents a specific range, in which data is selected to determine whether the data meet the predetermined condition C _x , specifically, it can be Select part of the data within a specific range, or select all the data to determine whether these data meet the predetermined condition C _x . For the window concept specifically used in the embodiment of the present invention, reference may be made to window W _x [p _x -A _x , p _x +B _x ], which will not be repeated here.

According to the embodiment of the present invention shown in Fig. 4 to Fig. 17, in the window W _x [p _x -A _x ,p _x +B _x ], (p _x -A _x ) and (p _x +B _x ) represent the window Two boundaries of W _x [p _x -A _x ,p _x +B _x ], where (p _x -A _x ) means window W _x [p _x -A _x ,p _x +B _x ] relative to point p _x The boundary located in the direction opposite to the data stream split point search, (p _x +B _x ) means the boundary of the window W _x [p _x -A _x ,p _x +B _x ] located in the direction of the data stream split point search relative to point p _x . Specifically, in the embodiment of the present invention, the search direction of the data stream segmentation point shown in Fig. 3 to Fig. 15 is from left to right, where (p _x -A _x ) represents the window W _x [p _x -A _x ,p _x +B _x ] relative to the point p _x is located at the boundary in the opposite direction of the data flow split point (ie the left boundary), (p _x +B _x ) means the window W _x [p _x -A _x ,p _x +B _x ] relative to the point p _x is located at the boundary (ie, the right boundary) of the search direction of the data flow split point. If the search direction of the data flow splitting point shown in Figure 3 to Figure 15 is from right to left, then (p _x -A _x ) represents the window W _x [p _x -A _x ,p _x +B _x ] relative to The point p _x is located at the boundary of the opposite direction of the data flow split point search (ie the right boundary), (p _x +B _x ) means that the window W _x [p _x -A _x ,p _x +B _x ] is located at the data stream relative to the point p _x The boundary of the stream split point lookup direction (i.e. the left boundary).

Those skilled in the art can realize that, in combination with the units and algorithm steps of the examples described in the embodiments of the present invention, the key features of the embodiments of the present invention can be combined with other technologies and presented in a more complex form, but still include Key features of the invention. It is possible to use alternate split points in a real environment. For example, one implementation is to determine 11 points p _x for potential split points _ki according to the preset rules on the deduplication server 103, where x is a continuous natural number from 1 to 11 , determine the window W _x [p _x -A _x ,p _x +B _x ] corresponding to p _x and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], when 11 windows At least part of the data in each window W _x [p _x -A _x ,p _x +B _x ] in each window satisfies the predetermined condition C _x , then the potential split point _ki is the data stream split point, when the set maximum data block is exceeded When the split point is not found yet, a spare preset rule may be used at this time. The spare preset rule is similar to the preset rule on the deduplication server 103. The spare preset rule is: for example, a potential split point k _i Determine 10 points p _x , x is a continuous natural number from 1 to 10, determine the window W _x [p _x -A _x ,p _x +B _x ] and the window W _x [p _x -A _x ,p corresponding to p _{x x} +B _x ] corresponding to the predetermined condition C _x , when at least part of the data in each of the 10 windows W _x [p _x -A _x ,p _x +B _x ] satisfies the predetermined condition C _x , the potential segmentation point k _i is the data stream split point. When the data stream split point is not found when the set maximum data block is exceeded, the end position of the largest data block is used as the mandatory split point.

There are preset rules on the deduplication server 103, in which M points are determined for the potential segmentation point k, and a potential segmentation point k is not necessarily required first, and the potential segmentation point k can be judged by the determined M points .

An embodiment of the present invention provides a method for finding a data stream segmentation point based on a deduplication server, as shown in FIG. 20 , including:

A rule is preset on the deduplication server 103, and the rule is: determine M windows W _x [kA _x , k+B _x ] corresponding to windows W _x [kA _x , k+B _x ] for a potential segmentation point k The predetermined condition C _x , wherein, x is a continuous natural number from 1 to M, M≥2, A _x and B _x are integers; in the embodiment shown in Figure 3, regarding the value of M, one of the realizations In this way, the value of M*U is not greater than the preset maximum distance between two adjacent data stream segmentation points, that is, the preset maximum length of the data block. Determine whether at least part of the data in the window W _z [kA _z , k+B _z ] satisfies the predetermined condition C _z , where z is an integer, 1≤z≤M, (kA _z ) and (k+B _z ) respectively represent the window The two boundaries of W _z . When it is judged that at least part of the data in any window W _z [kA _z , k+B _z ] does not meet the predetermined condition C _z , jump N bytes from the potential split point k along the search direction of the data stream split point, N≤‖B _z ‖+max _x (‖A _x ‖). Among them, ‖B _z ‖ represents the absolute value of B _z in W _z [kA _z , k+B _z ], max _x (‖A _x ‖) represents the maximum value of the absolute value of A _x in M windows, which will be shown below The principle of selecting the value of N is specifically introduced in the embodiment. When it is determined that at least part of the data in each of the M windows W _x [kA _x , k+B _x ] satisfies the predetermined condition C _x , then the potential split point k is a data stream split point.

Specifically, for the current potential segmentation point k _i , according to the rules, the following steps are performed:

Step 2001: Determine the corresponding window W _iz [k _i -A _z , _ki +B _z ] for the current potential segmentation point ki according to the rules, _i and z are integers, and 1≤z≤M;

Step 2002: judging whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip N data stream segmentation point minimum search unit U, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), obtain a new potential segmentation point, execute step 2001;

When at least part of the data in each of the M windows W _ix [k _i -A _x , _ki +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential segmentation point k _i is the data stream split point.

Further, the rule also includes: at least two windows W _ie [k _i -A _e , _ki +B _e ] and W _if [k _i -A _f , _ki +B _f ], satisfying the condition: |A _e +B _e |=|A _f +B _f |, C _e =C _f ; further, the rule also includes: A _e and A _f are positive integers; further, the rule also includes: A _e -1=A _f , _Be +1=B _f . Among them, |A _e +B _e | represents the size of the window W _ie , and |A _f +B _f | represents the size of the window W _if .

Further, judging whether at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] satisfies the predetermined condition C _z specifically includes: using a random function to judge whether the window W _iz [k _i Whether at least part of the data in -A _z , _ki +B _z ] satisfies the predetermined condition C _z ; further, the use of a random function to judge the window W _iz [ _ki -A _z , _ki +B _z ] whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z , specifically using a hash _function .

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip the minimum search unit U of N data stream segmentation points to obtain the new potential segmentation point, and according to the rules, the window W _ic [k _i -A _c , _ki +B determined for the new potential segmentation point _c ] is coincident with the right boundary of the window W _iz [k _i -A _z , _ki +B _z ] or the window W _ic [k _i -A _c , _ki +B _c ] the left boundary of the window W _iz [k _i -A _z , _ki +B _z ]; wherein, the window W _ic determined for the new potential segmentation point [k _i -A _c , _ki +B _c ] is the first window in the sequence obtained according to the search direction of the data flow among the M windows determined for the new potential segmentation point according to the rule.

In the embodiment of the present invention, the data flow segmentation point is searched by judging whether at least part of the data in one of the M windows satisfies the predetermined condition, and when at least part of the data in a certain window does not meet the predetermined condition, skip N*U length, wherein, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖), to obtain the next potential segmentation point, which improves the efficiency of searching for data stream segmentation points.

In the deduplication process, in order to ensure uniform data block size, the average data block (also called average block size) size will be considered, that is, the average data block size will be determined while meeting the minimum data block size and maximum data block size restrictions. Block size to ensure that the obtained data blocks are of uniform size. The number M of the window W _x [kA _x , k+B _x ] and the probability that at least part of the data in the window W _x [kA _x , k+B _x ] meet the preset conditions determine the finding of the data stream segmentation point The probability of (denoted by P(n)), the former affects the length of the jump, the latter affects the probability of the jump, and both affect the average block size. Generally speaking, when the average block size is fixed and the number of W _x [kA _x , k+B _x ] increases, the probability that at least part of the data in a single window W _x [kA _x , k+B _x ] satisfies the predetermined condition Also increase, for example, there are rules preset on the deduplication server 103, the rules are: determine 11 windows W _x [kA _x , k+B _x ] for the potential segmentation point k, and x are respectively 1 to 11 consecutive natural numbers , the probability that at least part of the data in any one of the 11 windows W _x [kA _x , k+B _x ] satisfies the preset condition is 1/2. Another set of rules preset on the deduplication server 103 is: determine 24 windows W _x [kA _x , k+B _x ] for the potential segmentation point k, x is a continuous natural number from 1 to 24, and 24 windows The probability that at least part of the data in any window W _x [kA _x , k+B _x ] meets the preset conditions is 3/4, and at least part of the data in the specific window W _x [kA _x , k+B _x ] meets the preset conditions For setting the probability of , please refer to the description of judging whether at least part of the data in the window W _x [kA _x , k+B _x ] satisfies the preset condition. The number M of the window W _x [kA _x , k+B _x ] and the probability that at least part of the data in the window W _x [kA _x , k+B _x ] satisfy the preset conditions determine P(n), P(n) represents: the probability that no data stream split point is found after searching n minimum search units of data stream split points from the start position of the data stream or from the previous data stream split point. The calculation process of determining P(n) by these two factors is actually a multi-step Fibonacci sequence, which will be described in detail later. After obtaining P(n), 1-P(n) is the distribution function of the data stream segmentation point, (1-P(n))-(1-P(n-1))=P(n-1)- P(n) is the probability of finding the data stream split point at the minimum search unit of n data stream split points, that is, the density function of the data stream split point, which can be integrated according to the density function of the data stream split point Thereby, the expected length of the split point of the data stream is obtained, that is, the average block size, wherein 4*1024 (bytes) represents the minimum data block length, and 12*1024 (bytes) represents the maximum data block length.

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 21, a rule is preset on the deduplication server 103, and the rule is: determine 11 windows for the potential segmentation point k W _x [kA _x , k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ], wherein x is a continuous natural number from 1 to 11, and A _x and B _x are integers. Among them, A ₁ =169, B ₁ =0; A ₂ =170, B ₂ =-1; A ₃ =171, B ₃ =-2; A ₄ =172, B ₄ =-3; A ₅ =173, B ₅ =-4; A ₆ =174, B ₆ =-5; A ₇ =175, B ₇ =-6; A ₈ =176, B ₈ =-7; A ₉ =177, B ₉ =-8; A ₁₀ =178, B ₁₀ =-9; A ₁₁ =179, B ₁₁ =-10, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ = C ₁₀ ＝C ₁₁ , then the 11 windows are W ₁ [k-169,k], W ₂ [k-170,k-1], W ₃ [k-171,k-2], W ₄ [k -172,k-3], W ₅ [k-173,k-4], W ₆ [k-174,k-5], W ₇ [k-175,k-6], W ₈ [k-176 ,k-7], W ₉ [k-177,k-8], W ₁₀ [k-178,k-9] and W ₁₁ [k-179,k-10]. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 21 is from left to right. After skipping the minimum data block 4KB from the data stream split point k _a , the minimum data block 4KB end position is taken as the next potential For split point k _i , according to the preset rules for the deduplication server 103, determine the window W _ix [k _i -A _x , k _i +B _x ] for the potential split point k _i , in this embodiment, x is 1 respectively to 11 consecutive natural numbers. In the embodiment shown in FIG. 21 , there are 11 windows determined for the potential segmentation point ki, which are respectively W _i1 [k _i -169, _ki ], W _i2 [ _{k i} -170, _ki -1] , W _i3 [k _i -171,k _i -2], W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4], W _i6 [k _i -174 ,k _i -5], W _i7 [k _i -175,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177,k _i -8], W _i10 [k _i -178, _ki -9] and W _i11 [k _i -179, _ki -10]. Judging whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _i3 [k _i -171, _ki -2] meets the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [k _i -172, _ki -3] meet the predetermined condition C ₄ , and judge W Whether at least part of the data in _i5 [k _i -173, _ki -4] meets the predetermined condition C ₅ , judge whether at least part of the data in W _i6 [k _i -174, _ki -5] meet the predetermined condition C ₆ , and judge W Whether at least part of the data in _i7 [k _i -175, _ki -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -176, _ki -7] meet the predetermined condition C ₈ , and judge W Whether at least part of the data in _i9 [k _i -177, _ki -8] meets the predetermined condition C ₉ , judge whether at least part of the data in W _i10 [k _i -178, _ki -9] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _i11 [k _i -179, _ki -10] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined conditions, as shown in Figure 22, W _i5 [k _i -173, _ki -4], then from the potential segmentation point _ki along the data The search direction of the stream split point jumps N bytes, wherein N bytes are not greater than ‖B ₅ ‖+max _x (‖A _x ‖), and in the embodiment shown in Figure 22, the jump N bytes are not greater than 183 bytes. In this embodiment, N=7 to obtain a new potential segmentation point. In order to distinguish it from the potential segmentation point ki, the new potential segmentation point is denoted as _{k j here} . According to the embodiment shown in FIG. 21 , a rule is preset on the deduplication server 103, and the rule is: determine a window W _jx [k _j -A _x , k _j +B _x ] for a potential segmentation point k _j , In this embodiment, x are consecutive natural numbers from 1 to 11, respectively. There are 11 windows determined for the potential segmentation point k _j , which are respectively W _j1 [k _j -169,k _j ], W _j2 [k _j -170,k _j -1], W _j3 [k _j -171,k _j -2], W _j4 [k _j -172,k _j -3], W _j5 [k _j -173,k _j -4], W _j6 [k _j -174,k _j -5], W _j7 [ k _j -175,k _j -6], W j8 [k _{j -176} ,k _j -7], W _j9 [k _j -177,k _j -8], W _j10 [k _j -178,k _j - 9] and W _j11 [k _j -179, k _j -10]. As shown in Figure 22, the 11th window W _j11 [k _j -179, k _j -10] determined for the potential segmentation point is guaranteed to be within the range between the potential segmentation point k _i and the potential segmentation point k _j range, then in this embodiment, it must be ensured that the left boundary of window W _j11 [k _j -179, k _j -10] and the right boundary of window W _i5 [k _i -173, k _i -4] (k _i -4) coincides, or is located within the range of window W _i5 [k _i -173, _ki -4], the window W _j11 [k _j -179, k _j -10] is based on the rule and is The M _windows determined by the above-mentioned potential segmentation point kj are the first-ranked windows in the sequence obtained according to the search direction of the data flow. Therefore, within this limit, when at least part of the data in the window W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , the jumping distance from the potential split point _ki along the direction of data stream split point search Not greater than ‖B ₅ ‖+max _x (‖A _x ‖). Judging whether at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _j3 [k _j -171,k _j -2] meets the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [k _j -172,k _j -3] meet the predetermined condition C ₄ , judge W Whether at least part of the data in _j5 [k _j -173, k _j -4] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174, k _j -5] meet the predetermined condition C ₆ , judge W Whether at least part of the data in _j7 [k _j -175, k _j -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W j8 [k _{j -176} , k _j -7] meet the predetermined condition C ₈ , and judge W Whether at least part of the data in _j9 [k _j -177, k _j -8] meets the predetermined condition C ₉ , judge whether at least part of the data in W _j10 [k _j -178, k _j -9] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _j11 [k _j -179, k _j -10] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _i is the data flow split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data flow split point. When it is judged that the potential split point k _j is not a data flow split point, the next potential split point is obtained in the same manner as k _i , and the next potential split point is judged according to the preset rules on the deduplication server 103 and the above method. Split point for data flow. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the forced split point.

In the embodiment shown in FIG. 21 , according to the preset rules on the deduplication server 103, it starts from judging whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , when judging W _i1 [k _i -169,k _i ], W _i2 [k _i -170,k _i -1], W _i3 [k _i -171,k _i -2] and W _i4 [k _i -172,k _i -3] At least part of the data in at least part of the data meet the predetermined conditions C ₁ , C ₂ , C ₃ and C ₄ respectively, and it is judged that at least part of the data in W _i5 [ _ki -173, _ki -4] does not meet the predetermined condition C ₅ , jump 6 bytes from the potential segmentation point _ki along the data stream segmentation point search direction, and obtain a new potential segmentation point at the end position of the 6th byte. To distinguish it from other potential segmentation points, it is expressed as k _g , according to the preset rules on the deduplication server 103, determine 11 windows for the potential segmentation point k _g , which are respectively W _g1 [k _g -169,k _g ], W _g2 [k _g -170,k _g -1], W _g3 [k _g -171,k _g -2], W _g4 [k _g -172,k _g -3], W _g5 [k _g -173,k _g -4], W _g6 [k _g -174,k _g -5], W _g7 [k _g -175,k _g -6], W _g8 [k _g -176,k _g -7], W _g9 [k _g -177,k _g -8 ], W _g10 [k _g -178,k _g -9] and W _g11 [k _g -179,k _g -10]. Judging whether at least part of the data in W _g1 [k _g -169, _kg ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _g2 [k _g -170, _kg -1] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _g3 [k _g -171, _kg -2] satisfies the predetermined condition C ₃ , judging whether at least part of the data in W _g4 [k _g -172, _kg -3] meets the predetermined condition C ₄ , judging W Whether at least part of the data in _g5 [k _g -173, _kg -4] satisfies the predetermined condition C ₅ , judging whether at least part of the data in _g6 [k _g -174, _kg -5] meets the predetermined condition C ₆ , judging W Whether at least part of the data in _g7 [k _g -175, _{kg g} -6] meets the predetermined condition C ₇ , judging whether at least part of the data in W _g8 [k _g -176, _{kg g} -7] meets the predetermined condition C ₈ , judging W Whether at least part of the data in _g9 [k _g -177, k _g -8] satisfies the predetermined condition C ₉ , judge whether at least part of the data in W _g10 [k _g -178, k _g -9] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _g11 [k _g -179,k _g -10] satisfies the predetermined condition C ₁₁ . Window W _g11 [k _g -179, k _g -10] coincides with window W _i5 [k _i -173, _ki -4], and C ₅ =C ₁₁ , therefore, when judging W _i5 [k _i -173, When at least part of the data in _ki -4] does not meet the predetermined condition C ₅ , jump T bytes from the potential segmentation point _ki along the data stream segmentation point search direction, and the obtained potential segmentation point k _g still does not meet the data stream Conditions for split points. Therefore, there will be double counting if 6 bytes are jumped from the potential split point _ki along the data flow split point lookup direction, so jumping 7 bytes from the potential split point _ki along the data stream split point lookup direction can reduce Repeated calculations are more efficient. Therefore, the speed of finding data stream split points is increased. When the probability that at least part of the data in the window W _x [kA _x , k+B _x ] satisfies the predetermined condition C _x is 1/2, that is to say, the jump is performed with the probability of 1/2, and each time the maximum can be Jump ‖B ₁₁ ‖+‖A ₁₁ ‖=189 bytes.

In this embodiment, the predetermined rule is: determine 11 windows W _x [kA _x , k+B _x ] for the potential segmentation point k and at least part of the data in the windows W _x [kA _x , k+B _x ] satisfy the preset Condition C _x , where the probability that at least part of the data in W _x [kA _x ,k+B _x ] meets the preset condition C _x is 1/2, x is a continuous natural number from 1 to 11 and A _x and B _x are integers . Among them, A ₁ =169, B ₁ =0; A ₂ =170, B ₂ =-1; A ₃ =171, B ₃ =-2; A ₄ =172, B ₄ =-3; A ₅ =173, B ₅ =-4; A ₆ =174, B ₆ =-5; A ₇ =175, B ₇ =-6; A ₈ =176, B ₈ =-7; A ₉ =177, B ₉ =-8; A ₁₀ =178, B ₁₀ =-9; A ₁₁ =179, B ₁₁ =-10, and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ = C ₁₀ =C ₁₁ . That is, 11 windows are selected for the potential segmentation point k, and 11 consecutive windows are selected, and P(n) can be calculated by these two factors. The selection method of 11 windows and judging that at least part of the data in each of the 11 windows meet the predetermined condition C _x follows the preset rules on the deduplication server 103, so whether there is at least Part of the data satisfies the predetermined condition C _x to determine whether the potential split point k is a data stream split point. We call the gap between two bytes a point. P(n) represents: the probability that there are no consecutive 11 windows satisfying the condition within the consecutive n windows, that is, the probability that there is no data stream segmentation point. After skipping the minimum block size of 4KB from the file header/previous split point, go back 10 bytes in the opposite direction to the data stream split point, and find the 4086th point. There is no data stream split point at this point, so P (4086)=1, and so on, P(4087)=1, ... P(4095)=1. At the 4096th point, that is, at the minimum block size, at least part of the data in each of the 11 windows satisfies the predetermined condition C _x with the probability of (1/2)^11, so with (1/2) There is a data stream split point with a probability of ^11, and there is no data stream split point with a probability of 1-(1/2)^11, so P(4096)=1-(1/2)^11.

As shown in Figure 35, at the nth window, P(n) can be deduced recursively in 12 cases.

Case 1: At least part of the data in the nth window does not meet the predetermined condition with a probability of 1/2. At this time, there are no consecutive 11 windows with a probability of P(n-1) in the n-1 windows in front of the nth window At least part of the data in each window in the window satisfies the predetermined condition, so P(n) includes 1/2*P(n-1). At least part of the data in the nth window does not meet the predetermined conditions, and there are 11 consecutive windows in the n-1 windows before the nth point, and at least part of the data in each window meets the predetermined conditions and P(n) irrelevant.

Case 2: At least part of the data in the nth window meets the predetermined condition with a probability of 1/2, and at least part of the data in the n-1th window does not meet the predetermined condition with a probability of 1/2. At this time, the n-1th window The previous n-2 windows do not exist with the probability of P(n-2). At least part of the data in each of the 11 consecutive windows meets the predetermined conditions, so P(n) contains 1/2*1/2* P(n-2). At least part of the data in the nth window meets the predetermined condition, at least part of the data in the n-1th point window does not meet the predetermined condition, and the n-2 windows in front of the n-1th window exist in each of the 11 consecutive windows. The fact that at least part of the data in a window satisfies the predetermined condition has nothing to do with P(n).

According to the above description, case 11: at least part of the data in the nth to n-9 windows meets the predetermined condition with a probability of (1/2)^10, and at least part of the data in the n-10th window does not meet the predetermined condition with a probability of 1/2 Satisfy the predetermined condition, at this time, the n-11 windows in front of the n-10th window do not exist with the probability of P(n-11). At least part of the data in each of the 11 consecutive windows meets the predetermined condition, so P (n) contains (1/2)^10*1/2*P(n-11). At least part of the data in the n-9th window meets the predetermined condition, at least part of the data in the n-10th window does not meet the predetermined condition, and there are consecutive 11 windows in the n-11 windows before the n-10th window The fact that at least part of the data in each of the windows satisfies the predetermined condition has nothing to do with P(n).

Case 12: at least part of the data in the nth to n-10th windows satisfies the predetermined condition with a probability of (1/2)^11, and this case has nothing to do with P(n).

Therefore, P(n)=1/2*P(n-1)+(1/2)^2*P(n-2)+...+(1/2)^11*P(n-11) . Another preset rule: determine 24 windows W _x [kA _x ,k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x ,k+B _x ] for potential segmentation point k, where x It is a continuous natural number from 1 to 11, A ₁ =169, B ₁ =0; A ₂ =170, B ₂ =-1; A ₃ =171, B ₃ =-2; A ₄ =172, B ₄ =-3 ; A ₅ =173, B ₅ =-4; A ₆ =174, B ₆ =-5; A ₇ =175, B ₇ =-6; A ₈ =176, B ₈ =-7; A ₉ =177, B ₉ =-8; A ₁₀ =178, B ₁₀ =-9; A ₁₁ =179, B ₁₁ =-10, ... A ₂₄ =192, B ₂₄ =-23, and C ₁ =C ₂ =C ₃ = C ₄ ＝C ₅ ＝C ₆ ＝C ₇ ＝C ₈ ＝C ₉ ＝...＝C ₂₄ , the probability that at least part of the data in the window W _x [kA _x ,k+B _x ] satisfies the predetermined condition C _x is 3/4 , P(n) can be calculated from these two factors.

Therefore, whether there is at least part of the data in each of the 24 consecutive windows satisfying the predetermined condition C _x determines whether the potential segmentation point k is a data stream segmentation point, which can be calculated by the following formula:

P(1)=1, P(2)...P(23)=1, P(24)=1-(3/4)^24,

P(n)＝1/4*P(n-1)+1/4*(3/4)*P(n-2)+......+1/4*(3/4)^23*P( n-24).

After calculation, P(5*1024)＝0.78, P(11*1024)＝0.17, P(12*1024)＝0.13, that is, after finding 12KB from the data stream start position/previous data stream split point, use 13 % probability that the data stream split point is still not found, forcing a split. Through this probability, the density function of the split point of the data stream is obtained, and the approximate average of the data stream split point is found when searching for 7.6 KB from the starting position of the data stream/the previous split point of the data stream through integration, that is, the average block length is about It is 7.6KB. Different from at least part of the data in 11 consecutive windows meeting the predetermined condition with a probability of 1/2, the traditional CDC algorithm can only achieve the effect of an average block length of 7.6KB when a window meets the condition with a probability of 1/2^12 .

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 23, a rule is preset on the deduplication server 103, and the rule is: determine 11 windows for the potential segmentation point k W _x [kA _x , k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ], wherein x is a continuous natural number from 1 to 11, and A _x and B _x are integers. Wherein, the probability that at least part of the data in the window W _x [kA _x ,k+B _x ] satisfies the predetermined condition C _x is 1/2, A ₁ =171, B ₁ =-2; A ₂ =172, B ₂ =- 3; A ₃ =173, B ₃ =-4; A ₄ =174, B ₄ =-5; A ₅ =175, B ₅ =-6; A ₆ =176, B ₆ =-7; A ₇ =177 , B ₇ =-8; A ₈ =178, B ₈ =-9; A ₉ =179, B ₉ =-10; A ₁₀ =170, B ₁₀ =-1; A ₁₁ =169, B ₁₁ =0, And C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ . k _a is the data stream split point. The search direction of the data stream split point shown in Figure 23 is from left to right. After skipping the smallest data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is used as the next Potential segmentation point k _i , according to the preset rules on the deduplication server 103, determine W _x [kA _x , k+B _x ] and window W _x [kA _x , k+B _x ] corresponding to potential segmentation point k _i The preset condition C _x , where x is a continuous natural number from 1 to 11. The 11 determined windows are respectively W _i1 [k _i -171, _ki -2], W _i2 [k _i -172, _ki -3], W _i3 [k _i -173, ki -4], W _{i i4} [k _i -174,k _i -5], W _i5 [k _i -175,k _i -6], W _i6 [k _i -176,k _i -7], W _i7 [k _i -177,k _i -8], W _i8 [k _i -178,k _i -9], W _i9 [k _i -179,k _i -10], W _i10 [k _i -170,k _i -1] and W _i11 [ k _i -169, k _i ]. Judging whether at least part of the data in W _i1 [k _i -171, _ki -2] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [k _i -172, _ki -3] meets the predetermined condition C ₂ , Judging whether at least part of the data in W _i3 [k _i -173, _ki -4] meets the predetermined condition C ₃ , judging whether at least part of the data in W _i4 [k _i -174, _ki -5] meets the predetermined condition C ₄ , Judging whether at least part of the data in W _i5 [k _i -175, _ki -6] meets the predetermined condition C ₅ , judging whether at least part of the data in W _i6 [k _i -176, _ki -7] meets the predetermined condition C ₆ , Judging whether at least part of the data in W _i7 [k _i -177, _ki -8] meets the predetermined condition C ₇ , judging whether at least part of the data in W _i8 [k _i -178, _ki -9] meets the predetermined condition C ₈ , Judging whether at least part of the data in W _i9 [k _i -179, _ki -10] meets the predetermined condition C ₉ , judging whether at least part of the data in W _i10 [k _i -170, _ki -1] meets the predetermined condition C ₁₀ and It is judged whether at least part of the data in W _i11 [k _i -169, _ki ] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined condition, as shown in Figure 24, at least part of the data in W _i3 [p _i3 -169, p _i3 ] does not meet the predetermined condition C ₃ , point p _i3 jumps 11 bytes along the search direction of the data stream split point as an example for description. As shown in Figure 24, when it is judged that W ₃ does not meet the predetermined condition C ₃ , start with _ki as the starting point and jump N bytes along the search direction of the data stream segmentation point, where N bytes are not greater than ‖B ₃ ‖ +max _x (‖A _x ‖), in the present embodiment, N=7, at the end position of the 7th byte, obtain the next potential segmentation point, for being different from the potential segmentation point _ki , here the new A potential segmentation point is denoted as k _j , according to the preset rules on the deduplication server 103, 11 windows W _jx [k _j -A _x , k _j +B _x ] are determined for the potential segmentation point k _j , respectively W _j1 [k _j -171,k _j -2], W _j2 [k _j -172,k _j -3], W _j3 [k _j -173,k _j -4], W _j4 [k _j -174,k _j -5], W _j5 [k _j -175,k _j -6], W _j6 [k _j -176,k _j -7], W _j7 [k _j -177,k _j -8], W _j8 [k j _j -178,k _j -9], W _j9 [k _j -179,k _j -10], W _j10 [k _j -170,k _j -1], and W _j11 [k _j -169,k _j ]. Judging whether at least part of the data in W _j1 [k _j -171,k _j -2] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [k _j -172,k _j -3] meets the predetermined condition C ₂ , Judging whether at least part of the data in W _j3 [k _j -173, k _j -4] meets the predetermined condition C ₃ , judging whether at least part of the data in W _j4 [k _j -174, k _j -5] meets the predetermined condition C ₄ , Judging whether at least part of the data in W _j5 [k _j -175, k _j -6] meets the predetermined condition C ₅ , judging whether at least part of the data in W _j6 [k _j -176, k _j -7] meets the predetermined condition C ₆ , Judging whether at least part of the data in W _j7 [k _j -177, k _j -8] meets the predetermined condition C ₇ , judging whether at least part of the data in W _j8 [k _j -178, k _j -9] meets the predetermined condition C ₈ , Judging whether at least part of the data in W _j9 [k _j -179, k _j -10] meets the predetermined condition C ₉ , judging whether at least part of the data in W _j10 [k _j -170, k _j -1] meets the predetermined condition C ₁₀ and It is judged whether at least part of the data in W _j11 [k _j −169,k _j ] satisfies the predetermined condition C ₁₁ . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data stream split point. When it is judged that the potential split point k _j is not a data flow split point, the next potential split point is obtained in the same manner as k _i , and the next potential split point is judged according to the preset rules on the deduplication server 103 and the above method. Split point for data flow. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the mandatory split point. Of course, the implementation of this method is limited by the maximum data block length and the size of the files constituting the data stream, so details will not be repeated here.

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 25, a rule is preset on the deduplication server 103, and the rule is: determine 11 windows for the potential segmentation point k W _x [kA _x , k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ], where x is a continuous natural number from 1 to 11, A ₁ =166, B ₁ =3; A ₂ =167, B ₂ =2; A ₃ =168, B ₃ =1; A ₄ =169, B ₄ =0; A ₅ =170, B ₅ =-1; A ₆ =171, B ₆ =- 2; A ₇ =172, B ₇ =-3; A ₈ =173, B ₈ =-4; A ₉ =174, B ₉ =-5; A ₁₀ =175, B ₁₀ =-6; A ₁₁ =176 , B ₁₁ =-7; and C ₁ ＝C ₂ ＝C ₃ ＝C ₄ ＝C ₅ ＝C ₆ ＝C ₇ ＝C ₈ ＝C ₉ ＝C ₁₀ ＝C ₁₁ , then the 11 windows are W ₁ [ k-166,k+3], W ₂ [k-167,k+2], W ₃ [k-168,k+1], W ₄ [k-169,k], W ₅ [k-170, k-1], W ₆ [k-171,k-2], W ₇ [k-172,k-3], W ₈ [k-173,k-4], W ₉ [k-174,k- 5], W ₁₀ [k-175,k-6] and W ₁₁ [k-176,k-7]. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 25 is from left to right. After skipping the smallest data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is taken as the next potential Segmentation point k _i , in this embodiment, according to the preset rules on the deduplication server 103 _, 11 windows W _ix [kA _x , k+B _x ] and windows W _ix [kA _x , k+B _x ] corresponding to the predetermined condition C _x , x is a continuous natural number from 1 to 11 respectively. In the embodiment shown in FIG. 25, 11 windows are determined for the potential segmentation point ki, which are respectively W _i1 [k _i -166, _ki +3], W _i2 [ _{k i} -167, _ki +2] , W _i3 [k _i -168,k _i +1], W _i4 [k _i -169,k _i ], W _i5 [k _i -170,k _i -1], W _i6 [k _i -171,k _i -2], W _i7 [k _i -172,k _i -3], W _i8 [k _i -173,k _i -4], W _i9 [k _i -174,k _i -5], W _i10 [ k _i -175, k _i -6] and W _i11 [k _i -176, k _i -7]. Judging whether at least part of the data in W _i1 [k _i -166, _ki +3] meets the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [k _i -167, _ki +2] meets the predetermined condition C ₂ , Judging whether at least part of the data in W _i3 [k _i -168, _ki +1] meets the predetermined condition C ₃ , judging whether at least part of the data in W _i4 [k _i -169, _ki ] meets the predetermined condition C ₄ , judging W Whether at least part of the data in _i5 [k _i -170, _ki -1] meets the predetermined condition C ₅ , judge whether at least part of the data in W _i6 [k _i -171, _ki -2] meet the predetermined condition C ₆ , and judge W _i7 Whether at least part of the data in W _i7 [k _i -172, _ki -3] meets the predetermined condition C ₇ , judging whether at least part of the data in W _i8 [k _i -173, _ki -4] meets the predetermined condition C ₈ , Judging whether at least part of the data in W _i9 [k _i -174, _ki -5] meets the predetermined condition C ₉ , judging whether at least part of the data in W _i10 [k _i -175, _ki -6] meets the predetermined condition C ₁₀ and It is judged whether at least part of the data in W _i11 [k _i -176, _ki -7] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined conditions, as shown in Figure 26, W _i7 [k _i -172, _ki -3], then from the potential segmentation point _ki along the data Stream split point search direction jumps N bytes, where N bytes are not greater than ‖B ₇ ‖+max _x (‖A _x ‖), in the embodiment shown in Figure 26, jump N bytes not greater than 185 bytes, in this embodiment, N=5, to obtain a new potential segmentation point, in order to distinguish it from the potential segmentation point ki, here the new potential segmentation point is represented as _{k j ,} according to the implementation shown in Figure 25 According to the preset rules on the deduplication server 103, 11 windows are determined for the potential segmentation point k _j , which are respectively W _j1 [k _j -166, k _j +3], W _j2 [k _j -167, k _j +2], W _j3 [k _j -168,k _j +1], W _j4 [k _j -169,k _j ], W _j5 [k _j -170,k _j -1], W _j6 [k _j -171,k _j -2], W _j7 [k _j -172,k _j -3], W j8 [k _{j -173} ,k _j -4], W _j9 [k _j -174,k _j -5] , W _j10 [k _j -175, k _j -6] and W _j11 [k _j -176, k _j -7]. Judging whether at least part of the data in W _j1 [k _j -166, k _j +3] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [k _j -167, k _j +2] meets the predetermined condition C ₂ , Judging whether at least part of the data in W _j3 [k _j -168,k _j +1] meets the predetermined condition C ₃ , judging whether at least part of the data in W _j4 [k _j -169,k _j ] meets the predetermined condition C ₄ , judging W Whether at least part of the data in _j5 [k _j -170, k _j -1] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -171, k _j -2] meet the predetermined condition C ₆ , and judge W Whether at least part of the data in _j7 [k _j -172, k _j -3] meets the predetermined condition C ₇ , judge whether at least part of the data in W j8 [k _{j -173} , k _j -4] meet the predetermined condition C ₈ , judge W Whether at least part of the data in _j9 [k _j -174, k _j -5] meets the predetermined condition C ₉ , judge whether at least part of the data in W _j10 [k _j -175, k _j -6] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _j11 [k _j -176,k _j -7] satisfies the predetermined condition C ₁₁ . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data stream split point. When it is judged that the potential split point k _j is not a data flow split point, the next potential split point is obtained in the same manner as k _i , and the next potential split point is judged according to the preset rules on the deduplication server 103 and the above method. Split point for data flow. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the forced split point.

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 27, a rule is preset on the deduplication server 103, and the rule is: determine 11 windows for the potential segmentation point k W _x [kA _x ,k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x ,k+B _x ], where x is a continuous natural number from 1 to 11, A ₁ =169, B ₁ =0 ; A ₂ =170, B ₂ =-1; A ₃ =171, B ₃ =-2; A ₄ =172, B ₄ =-3; A ₅ =173, B ₅ =-4; A ₆ =174, B ₆ =-5; A ₇ =175, B ₇ =-6; A ₈ =176, B ₈ =-7; A ₉ =177, B ₉ =-8; A ₁₀ =168, B ₁₀ =1; ₁₁ =179, B ₁₁ =3; and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ =C ₇ =C ₈ =C ₉ =C ₁₀ ≠C ₁₁ , then the 11 windows are respectively W ₁ [k-169,k], W ₂ [k-170,k-1], W ₃ [k-171,k-2], W ₄ [k-172,k-3], W ₅ [k- 173,k-4], W ₆ [k-174,k-5], W ₇ [k-175,k-6], W ₈ [k-176,k-7], W ₉ [k-177, k-8], W ₁₀ [k-168, k+1], and W ₁₁ [k-179, k+3]. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 27 is from left to right. After skipping the smallest data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is taken as the next potential Segmentation point k _i , in this embodiment, according to the preset rules on the deduplication server 103, determine the window W _ix [k _i -A _x , k _i +B _x ] for the potential segmentation point k _i , where x is respectively 1 to 11 consecutive natural numbers, in the embodiment shown in Figure 27, 11 windows are determined for the potential segmentation point k _i respectively W _i1 [k _i -169, k _i ], W _i2 [k _i -170, k _i -1], W _i3 [k _i -171,k _i -2], W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4], W _i6 [ k _i -174,k _i -5], W _i7 [k _i -175,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177,k _i - 8], W _i10 [k _i -168, _ki +1], and W _i11 [k _i -179, _ki +3]. Judging whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _i3 [k _i -171, _ki -2] meets the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [k _i -172, _ki -3] meet the predetermined condition C ₄ , and judge W Whether at least part of the data in _i5 [k _i -173, _ki -4] meets the predetermined condition C ₅ , judge whether at least part of the data in W _i6 [k _i -174, _ki -5] meet the predetermined condition C ₆ , and judge W Whether at least part of the data in _i7 [k _i -175, _ki -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -176, _ki -7] meet the predetermined condition C ₈ , and judge W Whether at least part of the data in _i9 [k _i -177, _ki -8] meets the predetermined condition C ₉ , judge whether at least part of the data in W _i10 [k _i -168, _ki +1] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _i11 [k _i -179, _ki +3] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in the judging window W _i11 does not meet the predetermined condition C ₁₁ , then jump 1 byte from the potential segmentation point _ki along the data stream segmentation point search direction to obtain a new potential segmentation point, which is the potential segmentation point _ki difference, here denote the new potential segmentation point as k _j . When at least part of the data in any of the 10 windows W _i1 , W _i2 , W _i3 , W _i4 , W _i5 , W _i6 , W _i7 , W _i8 , W _i9 , and W _i10 does not meet the corresponding predetermined conditions, such as As shown in Figure 28, W _i4 [k _i -172, _ki -3], then N bytes are jumped from point _ki along the direction of data stream segmentation point search, where N bytes are not greater than ‖B ₄ ‖+ max _x (‖A _x ‖), in the embodiment shown in Figure 28, skip N bytes and be no more than 182 bytes, in the present embodiment, N=6, obtain new potential segmentation point, be and Potential segmentation point _ki difference, here the new potential segmentation point is represented as k _j , according to the rules preset on the deduplication server 103 in the embodiment shown in FIG. 27 , the windows determined for the potential segmentation point k _j are respectively W _j1 [k _j -169,k _j ], W _j2 [k _j -170,k _j -1], W _j3 [k _j -171,k _j -2], W _j4 [k _j -172,k _j -3], W _j5 [k _j -173,k _j -4], W _j6 [k _j -174,k _j -5], W _j7 [k _j -175,k _j -6], W _j8 [k j _j -176,k _j -7], W _j9 [k _j -177,k _j -8], W _j10 [k _j -168,k _j +1] and W _j11 [k _j -179,k _j +3 ]. Judging whether at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _j3 [k _j -171,k _j -2] meets the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [k _j -172,k _j -3] meet the predetermined condition C ₄ , judge W Whether at least part of the data in _j5 [k _j -173, k _j -4] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174, k _j -5] meet the predetermined condition C ₆ , judge W Whether at least part of the data in _j7 [k _j -175, k _j -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W j8 [k _{j -176} , k _j -7] meet the predetermined condition C ₈ , and judge W Whether at least part of the data in _j9 [k _j -177, k _j -8] meets the predetermined condition C ₉ , judge whether at least part of the data in W _j10 [k _j -168, k _j +1] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _j11 [k _j −179, k _j +3] satisfies the predetermined condition C ₁₁ . Of course, in the embodiment of the present invention, this principle is also followed when judging whether a potential split point k _a is a data stream split point, and the specific implementation will not be described again, and reference may be made to the description of judging a potential split point k _i . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in window _Wj9 meets the predetermined condition _C9 , at least part of the data in window _Wj10 meets the predetermined condition _C10 , and at least part of the data in window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data stream split point, the data between k _j and k _a constitutes a data block, and at the same time skip the minimum block size of 4KB in the same way as k _a , get the next potential split point, and follow the Deduplicate the preset rules on the server 103 to determine whether the next potential split point is a data flow split point. When it is judged that the potential split point k _j is not a data flow split point, the next potential split point is obtained in the same manner as k _i , and the next potential split point is judged according to the preset rules on the deduplication server 103 and the above method. Split point for data flow. When the data flow split point is not found beyond the set maximum data block, the end position of the largest data block is used as the mandatory split point.

On the basis of the data stream segmentation point search shown in Figure 3, in the embodiment shown in Figure 29, a rule is preset on the deduplication server 103, and the rule is: determine 11 windows for the potential segmentation point k W _x [p _x -A _x ,p _x +B _x ] and the predetermined condition C _x corresponding to the window W _x [p _x -A _x ,p _x +B _x ], x is a continuous natural number from 1 to 11, where , the probability that at least part of the data in the window W _x [p _x -A _x ,p _x +B _x ] satisfies the predetermined condition is 1/2, A ₁ =169, B ₁ =0; A ₂ =171, B ₂ =- 2; A ₃ =173, B ₃ =-4; A ₄ =175, B ₄ =-6; A ₅ =177, B ₅ =-8; A ₆ =179, B ₆ =-10; A ₇ =181 , B ₇ =-12; A ₈ =183, B ₈ =-14; A ₉ =185, B ₉ =-16; A ₁₀ =187, B ₁₀ =-18; A ₁₁ =189, B ₁₁ =-20 ; and C ₁ ＝C ₂ ＝C ₃ ＝C ₄ ＝C ₅ ＝C ₆ ＝C ₇ ＝C ₈ ＝C ₉ ＝C ₁₀ ＝C ₁₁ , then the 11 windows are respectively W ₁ [k-169,k] , W ₂ [k-171,k-2], W ₃ [k-173,k-4], W ₄ [k-175,k-6], W ₅ [k-177,k-8], W ₆ [k-179,k-10], W ₇ [k-181,k-12], W ₈ [k-183,k-14], W ₉ [k-185,k-16], W ₁₀ [ k-187,k-18] and W ₁₁ [k-189,k-20]. k _a is the data stream split point. The search direction of the data stream split point shown in Figure 29 is from left to right. After skipping the smallest data block 4KB from the data stream split point k _a , the end position of the smallest data block 4KB is used as the next The potential segmentation point _ki is to determine the point p _ix for the potential segmentation point _ki . In this embodiment, according to the preset rules on the deduplication server 103, x is a continuous natural number from 1 to 11. In the embodiment shown in FIG. 29 , according to predetermined rules, the 11 windows determined for the potential segmentation point k _i are W _i1 [k _i -169, _ki ], W _i2 [k _i -171, _ki - 2], W _i3 [k _i -173, _ki -4], W _i4 [k _i -175, _ki -6], W _i5 [k _i -177, _ki -8], W _i6 [k _i -179,k _i -10], W _i7 [k _i -181,k _i -12], W _i8 [k _i -183,k _i -14], W _i9 [k _i -185,k _i -16] , W _i10 [k _i -187, _ki -18] and W _i11 [k _i -189, _ki -20]. Judging whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , judging whether at least part of the data in W _i2 [k _i -171, _ki -2] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _i3 [k _i -173, _ki -4] meets the predetermined condition C ₃ , judge whether at least some of the data in W _i4 [k _i -175, _ki -6] meet the predetermined condition C ₄ , and judge W Whether at least part of the data in _i5 [k _i -177, _ki -8] meets the predetermined condition C ₅ , judge whether at least part of the data in W _i6 [k _i -179, _ki -10] meet the predetermined condition C ₆ , and judge W Whether at least part of the data in _i7 [k _i -181, _ki -12] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -183, _ki -14] meet the predetermined condition C ₈ , and judge W Whether at least part of the data in _i9 [k _i -185, _ki -16] meet the predetermined condition C ₉ , judge whether at least part of the data in W _i10 [k _i -187, _ki -18] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _i11 [k _i -189, _ki -20] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _i1 meets the predetermined condition C ₁ , at least part of the data in window W _i2 meets the predetermined condition C ₂ , at least part of the data in window W _i3 meets the predetermined condition C ₃ , and at least part of the data in window W _i4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _i5 meets the predetermined condition C ₅ , at least part of the data in window W _i6 meets the predetermined condition C ₆ , at least part of the data in window W _i7 meets the predetermined condition C ₇ , and at least part of the data in window W _i8 When the predetermined condition C8 _{is met, at least part of the data in window W i9 meets} the predetermined condition _C9 , at least part of the data in window W _i10 meets the predetermined condition _C10 , and at least part of the data in window W _i11 meets the predetermined condition _C11 , then the current potential segmentation Point _ki is the data flow splitting point. When at least part of the data in any of the 11 windows does not meet the corresponding predetermined condition, as shown in Figure 30, at least part of the data in W _i4 [k _i -175, _ki -6] does not meet the predetermined condition C ₄ , Then select the next potential segmentation point, which is different from the potential segmentation point ki, expressed as k _j here, k _j is located on the right side of _ki , and the distance between _{k j and ki} is 1 byte. As shown in FIG. 30 , according to the preset rules for the deduplication server 103, 11 windows are determined for the potential segmentation point k _j as W _j1 [k _j -169, k _j ], W _j2 [k _j -171, k _j -2], W _j3 [k _j -173,k _j -4], W _j4 [k _j -175,k _j -6], W _j5 [k _j -177,k _j -8], W _j6 [ k _j -179,k _j -10], W _j7 [k _j -181,k _j -12], W _j8 [k _j -183,k _j -14], W _j9 [k _j -185,k _j - 16], W _j10 [k _j -187, k _j -18] and W _j11 [k _j -189, k _j -20], and C ₁ =C ₂ =C ₃ =C ₄ =C ₅ =C ₆ = C ₇ =C ₈ =C ₉ =C ₁₀ =C ₁₁ . Judging whether at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _j2 [k _j -171,k _j -2] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _j3 [k _j -173,k _j -4] meets the predetermined condition C ₃ , judge whether at least part of the data in W _j4 [k _j -175,k _j -6] meets the predetermined condition C ₄ , judge W Whether at least part of the data in _j5 [k _j -177, k _j -8] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -179, k _j -10] meet the predetermined condition C ₆ , judge W Whether at least part of the data in _j7 [k _j -181, k _j -12] meets the predetermined condition C ₇ , judge whether at least part of the data in W _j8 [k _j -183, k _j -14] meet the predetermined condition C ₈ , judge W Whether at least part of the data in _j9 [k _j -185, k _j -16] meets the predetermined condition C ₉ , judge whether at least part of the data in W _j10 [k _j -187, k _j -18] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _j11 [k _j -189,k _j -20] satisfies the predetermined condition C ₁₁ . When judging that at least part of the data in window W _j1 meets the predetermined condition C ₁ , at least part of the data in window W _j2 meets the predetermined condition C ₂ , at least part of the data in window W _j3 meets the predetermined condition C ₃ , and at least part of the data in window W _j4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _j5 meets the predetermined condition C ₅ , at least part of the data in window W _j6 meets the predetermined condition C ₆ , at least part of the data in window W _j7 meets the predetermined condition C ₇ , and at least part of the data in window W _j8 When the predetermined condition C8 is met, at least part of the data in the window W _i9 meets the predetermined condition _C9 , at least part of the data in the window _Wj10 meets the predetermined condition _C10 , and at least part of the data in the window _{Wj11 meets} the predetermined condition _C11 , then the current potential segmentation Point k _j is the data flow splitting point. When judging that at least part of the data in any of the windows W _j1 , W _j2 , W _j3 , W _j4 , W _j5 , W _j6 , W _j7 , W _j8 , W _j9 , W _j10 and W _j11 does not meet the predetermined conditions, as As shown in Figure 31, when at least part of the data in W _j3 [k _j -173,k _j -4] does not meet the predetermined condition C ₃ , k _j is located on the right of _ki and jumps N times from _ki along the direction of data stream segmentation point search Bytes, where N bytes are not greater than ‖B ₄ ‖+max _x (‖A _x ‖), in the embodiment shown in Figure 28, N bytes are not greater than 195 bytes, in this embodiment , N=15, to obtain the next potential segmentation point, which is denoted as k _l to distinguish it from potential segmentation points ki and _{k j .} According to the preset rules for the deduplication server 103 in the embodiment shown in FIG. 29, 11 windows are determined for the potential segmentation point k _l , which are respectively W _l1 [k _l -169, k _l ], W _l2 [k _l -171, k _l -2], W _l3 [k _l -173,k _l -4], W _l4 [k _l -175,k _l -6], W _l5 [k _l -177,k _l -8], W _l6 [ k _l -179,k _l -10], W _l7 [k _l -181,k _l -12], W _l8 [k _l -183,k _l -14], W _l9 [k _l -185,k _l - 16], W _l10 [k _l -187,k _l -18] and W _l11 [k _l -189,k _l -20]. Judging whether at least part of the data in W _l1 [k _l -169,k _l ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _l2 [k _l -171,k _l -2] meets the predetermined condition C ₂ , judging W _l3 Whether at least part of the data in [k _l -173,k _l -4] meets the predetermined condition C ₃ , judge whether at least part of the data in W _l4 [k _l -175,k _l -6] meet the predetermined condition C ₄ , judge W _l5 Whether at least part of the data in [k _l -177, k _l -8] satisfies the predetermined condition C ₅ , judging W _l6 whether at least part of the data in [k _l -179, k _l -10] meets the predetermined condition C ₆ , judging W _l7 Whether at least part of the data in [k _l -181, k _l -12] meets the predetermined condition C ₇ , judge whether at least part of the data in W _l8 [k _l -183, k _l -14] meet the predetermined condition C ₈ , judge W _l9 Whether at least part of the data in [k _l -185, k _l -16] meets the predetermined condition C ₉ , judgment W _l10 Whether at least part of the data in [k _l -187, k _l -18] meets the predetermined condition C ₁₀ and judgment W Whether at least part of the data in _l11 [k _l -189,k _l -20] satisfies the predetermined condition C ₁₁ . When it is judged that at least part of the data in window W _l1 meets the predetermined condition C ₁ , at least part of the data in window W _l2 meets the predetermined condition C ₂ , at least part of the data in window W _l3 meets the predetermined condition C ₃ , and at least part of the data in window W _l4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _l5 meet the predetermined condition C ₅ , at least part of the data in window W _l6 meet the predetermined condition C ₆ , at least part of the data in window W _l7 meet the predetermined condition C ₇ , at least part of the data in window W _l8 When the predetermined condition C8 is met, at least part of the data in window _W19 meets the predetermined condition _C9 , at least part of the data in window _W110 meets the predetermined condition _C10 , and at least part of the data in window _{W111 meets} the predetermined condition _C11 , then the current potential segmentation Point k _l is the data flow splitting point. When at least some of the data in any of the windows W _l1 , W _l2 , W _l3 , W _l4 , W _l5 , W _l6 , W _l7 , W _l8 , W _l9 , W _l10 and W _l11 do not meet the predetermined conditions, select the following A potential segmentation point, to be distinguished from potential segmentation points _ki , _kj and _kl , is denoted as km, km is located to the right of _{kl ,} and the distance between _km and _kl is ₁ byte. According to the preset rules for the deduplication server 103 in the embodiment shown in FIG. 29, the 11 windows determined for the potential segmentation point k _m are respectively W _m1 [k _m -169, km ], W _m2 [ _{k m -171} , _{km -2], W m3 [} km _-173 , _km -4], W _m4 [ _km -175, _{km -6], W m5 [} km _-177 , _km -8], W _m6 [km -179, _{km -10], W m7 [} km _-181 , km -12], W _m8 [km _-183 , _{km -14} ], W _m9 [ _{km -185} , _km -16], W _m10 [km -187, _km -18] and W _m11 [ _{km -189} , _km -20]. Judging whether at least part of the data in W _m1 [km _-169 , _km ] meets the predetermined condition C ₁ , judging whether at least part of the data in W _m2 [ _km -171, _km -2] meets the predetermined condition C ₂ , judging W Whether at least part of the data in _m3 [k _m -173, _km -4] meets the predetermined condition C ₃ , judge whether at least part of the data in W _m4 [ _km -175, _km -6] meet the predetermined condition C ₄ , and judge W Whether at least part of the data in _m5 [k _{m -177} , km -8] meets the predetermined condition C ₅ , judge whether at least part of the data in W _m6 [ _km -179, _km -10] meet the predetermined condition C ₆ , and judge W Whether at least part of the data in _m7 [k _m -181, _km -12] meets the predetermined condition C ₇ , judging whether at least part of the data in W _m8 [ _km -183, _km -14] meets the predetermined condition C ₈ , judging W Whether at least part of the data in _m9 [k _m -185, _km -16] meets the predetermined condition C ₉ , judge whether at least part of the data in W _m10 [ _km -187, _km -18] meet the predetermined condition C ₁₀ and judge W Whether at least part of the data in _m11 [k _m -189,k _m -20] satisfies the predetermined condition C ₁₁ . When it is judged that at least part of the data in window W _m1 meets the predetermined condition C ₁ , at least part of the data in window W _m2 meets the predetermined condition C ₂ , at least part of the data in window W _m3 meets the predetermined condition C ₃ , and at least part of the data in window W _m4 meets the predetermined condition Condition C ₄ , at least part of the data in window W _m5 meet the predetermined condition C ₅ , at least part of the data in window W _m6 meet the predetermined condition C ₆ , at least part of the data in window W _m7 meet the predetermined condition C ₇ , and at least part of the data in window W _m8 When the predetermined condition C8 is met, at least part of the data in window _Wm9 meets the predetermined condition _C9 , at least part of the data in window _Wm10 meets the predetermined condition _C10 , and at least part of the data in window _{Wm11 meets} the predetermined condition _C11 , then the current potential segmentation The point k _m is the data flow splitting point. When at least part of the data in any window does not meet the predetermined condition, jumping is performed according to the scheme described above to obtain the next potential split point and determine whether it is a data flow split point.

The embodiment of the present invention provides a method for judging whether at least part of the data in the window W _iz [k _i -A _z , k _i +B _z ] satisfies the predetermined condition C _z . In this embodiment, a random function is used to judge the window W _iz [ k _i -A _z , k _i +B _z ] whether at least part of the data satisfies the predetermined condition C _z , taking the implementation shown in FIG. k _i determines the window W _i1 [k _i -169, _ki ], and judges whether at least part of the data in W _i1 [k _i -169, _ki ] meets the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , select 5 bytes, "■" in Figure 32 indicates selection 1 byte, the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, the bits corresponding to 255 bytes It can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 For any one, the bits corresponding to 255 bytes are obtained according to the conversion relationship between a _m,n and V _am,n to obtain the matrix V _a , which can be expressed as: . Select a large number of random numbers to form a matrix. Once the matrix composed of random data is formed, it remains unchanged. For example, select 255*8 random numbers from random numbers that obey a specific distribution (here, take the normal distribution as an example) to form a matrix R: Multiply the m-th row of the matrix V _a with the random number in the m-th row of the matrix R, and then sum to get a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _am,2 *h _m,2 +...+V _am,8 *h _m,8 . According to this method, S _a1 , S _a2 . . . to S _a255 are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _a1 , S _a2 . . . to S _a255 is counted. Since the matrix R obeys the normal distribution, S _am , like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of the normal distribution random number being greater than 0 is 1/2, in S _a1 , S _a2 ... to S _a255 In , the probability of each value greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K _whose value is greater than 0 from S _a1 , S _a2 . condition. When K is an even number, it indicates that at least part of the data in W _i1 [k _i -169, _ki ] meets the predetermined condition C ₁ ; when K is an odd number, it indicates that at least part of the data in W ₁ [k _i -169, _ki ] The predetermined condition C ₁ is not satisfied, where C ₁ means that the number K of _{S a1} , S _a2 . In the embodiment shown in FIG. 21 , in W _i1 [k _i -169, _ki ], W _i2 [k _i -170, _ki -1], W _i3 [k _i -171, _ki -2] , W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4], W _i6 [k _i -174,k _i -5], W _i7 [k _i -175 ,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177,k _i -8], W _i10 [k _i -178,k _i -9] and W In _i11 [k _i -179,k _i -10], the size of each window is the same, that is, the size of the window is 169 bytes. At the same time, the method of judging whether at least part of the data in the window meets the predetermined condition is also the same. For details, see the above judgment W _i1 Whether at least part of the data in [k _i −169, _ki ] satisfies the description of the predetermined condition C ₁ . Therefore, as shown in Figure 32, Indicates 1 byte selected when judging whether at least part of the data in the window W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as b _m,1 ...b _m,8 , which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, the corresponding bits of 255 bytes It can be expressed as: When b _m,n =1, V _bm,n =1, when b _m,n =0, V _bm,n =-1, where b _m,n represents b _m,1 ...b _m,8 For any one, the bits corresponding to 255 bytes are obtained according to the conversion relationship between b _m,n and V _bm,n to obtain the matrix V _b , which can be expressed as: The method of judging whether at least part of the data in W _i1 [k _i -169, _ki ] meets the predetermined condition is the same as the method of judging whether at least part of the data in the window W _i2 [k _i -170, _ki -1] meets the predetermined condition, So using the matrix R: Multiply the m-th row of the matrix V _b with the random number in the m-th row of the matrix R, and then sum to get a value, specifically expressed as S _bm =V _bm,1 *h _m,1 +V _bm,2 *h _m,2 +...+V _bm,8 *h _m,8 . According to this method, obtain S _b1 , S _b2 . . . to S _b255 , and _count the number K of values in S _b1 , S _b2 . Since the matrix R obeys the normal distribution, S _bm , like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of the normal distribution random number being greater than 0 is 1/2, in S _b1 , S _b2 ... to S _b255 In , the probability of each value greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K whose value is greater than 0 from S _b1 , S _b2 ... to S _b255 is an even number, the probability that the random number of the binomial distribution is an even number is 1/2, so K is with the probability of 1/2 To meet the conditions. When K is an even number, it indicates that at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ ; when K is an odd number, it indicates that W _i2 [k _i -170, _ki -1] At least some of the data in do not satisfy the predetermined condition C ₂ , where C ₂ means that the number _K of S _b1 , S _b2 . In the implementation shown in FIG. 21 , at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ .

Therefore, as shown in Figure 32, Indicates one byte selected when judging whether at least part of the data in the window W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the method of judging whether at least part of the data in the windows W _i1 [k _i -169, _ki ] and W _i2 [k _i -170, _ki -1] meet the predetermined conditions, and judge W _i3 [k _i -171, k Whether at least the data in _i -2] satisfies the predetermined condition C ₃ . In the implementation shown in FIG. 21 , at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition. As shown in Figure 32, Indicates 1 byte selected when judging whether at least part of the data in the window W _i4 [k _i -172, _ki -3] satisfies the predetermined condition C ₄ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the judgment window W _i1 [k _i -169, _ki -1], W _i2 [k _i -170, _ki -1] and W _i3 [k _i -171, _ki -2] to determine whether at least part of the data meets the predetermined A conditional method, judging whether at least part of the data in W _i4 [k _i -172, _ki -3] satisfies the predetermined condition C ₄ . In the implementation shown in FIG. 21 , at least part of the data in W _i4 [k _i -172, _ki -3] satisfies the predetermined condition C ₄ . As shown in Figure 32, Indicates one byte selected when judging whether at least part of the data in the window W _i5 [k _i -173, _ki -4] satisfies the predetermined condition C ₅ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Then use the judgment windows W _i1 [k _i -169,k _i ], W _i2 [k _i -170,k _i -1], W _i3 [k _i -171,k _i -2] and W _i4 [k _i - 172, _ki -3] at least part of the data satisfy the predetermined condition, judging whether at least the data in W _i5 [ _ki -173, _ki -4] satisfy the predetermined condition C ₅ . In the embodiment shown in FIG. 21 , at least part of the data in W _i5 [k _i -173, _ki -4] does not satisfy the predetermined condition C ₅ .

When at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , jump 7 bytes from point p _i5 along the direction of data flow splitting point search, at the seventh byte The end position of the next potential segmentation point k _j is obtained, as shown in Figure 22, according to the rules preset for the deduplication server 103, the window W _j1 [k _j -169, k _j ] is determined for the potential segmentation point k _j , and the judgment The method of whether at least part of the data in the window W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ and the method of judging whether at least part of the data in the window W _i1 [k _i -169,k _i ] meet the predetermined condition C ₁ Same, so as shown in Figure 33, W _j1 represents the window, in order to judge whether at least part of the data satisfies the predetermined condition C ₁ , select 5 bytes, "■" in Figure 33 represents the selected 1 byte, adjacent two The difference between selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as a _m,1 '...a _m,8 ', indicating the 1st to 8th bits of the mth byte in 255 bytes, therefore, 255 bytes correspond to bits can be expressed as: When a _m,n '=1, V _am,n '=1, when a _m,n '=0, V _am,n '=-1, where a _m,n ' means a _m,1 '... For any one of a _m,8 ', the bits corresponding to 255 bytes are obtained according to the conversion relationship between a _m,n ' and V _am,n ' to obtain a matrix V _a ', which can be expressed as: Judging whether at least part of the data in the window meets the predetermined condition is the same as judging whether at least part of the data in the window W _i1 [k _i -169, _ki ] meets the predetermined condition, so the matrix R is used: Multiply the mth row of the matrix V _a ' with the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am '=V _am,1 '*h _m,1 +V _{am, 2} '*h _m,2 +...+V _am,8 '*h _m,8 . According to this method, S _a1 ′, S _a2 ′… to S _a255 ′ are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _a1 ′, S _a2 ′… to S _a255 ′ is counted. Since the matrix R obeys the normal distribution, S _am ', like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of a normal distribution random number being greater than 0 is 1/2. In S _a1 ', S _a2 '... To S _a255 ', the probability of each value being greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K whose value is greater than 0 from S _a1 ', S _a2 '... to S _a255 ' is an even number, and the probability that the random number of the binomial distribution is an even number is 1/2, so K is 1/2 The probability satisfies the condition. When K is an even number, it indicates that at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ ; when K is an odd number, it indicates that at least part of the data in W _j1 [k _j -169,k _j ] The predetermined condition C ₁ is not satisfied.

Judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₂ in the same way, so, as shown in Figure 33, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. The selected 5-byte data is reused 51 times to obtain a total of 255 bytes to increase randomness. Each byte is composed of 8 bits, recorded as b _m,1 '...b _m,8 ', which means the 1st to 8th bits of the mth byte in 255 bytes, therefore, 255 bytes correspond to bits can be expressed as: When b _m,n '=1, V _bm,n '=1, when b _m,n '=0, V _bm,n '=-1, where b _m,n ' means b _m,1 '... For any one of b _m,8 ', the bits corresponding to 255 bytes are obtained according to the conversion relationship between b _m,n ' and V _bm,n ' to obtain matrix V _b ', which can be expressed as: Judging whether at least part of the data in the window W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₁ and whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₁ In the same way, so the matrix R is still used: Multiply the mth row of the matrix V _b ' with the random number of the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _bm '=V _bm,1 '*h _m,1 +V _{bm, 2} '*h _m,2 +...+V _bm,8 '*h _m,8 . According to this method, S _b1 ′, S _b2 ′… to S _b255 ′ are obtained, and the number K of values satisfying a specific condition (here greater than 0 is taken as an example) among S _b1 ′, S _b2 ′… to S _b255 ′ is counted. Since the matrix R obeys the normal distribution, S _bm ', like the matrix R, still obeys the normal distribution. According to the probability theory, the probability of a normal distribution random number being greater than 0 is 1/2. In S _b1 ', S _b2 '... To S _b255 ', the probability of each value being greater than 0 is 1/2, so K satisfies the binomial distribution: According to the statistical results, judge whether the number K of S _b1 ', S _b2 '... to S _b255 ' whose value is greater than 0 is an even number, and the probability that the random number of the binomial distribution is an even number is 1/2, so K is 1/ The probability of 2 satisfies the condition. When K is an even number, it indicates that at least part of the data in W _j2 [k _j -170, k _{j -1} ] does not meet the predetermined condition C ₂ when K is an odd number. Similarly, the method of judging whether at least part of the data in W _i3 [k _i -171, k _i -2] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [k _j -171, k _j -2] satisfies The way of the predetermined condition C ₃ is the same, and similarly, judge whether at least part of the data in W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ , judge W _j5 [k _j -173, k _j -4] Whether at least part of the data in W j6 [k j -174,k j -5] satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174,k _j -5] meets the predetermined condition C ₆ , and judge W _j7 [k _j -175,k _j -6] Whether at least part of the data in W j8 [k j -176,k j -7] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _j8 [k _j -176,k _j -7] meets the predetermined condition C ₈ , judge W _j9 [k _j -177,k _j -8] Whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined condition C ₁₀ and whether at least part of the data in W _j10 [k _j -179, k _{j -10} ] Whether or not at least some of the data satisfy the predetermined condition C ₁₁ will not be repeated here.

In this embodiment, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z , still taking the implementation shown in Figure 21 as an example, according to the deduplication The preset rule on the server 103 determines the window W _i1 [k _i -169, _ki ] for the potential segmentation point _ki , and judges whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , choose 5 In Figure 32, "■" indicates one byte selected, and the difference between two adjacent bytes selected "■" is 42 bytes. One of the implementation methods is to use the HASH function to calculate the selected 5 bytes. The value calculated by the HASH function is a fixed uniform distribution. If the value calculated by the HASH function is an even number, then judge W _i1 [k _i -169 , _ki ] at least part of the data satisfies the predetermined condition C ₁ , that is, C ₁ indicates that the value calculated by using the HASH function in the above manner is an even number. Therefore, the probability of whether at least part of the data in W _i1 [k _i −169, _ki ] satisfies the predetermined condition is 1/2. In the embodiment shown in FIG. 21, the Hash function is used to determine whether at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined conditions C ₂ and W _i3 [k _i -171, _ki -2 ] whether at least part of the data satisfies the predetermined condition C ₃ , whether at least part of the data in W _i4 [k _i -172, _ki -3] meets the predetermined condition C ₄ and W _i5 [k _i -173, _ki -4] Whether at least part of the data satisfies the predetermined condition C ₅ , for specific implementation, please refer to the method C ₁ that describes whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition by using the Hash function in the embodiment shown in FIG. 21 . This will not be repeated here.

When at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , jump 7 bytes from the potential split point _ki along the direction of data flow split point search, at the seventh The end position of the byte obtains the current potential segmentation point k _j , as shown in Figure 22, according to the rules preset for the deduplication server 103, determine the window W _j1 [k _j -169, k _j ] for the potential segmentation point k _j , The method of judging whether at least part of the data in the window W _j1 [k _j -169, k _j ] satisfies the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i1 [k _i -169, k _i ] meets the predetermined condition C ₁ The same way, so as shown in Figure 33, W _j1 represents the window W _j1 [k _j -169, k _j ], in order to judge whether at least part of the data in W _j1 [k _j -169, k _j ] satisfies the predetermined condition C ₁ , Select 5 bytes, "■" in Figure 33 indicates the selected 1 byte, and the difference between two adjacent selected bytes "■" is 42 bytes. Use the Hash function to calculate the 5 bytes selected from the window W _j1 [k _j -169, k _j ], if the obtained value is an even number, then at least part of the data in W _j1 [k _j -169, k _j ] meets the predetermined Condition C ₁ . In Fig. 33, the method of judging whether at least part of the data in W _i2 [k _i -170, k _i -1] satisfies the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] _The way to satisfy the predetermined condition C2 is the same, therefore, as shown in Fig. 33, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , and the adjacent two selected bytes There is a difference of 42 bytes between them. Use the Hash function to calculate the selected 5 bytes, and if the obtained value is an even number, then at least part of the data in W _j2 [k _j -170,k _j -1] satisfies the predetermined condition C ₂ . In Fig. 33, the method of judging whether at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [k _j -171, k _j -2] _The way to satisfy the predetermined condition C3 is the same, therefore, as shown in Fig. 33, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j3 [k _j -171,k _j -2] meets the predetermined condition C ₃ , and the adjacent two selected bytes There is a difference of 42 bytes between them. Use the Hash function to calculate the selected 5 bytes, and the value obtained is an even number, then at least part of the data in W _j3 [k _j -171, k _j -2] satisfies the predetermined condition C ₃ . In Fig. 33, the way of judging whether at least part of the data in W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ and judging at least part of the data in the window W _i4 [k _i -172, k _i -3] whether it satisfies the predetermined condition C ₄ way, therefore, as shown in Fig. 33, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ , and the adjacent two selected bytes There is a difference of 42 bytes between them. Use the Hash function to calculate the selected 5 bytes, and the value obtained is an even number, then at least part of the data in W _j4 [k _j -172, k _j -3] satisfies the predetermined condition C ₄ . According to the above method, judge whether at least part of the data in W _j5 [k _j -173, k _j -4] meets the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174, k _j -5] meet the predetermined condition Condition C ₆ , judging whether at least part of the data in W _j7 [k _j -175, k _j -6] meets the predetermined condition C ₇ , judging whether at least part of the data in W j8 [k _{j -176} , k _j -7] meets the predetermined Condition C ₈ , judging whether at least part of the data in W _j9 [k _j -177, k _j -8] meets the predetermined condition C ₉ , judging whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined The condition C ₁₀ and judging whether at least part of the data in W _j11 [k _j -179, k _j -10] satisfy the predetermined condition C ₁₁ will not be repeated here.

In this embodiment, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z . Taking the implementation shown in FIG. 21 as an example, according to the deduplication The preset rule on the server 103 determines the window W _i1 [k _i -169, _ki ] for the potential segmentation point _ki , and judges whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , select five Byte, the byte "■" with serial numbers 169, 127, 85, 43 and 1 in Fig. 32 represents one selected byte respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are respectively converted into a decimal value, expressed as a ₁ , a ₂ , a ₃ , a ₄ and a ₅ respectively. Because 1 byte is composed of 8 bits, each byte "■" is used as a value, and any a _r among a ₁ , a ₂ , a ₃ , a ₄ and a ₅ satisfies 0≤a _r ≤ 255. a ₁ , a ₂ , a ₃ , a ₄ and a ₅ form a 1*5 matrix. Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix R, which is expressed as:

According to the value of a ₁ and the column where it is located, find the corresponding value from the matrix R, such as a ₁ = 36, a ₁ is located in the first column, then look for the value corresponding to h _36,1 ; according to the value of a ₂ and where it is column, find the corresponding value from the matrix R, such as a ₂ = 48, a ₂ is located in the second column, then find the value corresponding to h _48,2 ; according to the value of a ₃ and the column where it is located, find the corresponding value from the matrix R value, such as a ₃ = 26, a ₃ is located in the third column, then look for the value corresponding to h _26,3 ; according to the value of a ₄ and the column where it is located, find the corresponding value from the matrix R, such as a ₄ = 26 , a ₄ is located in the fourth column, then search for the value corresponding to h _26,4 ; according to the value of a ₅ and the column where it is located, find the corresponding value from the matrix R, such as a ₅ = 88, a ₅ is located in the fifth column, Then find the value corresponding to h _88,5 . S ₁ =h _36,1 +h _48,2 +h _26,3 +h _26,4 +h _88,5 , because the matrix R obeys the binomial distribution, therefore, S ₁ also obeys the binomial distribution. When S ₁ is an even number, at least part of the data in W _i1 [k _i -169, _ki ] meets the predetermined condition C ₁ , and when S ₁ is an odd number, then at least part of the data in W _i1 [k _i -169, _ki ] If the predetermined condition C ₁ is not met, the probability that S ₁ is an even number is 1/2, and C ₁ indicates that S ₁ is an even number calculated in the above manner. In the embodiment shown in FIG. 21 , at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ . As shown in Figure 32, Represents the 1 byte selected when at least part of the data in the judgment window W _i2 [k _i -170, _ki -1] meets the predetermined condition C _2. In Fig. 32, sequence numbers 170, 128, 86, 44 are used and 2 indicate that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 respectively converted into a decimal value, respectively expressed as b ₁ , b ₂ , b ₃ , b ₄ and b ₅ . Since 1 byte consists of 8 bits, each byte As a numerical value, any b _r among b ₁ , b ₂ , b ₃ , b ₄ and b ₅ satisfies 0≤b _r ≤255. b ₁ , b ₂ , b ₃ , b ₄ and b ₅ form a 1*5 matrix. In this embodiment, the method of judging whether at least part of the data in W _i1 and W _i2 meets the predetermined conditions is the same, so the matrix R is still used, and the corresponding value is searched from the matrix R according to the value _of b1 and the column where it is located, such as b ₁ = 66, b ₁ is located in the first column, then search for the value corresponding to h _66,1 ; according to the value of b ₂ and the column where it is located, find the corresponding value from the matrix R, such as b ₂ = 48, b ₂ is in the first column 2 columns, then search for the value corresponding to h _48,2 ; according to the value of b ₃ and the column where it is located, find the corresponding value from the matrix R, such as b ₃ =99, b ₃ is located in the third column, then search for h _99, The value corresponding to ₃ ; according to the value of b ₄ and the column where it is located, find the corresponding value from the matrix R, such as b ₄ = 26, b ₄ is located in the fourth column, then find the value corresponding to h ₂₆ , 4; according to b ₅ Find the corresponding value from the matrix R, such as b ₅ =90, and b ₅ is located in the fifth column, then find the value corresponding to h _90,5 . S ₂ =h _66,1 +h _48,2 +h _99,3 +h _26,4 +h _90,5 , because the matrix R obeys the binomial distribution, therefore, S ₂ also obeys the binomial distribution. When S ₂ is an even number, at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ , and when S ₂ is an odd number, then W _i2 [k _i -170, _ki -1] At least some of the data do not satisfy the predetermined condition C ₂ , and the probability that S ₂ is an even number is 1/2. In the embodiment shown in FIG. 21 , at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [k _i -172, _ki -3] Satisfy the predetermined condition C ₄ , determine whether at least part of the data in W _i5 [k _i -173, _ki -4] meets the predetermined condition C ₅ , determine whether at least part of the data in W _i6 [k _i -174, _ki -5] Satisfy the predetermined condition C ₆ , judge whether at least part of the data in W _i7 [k _i -175, _ki -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -176, _ki -7] Satisfy the predetermined condition C ₈ , determine whether at least part of the data in W _i9 [k _i -177, _ki -8] meets the predetermined condition C ₉ , determine whether at least part of the data in W _i10 [k _i -178, _ki -9] Satisfying the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [k _i -179, _ki -10] satisfies the predetermined condition C ₁₁ . In the embodiment shown in Fig. 21, at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , and 7 jumps are made from the potential split point _ki along the direction of data stream split point search Byte, obtain the current potential segmentation point k _j at the end position of the seventh byte, as shown in Figure 22, according to the rules preset for the deduplication server 103, determine the window W _j1 [k _j for the potential segmentation point k _j -169,k _j ], the method of judging whether at least part of the data in the window W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ is the same as judging at least part of the data in the window W _i1 [k _i -169,k _i ] The method of whether the predetermined condition _C1 is satisfied is the same, so as shown in Figure 33, W _j1 represents the window W _j1 [k _j -169, k _j ], to judge that at least part of the data in W _j1 [k _j -169, k _j ] Whether the predetermined condition C ₁ is met, the bytes "■" with serial numbers 169, 127, 85, 43 and 1 in Fig. 33 represent 1 selected byte respectively, and the difference between two adjacent selected bytes is 42 characters Festival. Convert the bytes "■" with sequence numbers 169, 127, 85, 43 and 1 into a decimal value, which are respectively expressed as a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ '. Because 1 byte consists of 8 bits, each byte "■" is used as a value, and any a _r 'in a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' is Satisfy 0≤a _r '≤255. a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' form a 1*5 matrix. The method of judging whether at least part of the data in the window W _j1 [k _j -169, k _j ] satisfies the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i1 [k _i -169, k _i ] meets the predetermined condition C ₁ In the same way, therefore, still using the matrix R, expressed as:

According to the value of a ₁ ' and the column where it is located, find the corresponding value from the matrix R, such as a ₁ '=16, and a ₁ ' is located in the first column, then find the value corresponding to h _16,1 ; according to the value of a ₂ ' value and the column where it is located, find the corresponding value from the matrix R, such as a ₂ '=98, a ₂ 'is located in the second column, then find the value corresponding to h _98,2 ; according to the value of a ₃ ' and the column where it is located , look up the corresponding value from the matrix R, such as a ₃ '=56, a ₃ 'is located in the third column, then look for the value corresponding to h _56,3 ; according to the value and column of a ₄ ', from the matrix R Find the corresponding value, such as a ₄ '=36, a ₄ ' is located in the fourth column, then find the value corresponding to h _36,4 ; according to the value of a ₅ ' and the column where it is located, find the corresponding value from the matrix R, For example, a ₅ '=99, and a ₅ 'is located in the fifth column, then search for the value corresponding to h _99,5 . S ₁ '=h _16,1 +h _98,2 +h _56,3 +h _36,4 +h _99,5 , because the matrix R obeys the binomial distribution, therefore, S ₁ ' also obeys the binomial distribution. When S ₁ ' is an even number, at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ , and when S ₁ ' is an odd number, then at least part of the data in W _j1 [k _j -169,k _j ] Part of the data does not satisfy the predetermined condition C ₁ , and the probability that S ₁ ′ is an even number is 1/2.

Judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₂ in the same manner, so, as shown in Figure 33, Indicates the 1 byte selected when judging whether at least part of the data in the window W _j2 [k _j -170, k _j -1] satisfies the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes, Respectively represented by sequence numbers 170, 128, 86, 44 and 2, the difference between two adjacent selected bytes is 42 bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 converted into a decimal value, respectively, expressed as b ₁ ', b ₂ ', b ₃ ', b ₄ ' and b ₅ '. Since 1 byte consists of 8 bits, each byte As a numerical value, b _r ' among b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ satisfies _0≤br ′≤255. b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ form a 1*5 matrix. Use the same matrix R as judging whether at least part of the data in the window W _i2 [k _i -170, _{ki -1} ] meets the predetermined condition C ₂ , and find the corresponding Value, such as b ₁ '=210, b ₁ 'is located in the first column, then find the value corresponding to h _210,1 ; according to the value and column of b ₂ ', find the corresponding value from the matrix R, such as b ₂ '=156, b ₂ 'is located in the second column, then search for the value corresponding to h _156,2 ; according to the value of b ₃ ' and the column where it is located, find the corresponding value from the matrix R, such as b ₃ '=144, b ₃ ' is located in the third column, then search for the value corresponding to h _144,3 ; according to the value of b ₄ ' and the column where it is located, find the corresponding value from the matrix R, such as b ₄ '=60, b ₄ 'is in the 4th column, then look for the value corresponding to h _60,4 ; look up the corresponding value from the matrix R according to the value of b ₅ ' and the column where it is located, such as b ₅ '=90, b ₅ 'is in the fifth column, then look for h The value corresponding to _90,5 . S ₂ '=h _210,1 +h _156,2 +h _144,3 +h _60,4 +h _90,5 , same as S ₂ judgment conditions, when S ₂ ' is an even number, then W _j2 [k _j -170, k _j -1] at least part of the data satisfies the predetermined condition C ₂ , when S ₂ ' is an odd number, then at least part of the data in W _j2 [k _j -170, k _j -1] does not meet the predetermined condition C ₂ , The probability that S ₂ ' is an even number is 1/2.

Similarly, the method of judging whether at least part of the data in W _i3 [k _i -171, k _i -2] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [k _j -171, k _j -2] satisfies The way of the predetermined condition C ₃ is the same, and similarly, judge whether at least part of the data in W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ , judge W _j5 [k _j -173, k _j -4] Whether at least part of the data in W j6 [k j -174,k j -5] satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174,k _j -5] meets the predetermined condition C ₆ , and judge W _j7 [k _j -175,k _j -6] Whether at least part of the data in W j8 [k j -176,k j -7] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _j8 [k _j -176,k _j -7] meets the predetermined condition C ₈ , judge W _j9 [k _j -177,k _j -8] Whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined condition C ₁₀ and whether at least part of the data in W _j10 [k _j -179, k _{j -10} ] Whether or not at least some of the data satisfy the predetermined condition C ₁₁ will not be repeated here.

In this embodiment, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z . Taking the implementation shown in FIG. 21 as an example, according to the deduplication The preset rule on the server 103 determines the window W _i1 [k _i -169, _ki ] for the potential segmentation point _ki , and judges whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , choose 5 byte, the byte "■" with serial number 169, 127, 85, 43 and 1 in Fig. 32 represents one byte selected respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are respectively converted into a decimal value, expressed as a ₁ , a ₂ , a ₃ , a ₄ and a ₅ respectively. Because 1 byte consists of 8 bits, each byte "■" is used as a value, and any a _s in a ₁ , a ₂ , a ₃ , a ₄ and a ₅ satisfies 0≤a _s ≤ 255. a ₁ , a ₂ , a ₃ , a ₄ and a ₅ form a 1*5 matrix. Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix R, which is expressed as: Select 256*5 random numbers from the random numbers subject to the binomial distribution to form a matrix G, which is expressed as:

According to the value of a ₁ and the column where it is located, such as a ₁ = 36, and a ₁ is located in the first column, then look up the value corresponding to h _36,1 from the matrix R, and look up the value corresponding to g _36,1 from the matrix G ;According to the value of a ₂ and the column where it is located, such as a ₂ = 48, a ₂ is located in the second column, then look up the value corresponding to h _48,2 from the matrix R, and look up the value corresponding to g _48,2 from the matrix G ;According to the value of a ₃ and the column where it is located, such as a ₃ =26, a ₃ is located in the third column, then look up the value corresponding to h _26,3 from the matrix R, and look up the value corresponding to g _26,3 from the matrix G ;According to the value of a ₄ and the column where it is located, such as a ₄ =26, a ₄ is located in the fourth column, then look up the value corresponding to h _26,4 from the matrix R, and look up the value corresponding to g _26,4 from the matrix G ;According to the value of a ₅ and the column where it is located, as a ₅ =88, a ₅ is located in the fifth column, then look up the value corresponding to h _88,5 from the matrix R, and look up the value corresponding to g _88,5 from the matrix G . S _1h ＝h _36,1 +h _48,2 +h _26,3 +h _26,4 +h _88,5 , because matrix R obeys binomial distribution, therefore, S _1h also obeys binomial distribution; S _1g ＝g _36,1 +g _48,2 +g _26,3 +g _26,4 +g _88,5 , because matrix G obeys binomial distribution, so S _1g also obeys binomial distribution. When one of S _1h and S _1g is an even number, at least part of the data in W _i1 [k _i -169,k _i ] meets the predetermined condition C ₁ , and when S _1h and S _1g are both odd numbers, then W _i1 [k _i -169, _ki ] at least part of the data do not meet the predetermined condition C ₁ , C ₁ means that one of S _1h and S _1g obtained by the above method is an even number. Because both S _1h and S _1g follow the binomial distribution, the probability that S _1h is even is 1/2, the probability that S _1g is even is 1/2, and the probability that one of S _1h and S _1g is even is 1 -1/4=3/4, therefore, the probability that at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ is 3/4. In the embodiment shown in FIG. 21 , at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ . In the embodiment shown in FIG. 21 , in W _i1 [k _i -169, _ki ], W _i2 [k _i -170, _ki -1], W _i3 [k _i -171, _ki -2] , W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4], W _i6 [k _i -174,k _i -5], W _i7 [k _i -175 ,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177,k _i -8], W _i10 [k _i -178,k _i -9] and W In _i11 [k _i -179, k _i -10], the size of each window is the same, that is, the size of the window is 169 bytes, and the method of judging whether at least part of the data in the window meets the predetermined condition is also the same, see the above judgment W _i1 for details Whether at least part of the data in [k _i −169, _ki ] satisfies the description of the predetermined condition C ₁ . Therefore, as shown in Figure 32, Represents the 1 byte selected when at least part of the data in the judgment window W _i2 [k _i -170, _ki -1] meets the predetermined condition C _2. In Fig. 32, sequence numbers 170, 128, 86, 44 are used and 2 indicate that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 respectively converted into a decimal value, respectively expressed as b ₁ , b ₂ , b ₃ , b ₄ and b ₅ . Since 1 byte consists of 8 bits, each byte As a numerical value, any b _s among b ₁ , b ₂ , b ₃ , b ₄ and b ₅ satisfies 0≤b _s ≤255. b ₁ , b ₂ , b ₃ , b ₄ and b ₅ form a 1*5 matrix. In this embodiment, the method of judging whether at least part of the data in each window satisfies the predetermined condition is the same, so the same matrices R and G are still used. According to the value of b ₁ and the column where it is located, such as b ₁ = 66, b ₁ is located in the first column, then look up the value corresponding to h _66,1 from the matrix R, and look up the value corresponding to g _66,1 from the matrix G; According to the value of b2 and the column where it is located, as b2=48, b2 is located in the _second column, then look _{up the value corresponding to h 48,2 from} matrix R, and look up the value corresponding to _{g 48,2} from matrix G; According to the value of b ₃ and the column where it is located, as b ₃ =99, b ₃ is located in the third column, then the value corresponding to h _99,3 is searched from matrix R, and the value corresponding to g _99,3 is searched from matrix G; According to the value of b ₄ and the column where it is located, as b ₄ =26, b ₄ is located in the fourth column, then look up the value corresponding to h _26,4 from the matrix R, and look up the value corresponding to g _26,4 from the matrix G; According to the value of b ₅ and the column where it is located, for example, b ₅ =90, and b ₅ is located in the fifth column, the value corresponding to h _90,5 is searched from the matrix R, and the value corresponding to g _90,5 is searched from the matrix G. S _2h =h _66,1 +h _48,2 +h _99,3 +h _26,4 +h _90,5 , because the matrix R obeys the binomial distribution, therefore, S _2h also obeys the binomial distribution. S _2g =g _66,1 +g _48,2 +g _99,3 +g _26,4 +g _90,5 , because the matrix G obeys the binomial distribution, therefore, S _2g also obeys the binomial distribution. When one of S _2h and S _2g is an even number, at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ , and when both S _2h and S _2g are odd numbers, then W _i2 At least part of the data in [k _i -170, _ki -1] does not satisfy the predetermined condition C ₂ , and the probability that one of S _2h and S _2g is an even number is 3/4. In the embodiment shown in FIG. 21 , at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [k _i -172, _ki -3] Satisfy the predetermined condition C ₄ , determine whether at least part of the data in W _i5 [k _i -173, _ki -4] meets the predetermined condition C ₅ , determine whether at least part of the data in W _i6 [k _i -174, _ki -5] Satisfy the predetermined condition C ₆ , judge whether at least part of the data in W _i7 [k _i -175, _ki -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -176, _ki -7] Satisfy the predetermined condition C ₈ , determine whether at least part of the data in W _i9 [k _i -177, _ki -8] meets the predetermined condition C ₉ , determine whether at least part of the data in W _i10 [k _i -178, _ki -9] Satisfying the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [k _i -179, _ki -10] satisfies the predetermined condition C ₁₁ . In the embodiment shown in Fig. 21, at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , and 7 jumps are made from the potential split point _ki along the direction of data stream split point search Byte, obtain the current potential segmentation point k _j at the end position of the seventh byte, as shown in Figure 22, according to the rules preset for the deduplication server 103, determine the window W _j1 [k _j for the potential segmentation point k _j -169,k _j ], the method of judging whether at least part of the data in the window W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ is the same as judging at least part of the data in the window W _i1 [k _i -169,k _i ] The method of whether the predetermined condition _C1 is satisfied is the same, so as shown in Figure 33, W _j1 represents the window W _j1 [k _j -169, k _j ], in order to judge that at least part of the data in W _j1 [k _j -169, k _j ] Whether the predetermined condition C ₁ is met, the bytes "■" with serial numbers 169, 127, 85, 43 and 1 in Fig. 33 respectively represent 1 selected byte, and the difference between two adjacent selected bytes is 42 characters Festival. Convert the bytes "■" with sequence numbers 169, 127, 85, 43, and 1 into a decimal value, which are respectively expressed as a ₁ ', a ₂ ', a ₃ ', a ₄ ', and a ₅ '. Because 1 byte consists of 8 bits, each byte "■" is used as a value, and any a _s 'in a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' is Satisfy 0≤a _s '≤255. a ₁ ', a ₂ ', a ₃ ', a ₄ ' and a ₅ ' form a 1*5 matrix. Using the same matrix R and G as judging whether at least part of the data in the window W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ is expressed as: and

According to the value of a ₁ ' and the column where it is located, such as a ₁ '=16, and a ₁ 'is located in the first column, then look up the value corresponding to h _16,1 from the matrix R, and find the corresponding value of g _16,1 from the matrix G value; according to the value of a ₂ ' and the column where it is located, such as a ₂ '=98, a ₂ 'is located in the second column, then look up the value corresponding to h ₉₈ and 2 from the matrix R, and look up g ₉₈ from the matrix G _{, the value corresponding to 2} ; according to the value of a ₃ ' and the column where it is located, such as a ₃ '=56, a ₃ 'is located in the third column, then look up the value corresponding to h _56,3 from the matrix R, and from the matrix G Find the value corresponding to g _56,3 ; according to the value of a ₄ ' and the column where it is located, such as a ₄ '=36, a ₄ ' is located in the fourth column, then find the value corresponding to h _36,4 from the matrix R, from Find the value corresponding to g _36,4 in the matrix G; according to the value of a ₅ ' and the column where it is located, such as a ₅ '=99, a ₅ ' is located in the fifth column, then find the value corresponding to h _99,5 from the matrix R value, find the value corresponding to g _99,5 from the matrix G. S _1h '=h _16,1 +h _98,2 +h _56,3 +h _36,4 +h _99,5 , because matrix R obeys binomial distribution, therefore, S _1h ' also obeys binomial distribution; S _1g '=g _16,1 +g _98,2 +g _56,3 +g _36,4 +g _99,5 , because the matrix G obeys the binomial distribution, so S _1g ' also obeys the binomial distribution. When one of S _1h ' and S _1g ' is an even number, then at least part of the data in W _j1 [k _j -169,k _j ] satisfies the predetermined condition C ₁ , and when both S _1h ' and S _1g ' are odd numbers, then At least part of the data in W _j1 [k _j -169,k _j ] does not satisfy the predetermined condition C ₁ , and the probability that one of S _1h ' and S _1g ' is an even number is 3/4.

Judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₂ in the same way, so, as shown in Figure 33, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. In FIG. 33 , they are represented by sequence numbers 170, 128, 86, 44 and 2 respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 converted into a decimal value, respectively, expressed as b ₁ ', b ₂ ', b ₃ ', b ₄ ' and b ₅ '. Since 1 byte consists of 8 bits, each byte As a numerical value, any one of b _s ' among b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ satisfies 0≤b _s '≤255. b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′, and b ₅ ′ form a 1*5 matrix. Using the same matrix R and G as judging whether at least part of the data in the window W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , according to the value of b ₁ ' and the column where it is located, such as b ₁ '= 210, b ₁ ' is located in the first column, then look up the value corresponding to h _210,1 from the matrix R, and find the value corresponding to g _210,1 from the matrix G; according to the value of b ₂ ' and the column where it is located, such as b ₂ '=156, b ₂ 'is located in the second column, then look up the value corresponding to h _156,2 from the matrix R, and look up the value corresponding to g _156,2 from the matrix G; according to the value of b ₃ ' and the column where it is located , such as b ₃ '=144, b ₃ 'is located in the third column, then look up the value corresponding to h _144,3 from the matrix R, and look up the value corresponding to g _144,3 from the matrix G; according to the value of b ₄ ' and The column where it is located, such as b ₄ '=60, b ₄ 'is located in the fourth column, then look up the value corresponding to h _60,4 from the matrix R, and look up the value corresponding to g _60,4 from the matrix G; according to b ₅ ' The value and the column where b ₅ '=90, b ₅ 'is located in the fifth column, then look up the value corresponding to h _90,5 from the matrix R, and look up the value corresponding to g _90,5 from the matrix G. S _2h '=h _210,1 +h _156,2 +h _144,3 +h _60,4 +h _90,5 , S _2g '=g _210,1 +g _156,2 +g _144,3 +g _{60 ,4} +g _90,5 . When one of S _2h ' and S _2g ' is an even number, then at least part of the data in W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , when both S _2h ' and S _2g ' are odd , then at least part of the data in W _j2 [k _j -170,k _j -1] does not meet the predetermined condition C ₂ , and the probability that one of S _2h ' and S _2g ' is an even number is 3/4.

Similarly, the method of judging whether at least part of the data in W _i3 [k _i -171, k _i -2] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [k _j -171, k _j -2] satisfies The way of the predetermined condition C ₃ is the same, and similarly, judge whether at least part of the data in W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ , judge W _j5 [k _j -173, k _j -4] Whether at least part of the data in W j6 [k j -174,k j -5] satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174,k _j -5] meets the predetermined condition C ₆ , and judge W _j7 [k _j -175,k _j -6] Whether at least part of the data in W j8 [k j -176,k j -7] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _j8 [k _j -176,k _j -7] meets the predetermined condition C ₈ , judge W _j9 [k _j -177,k _j -8] Whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined condition C ₁₀ and whether at least part of the data in W _j10 [k _j -179, k _{j -10} ] Whether or not at least some of the data satisfy the predetermined condition C ₁₁ will not be repeated here.

In this embodiment, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z . Taking the implementation shown in FIG. 21 as an example, according to the deduplication The preset rule on the server 103 determines the window W _i1 [k _i -169, _ki ] for the potential segmentation point _ki , and judges whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , choose 5 byte, the byte "■" with serial number 169, 127, 85, 43 and 1 in Fig. 32 represents one byte selected respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are regarded as 40 bits in turn, which are expressed as a ₁ , a ₂ , a ₃ , a ₄ ...a ₄₀ respectively. For any at of a ₁ , a ₂ , a ₃ , a ₄ ...a ₄₀ , when _at = 0, V _at = _-1 , when _at = 1, V _at = 1, according to _at and _Vat correspondence relationship generates V _a1 , V _a2 , V _a3 , V _a4 . . . V _a40 . Select 40 random numbers from the random numbers subject to the normal distribution, respectively denoted as: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ . S _a =V _a1 *h ₁ +V _a2 *h ₂ +V _a3 *h ₃ +V _a4 *h ₄ + . . . +V _a40 *h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey the normal distribution, therefore, S _a also obeys the normal distribution. When S _a is a positive number, at least part of the data in W _i1 [ _ki -169, _ki ] meets the predetermined condition C ₁ , when S _a is negative or 0, then in W _i1 [ _ki -169, _ki ] At least part of the data does not satisfy the predetermined condition C ₁ , and the probability that S _a is a positive number is 1/2. In the embodiment shown in FIG. 21 , at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ . As shown in Figure 32, Represents the 1 byte selected when at least part of the data in the judgment window W _i2 [k _i -170, _ki -1] meets the predetermined condition C _2. In Fig. 32, sequence numbers 170, 128, 86, 44 are used and 2 indicate that there is a difference of 42 bytes between two adjacent selected bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 They are regarded as 40 bits in turn, represented as b ₁ , b ₂ , b ₃ , b ₄ . . . b ₄₀ . For any b _t in b ₁ , b ₂ , b ₃ , b ₄ ... b ₄₀ , when b _t =0, V _bt =-1, when b _t =1, V _bt =1, according to b _t and The V _bt correspondence relationship generates V _b1 , V _b2 , V _b3 , V _b4 . . . V _b40 . The method of judging whether at least part of the data in the window W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ is the same as judging whether at least part of the data in the window W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ in the same way, therefore, use the same random numbers: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ , S _b =V _b1 *h ₁ +V _b2 *h ₂ +V _b3 *h ₃ +V _b4 *h ₄ +...+V _b40 *h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _b also obeys normal distribution. When S _b is a positive number, at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ , and when S _b is negative or 0, then W _i2 [k _i -170, _ki -1] -1] at least part of the data does not meet the predetermined condition C ₂ , the probability that S _b is a positive number is 1/2. In the embodiment shown in FIG. 21 , at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ . Using the same rule, judge whether at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [k _i -172, _ki -3] Satisfy the predetermined condition C ₄ , determine whether at least part of the data in W _i5 [k _i -173, _ki -4] meets the predetermined condition C ₅ , determine whether at least part of the data in W _i6 [k _i -174, _ki -5] Satisfy the predetermined condition C ₆ , judge whether at least part of the data in W _i7 [k _i -175, _ki -6] meets the predetermined condition C ₇ , judge whether at least part of the data in W _i8 [k _i -176, _ki -7] Satisfy the predetermined condition C ₈ , determine whether at least part of the data in W _i9 [k _i -177, _ki -8] meets the predetermined condition C ₉ , determine whether at least part of the data in W _i10 [k _i -178, _ki -9] Satisfying the predetermined condition C ₁₀ and judging whether at least part of the data in W _i11 [k _i -179, _ki -10] satisfies the predetermined condition C ₁₁ . In the embodiment shown in Fig. 21, at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , and 7 jumps are made from the potential split point _ki along the direction of data stream split point search Byte, obtain the current potential segmentation point k _j at the end position of the seventh byte, as shown in Figure 22, according to the rules preset for the deduplication server 103, determine the window W _j1 [k _j for the potential segmentation point k _j -169,k _j ], the method of judging whether at least part of the data in the window W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ is the same as judging at least part of the data in the window W _i1 [k _i -169,k _i ] The method of whether the predetermined condition _C1 is satisfied is the same, so as shown in Figure 33, W _j1 represents the window W _j1 [k _j -169, k _j ], in order to judge that at least part of the data in W _j1 [k _j -169, k _j ] Whether the predetermined condition C ₁ is met, select 5 bytes, and the bytes "■" with serial numbers 169, 127, 85, 43 and 1 in Figure 33 respectively represent the selected 1 byte, and the adjacent two selected bytes There is a difference of 42 bytes between them. The bytes "■" with sequence numbers 169, 127, 85, 43 and 1 are regarded as 40 bits in sequence, which are represented as a ₁ ', a ₂ ', a ₃ ', a ₄ '...a ₄₀ ' respectively. For any a _t 'in a ₁ ', a ₂ ', a ₃ ', a ₄ '...a ₄₀ ', when a _t '=0, V _at '=-1, when a _t '=1, V _at '=1, V _a1 ', V _a2 ', V _a3 ', V _a4 '...V _a40 ' _are generated according to the correspondence between at ' and V _at '. The method of judging whether at least part of the data in the window W _j1 [k _j -169, k _j ] satisfies the predetermined condition C ₁ is the same as the method of judging whether at least part of the data in the window W _i1 [k _i -169, k _i ] meets the predetermined condition C ₁ In the same way, so the same random numbers are used: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ . S _a '=V _a1 '*h ₁ +V _a2 '*h ₂ +V _a3 '*h ₃ +V _a4 '*h ₄ +...+V _a40 '*h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _a ' also obeys normal distribution. When S _a 'is a positive number, at least part of the data in W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ , and when S _a ' is negative or 0, then W _j1 [k _j -169,k _j ] at least part of the data does not meet the predetermined condition C ₁ , the probability that S _a ' is a positive number is 1/2.

Judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₂ in the same way, so, as shown in Figure 33, Indicates one byte selected when judging whether at least part of the data in the window W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , and the difference between two adjacent selected bytes is 42 bytes. In FIG. 33 , they are represented by sequence numbers 170, 128, 86, 44 and 2 respectively, and the difference between two adjacent selected bytes is 42 bytes. The bytes with sequence numbers 170, 128, 86, 44, and 2 It is regarded as 40 bits in turn, which are respectively expressed as b ₁ ′, b ₂ ′, b ₃ ′, b ₄ ′…b ₄₀ ′. Any b _t 'in b ₁ ', b ₂ ', b ₃ ', b ₄ '...b ₄₀ ', when b _t '=0, V _bt '=-1, when b _t '=1, V _bt '=1, V _b1 ', V _b2 ', V _b3 ', V _b4 '...V _b40 ' are generated according to the corresponding relationship between b _t ' and V _bt '. Judging whether at least part of the data in W _i2 [k _i -170, _ki -1] meets the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meets the predetermined condition C ₂ in the same way, therefore, use the same random numbers: h ₁ , h ₂ , h ₃ , h ₄ ... h ₄₀ , S _b '=V _b1 '*h ₁ +V _b2 '*h ₂ +V _b3 '*h ₃ +V _b4 '*h ₄ +...+V _b40 '*h ₄₀ . Because h ₁ , h ₂ , h ₃ , h ₄ . . . h ₄₀ obey normal distribution, therefore, S _b ' also obeys normal distribution. When S _b ' is a positive number, at least part of the data in W _j2 [k _j -170,k _j -1] meets the predetermined condition C ₂ , and when S _b ' is negative or 0, then W _j2 [k _j -170, At least part of the data in k _j -1] does not satisfy the predetermined condition C ₂ , and the probability that S _b ' is a positive number is 1/2.

Similarly, the method of judging whether at least part of the data in W _i3 [k _i -171, k _i -2] satisfies the predetermined condition C ₃ is the same as judging whether at least part of the data in W _j3 [k _j -171, k _j -2] satisfies The way of the predetermined condition C ₃ is the same, and similarly, judge whether at least part of the data in W _j4 [k _j -172, k _j -3] meets the predetermined condition C ₄ , judge W _j5 [k _j -173, k _j -4] Whether at least part of the data in W j6 [k j -174,k j -5] satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174,k _j -5] meets the predetermined condition C ₆ , and judge W _j7 [k _j -175,k _j -6] Whether at least part of the data in W j8 [k j -176,k j -7] satisfies the predetermined condition C ₇ , judge whether at least part of the data in W _j8 [k _j -176,k _j -7] meets the predetermined condition C ₈ , judge W _j9 [k _j -177,k _j -8] Whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined condition C ₁₀ and whether at least part of the data in W _j10 [k _j -179, k _{j -10} ] Whether or not at least some of the data satisfy the predetermined condition C ₁₁ will not be repeated here.

In this embodiment, a random function is used to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z , still taking the implementation shown in Figure 21 as an example, according to the deduplication The preset rule on the server 103 determines the window W _i1 [k _i -169, _ki ] for the potential segmentation point _ki , and judges whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , as shown in Figure 32, W _i1 represents the window W _i1 [k _i -169, _ki ], in order to judge whether at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ , choose 5 byte, the byte "■" with serial number 169, 127, 85, 43 and 1 in Fig. 32 represents one byte selected respectively, and the difference between two adjacent selected bytes is 42 bytes. Convert the byte "■" with serial numbers 169, 127, 85, 43 and 1 into a decimal number in the range of 0-(2^40-1), using a uniformly distributed random number generator as 0-(2^ Each decimal number in 40-1) generates a specified value, and records the correspondence R between each decimal number in 0-(2^40-1) and the specified value. Once specified, the corresponding decimal number The specified value remains unchanged, and the specified value obeys the uniform distribution. If the specified value is an even number, then at least part of the data in W _i1 [k _i -169, _ki ] satisfies the predetermined condition C ₁ . If the specified value is an odd number, then At least part of the data in W _i1 [k _i −169, _ki ] does not satisfy the predetermined condition C ₁ , and C ₁ indicates that the specified value obtained by the above method is an even number. Because the probability that a uniformly distributed random number is an even number is 1/2, therefore, the probability that at least part of the data in W _i1 [k _i −169, _ki ] satisfies the predetermined condition C ₁ is 1/2. In the embodiment shown in FIG. 21 , the same rules are used to determine whether at least part of the data in W _i2 [k _i -170, _ki -1] satisfies the predetermined condition C ₂ , and determine whether W _i3 [k _i -171, Whether at least part of the data in _ki -2] meets the predetermined condition C ₃ , judge whether at least part of the data in W _i4 [ _ki -172, _ki -3] meets the predetermined condition C ₄ , and judge W _i5 [ _ki -173, Whether at least part of the data in k _i -4] satisfies the predetermined condition C ₅ will not be repeated here.

When at least part of the data in W _i5 [k _i -173, _ki -4] does not meet the predetermined condition C ₅ , jump 7 bytes from the potential split point _ki along the data flow split point search direction, and at the seventh word The end position of the section obtains the current potential segmentation point k _j , as shown in Figure 22, according to the rules preset for the deduplication server 103, determine the window W _j1 [k _j -169, k _j ] for the potential segmentation point k _j , judge The method of whether at least part of the data in the window W _j1 [k _j -169,k _j ] meets the predetermined condition C ₁ and the method of judging whether at least part of the data in the window W _i1 [k _i -169,k _i ] meet the predetermined condition C ₁ The same, therefore, use the corresponding relationship R between each decimal number in the same 0-(2^40-1) and the specified value, as shown in Figure 33, W _j1 represents the window, for judging W _j1 [k _j -169, k _j ] Whether at least part of the data satisfies the predetermined condition C ₁ , select 5 bytes, "■" in Figure 33 represents the selected 1 byte, between two adjacent selected bytes "■" A difference of 42 bytes. Convert the byte "■" with serial numbers 169, 127, 85, 43 and 1 into a decimal number, and find the specified value corresponding to the decimal number in R. If the specified value is even, then W _j1 [k _j - 169,k _j ] at least part of the data meet the predetermined condition C ₁ , if the specified value is an odd number, then at least part of the data in W _j1 [k _j -169,k _j ] does not meet the predetermined condition C ₁ , because the random The probability that the number is even is 1/2, therefore, the probability that at least part of the data in W _j1 [k _j -169, k _j ] satisfies the predetermined condition C ₁ is 1/2. Similarly, the method of judging whether at least part of the data in W _i2 [k _i -170, k _i -1] satisfies the predetermined condition C ₂ and judging whether at least part of the data in W _j2 [k _j -170, k _j -1] meet The method of the predetermined condition C ₂ is the same, the method of judging whether at least part of the data in W _i3 [k _i -171, _ki -2] satisfies the predetermined condition C ₃ is the same as that of judging W _j3 [k _j -171, k _j -2] Whether at least part of the data satisfies the predetermined condition C ₃ is the same. Similarly, judge whether at least part of the data in W _j4 [k _j -172,k _j -3] meets the predetermined condition C ₄ , and judge W _j5 [k _j -173, Whether at least part of the data in k _j -4] satisfies the predetermined condition C ₅ , judge whether at least part of the data in W _j6 [k _j -174,k _j -5] meets the predetermined condition C ₆ , judge W _j7 [k _j -175, Whether at least part of the data in k _j -6] satisfies the predetermined condition C ₇ , judging whether at least part of the data in W j8 [k _{j -176} ,k _j -7] meets the predetermined condition C ₈ , judging W _j9 [k _j -177, Whether at least part of the data in k _j -8] meets the predetermined condition C ₉ , judging whether at least part of the data in W _j10 [k _j -178, k _j -9] meets the predetermined condition C ₁₀ and judging W _j11 [k _j -179, Whether at least part of the data in k _j -10] satisfies the predetermined condition C ₁₁ will not be repeated here.

The deduplication server 103 in the embodiment of the present invention shown in Figure 1 refers to the device capable of implementing the technical solution described in the embodiment of the present invention, as shown in Figure 18, generally includes a central processing unit, a main memory, and an input and output interface . The central processing unit, the main memory, and the input and output interfaces communicate with each other, the main memory stores executable instructions, and the central processing unit executes the executable instructions stored in the main memory, thereby performing specific functions, so that the deduplication server 103 has specific functions, Find the data flow split point as described in FIG. 20 to FIG. 33 in the embodiment of the present invention. Therefore, as shown in FIG. 19, according to the embodiments of the present invention shown in FIG. 20 to FIG. 33, the deduplication server 103 has rules preset on the deduplication server 103, and the rules are: determine M The window W _x [kA _x ,k+B _x ] and the predetermined condition C _x corresponding to the window W _x [kA _x ,k+B _x ], where x is a continuous natural number from 1 to M, M≥2, A _x and B _x is an integer;

The deduplication server 103 includes a determination unit 1901 and a judgment processing unit 1902 . Wherein, the determination unit 1901 is used to perform step a):

a) Determine the corresponding window W _iz [k _i -A _z , _ki +B _z ] for the current potential segmentation point ki according to the rules, _i and z are integers, and 1≤z≤M;

A judging processing unit 1902, configured to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] satisfies a predetermined condition C _z ;

When at least part of the data in the window W _iz [k _i -A _z , _ki + B _z ] does not meet the predetermined condition C _z , search for the direction from the current potential split point _ki along the data flow split point Skip N data stream segmentation point minimum search unit U, N*U is not greater than ‖B _z ‖+max _x (‖A _x ‖) to obtain a new potential segmentation point, then the determination unit 1901 is the new potential The split point executes step a);

When at least part of the data in each of the M windows W _ix [k _i -A _x , _ki +B _x ] of the current potential segmentation point k _i satisfies the predetermined condition C _x , then the current potential segmentation point k _i is the data stream split point.

Further, the rule also includes: at least two windows W _ie [k _i -A _e , _ki +B _e ] and W _if [k _i -A _f , _ki +B _f ], satisfying the condition: |A _e +B _e |=|A _f +B _f |, C _e =C _f . Further, the rule further includes: A _e and A _f are positive integers. Further, the rule further includes: A _e −1=A _f , _Be +1=B _f .

Further, the judging processing unit 1902 is specifically configured to use a random function to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z . Furthermore, the judging processing unit 1902 specifically uses a hash function to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

Further, the _judging processing unit ₁₉₀₂ is _configured to _select from the current potential _{segmentation point ki} Jumping N minimum search units U of data stream segmentation points along the search direction of the data stream segmentation point to obtain the new potential segmentation point, the determining unit 1901 performs step a) for the new potential segmentation point, according to the According to the above rules, the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is the same as the window W _iz [k _i -A _z , _ki +B _z ] or the left boundary of the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is located at the window W _iz [k _i -A _z , k _i +B _z ] range; wherein, the window W _ic [k _i -A _c , _ki +B _c ] determined for the new potential segmentation point is based on the rule, for the new The M windows determined by the potential splitting points are the first windows in the sequence obtained according to the search direction of the data flow.

Further, the judgment processing unit 1902 uses a random function to judge whether at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z , specifically including:

Select F bytes in the window W _iz [k _i -A _z , _ki +B _z ], use the F bytes repeatedly H times, and obtain F*H bytes in total, where each A byte is composed of 8 bits, recorded as a _m,1 ... a _m,8 , indicating the 1st to 8th bits of the mth byte in the F*H bytes, and the F*H bytes The corresponding bits can be expressed as: When a _m,n =1, V _am,n =1, when a _m,n =0, V _am,n =-1, where a _m,n represents a _m,1 ... a _m,8 Any one, the bits corresponding to the F*H bytes obtain a matrix V _a according to the conversion relationship between am, _n and V _{am, n} , and the matrix V _a is expressed as: Select F*H*8 random numbers from the random numbers of the service normal distribution to form a matrix R, and the matrix R is expressed as: Multiply the mth row of the matrix V _a by the random number in the mth row of the matrix R, and then sum to obtain a value, specifically expressed as S _am =V _am,1 *h _m,1 +V _{am ,2} *h _m,2 +…+V _am,8 *h _m,8 , similarly, obtain S _a1 , S _a2 ... to S _aF*H , count S _a1 , S _a2 ... to S _aF*H to satisfy The number K of values greater than 0, when K is an even number, at least part of the data in the window W _iz [k _i -A _z , _ki +B _z ] satisfies the predetermined condition C _z .

According to the server-based method for searching data stream segmentation points provided by the embodiments of the present invention shown in 20 to FIG. 33 , a window W _ix [ _ki -A _x , _ki +B _x ] is determined for a potential segmentation point _ki , where , x are consecutive natural numbers from 1 to M, and M≥2, it is possible to judge in parallel whether at least part of the data in each of the M windows satisfies the predetermined condition C _x , or sequentially judge whether at least part of the data in the window satisfies the predetermined condition, or When at least part of the data in the window W _i1 [k _i -A ₁ , _ki +B ₁ ] satisfies the predetermined condition C ₁ , at least part of the data in W _i2 [k _i -A ₂ , _ki +B ₂ ] can be judged When the data satisfies the predetermined condition C ₂ , until it is judged that at least part of the data in W _im [k _i -A _m , _ki + B _m ] satisfies the predetermined condition C _m . The determination of other windows in the embodiment is the same as this, and will not be repeated here.

In addition, according to the embodiments of the present invention shown in Figures 20 to 33, rules are preset on the deduplication server 103, the rules: determine M windows W _x [kA _x , k+B _x ] for a potential segmentation point k The predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ], x is a continuous natural number from 1 to M, and M≥2. In this preset rule, A ₁ , A ₂ , A ₃ ... All A _m may not be equal, B ₁ , B ₂ , B ₃ . . . B _m may not all be equal, and C ₁ , C ₂ , C ₃ . . . C _M may not all be equal. In the embodiment shown in FIG. 21 , in W _i1 [k _i -169, _ki ], W _i2 [k _i -170, _ki -1], W _i3 [k _i -171, _ki -2] , W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4], W _i6 [k _i -174,k _i -5], W _i7 [k _i -175 ,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177,k _i -8], W _i10 [k _i -178,k _i -9] and W In _i11 [k _i -179, k _i -10], the size of each window is the same, that is, the size of the window is 169 bytes, and the method of judging whether at least part of the data in the window meets the predetermined condition is also the same, see the above judgment W _i1 for details Whether at least part of the data in [k _i -169, _ki ] satisfies the description of the predetermined condition C ₁ , but in the embodiment shown in Fig. 11 , W _i1 [k _i -169, _ki ], W _i2 [k _i -170,k _i -1], W _i3 [k _i -171,k _i -2], W _i4 [k _i -172,k _i -3], W _i5 [k _i -173,k _i -4] , W _i6 [k _i -174,k _i -5], W _i7 [k _i -175,k _i -6], W _i8 [k _i -176,k _i -7], W _i9 [k _i -177 , ki -8], W _i10 [k _i -168, _ki +1] and W _i11 [ _{k i} -179, _ki +3] window sizes can be different, and at the same time determine whether at least part of the data in the window meets the predetermined The form of the condition may also be different. In all embodiments, according to the preset rules for the deduplication server 103, the method of judging whether at least part of the data in the window W _i1 satisfies the predetermined condition _C1 is the same as the method of judging whether at least part of the data in the window W _j1 satisfies the predetermined condition _C1 It must be the same, the method of judging whether at least part of the data in W _i2 satisfies the predetermined condition _C2 is necessarily the same as the method of judging whether at least part of the data in W _j2 satisfies the predetermined condition _C2 ...judging whether at least part of the data in the window W _iM satisfies the predetermined condition C The way of _M is necessarily the same as the way of judging whether at least part of the data in the window W _jM satisfies the predetermined condition C _M . I won't repeat them here.

According to the embodiment of the present invention shown in 20 to FIG. 33 , there are preset rules on the deduplication server 103, k _a , ki , k _j , _{k l} and _km are to search for split points along the direction of data flow split point search The potential segmentation points obtained when , k _a , ki , _{k j ,} k _l and k _m all follow this rule. The window W _x [kA _x , k+B _x ] in the embodiment of the present invention represents a specific range, select data in this specific range to judge whether these data meet the predetermined condition C _x , specifically, you can select within this specific range Part of the data, or all the data may be selected to determine whether the data satisfy the predetermined condition C _x . For the window concept specifically used in the embodiment of the present invention, reference may be made to window W _x [kA _x , k+B _x ], which will not be repeated here.

In the window W _x [kA _x ,k+B _x ], (kA _x ) and (k+B _x ) represent the two boundaries of the window W _x [kA _x ,k+B _x ], where (kA _x ) represents The window W _x [kA _x ,k+B _x ] is located at the boundary of the opposite direction of the data flow split point search relative to the potential split point k, and (k+B _x ) means that the window W _x [kA _x ,k+B _x ] is relative to The potential split point k is located at the boundary of the data stream split point search direction. Specifically, in the embodiment of the present invention, the search direction of the data stream segmentation point shown in Figure 20 to Figure 33 is from left to right, where (kA _x ) represents the window W _x [kA _x , k+B _x ] Relative to the potential segmentation point k, the boundary in the opposite direction of the data flow segmentation point search (ie, the left boundary), (k+B _x ) means that the window W _x [kA _x , k+B _x ] is located in the data flow relative to the potential segmentation point k The boundary (i.e. the right boundary) of the split point lookup direction. If the search direction of the data stream split point shown in Figure 20 to Figure 33 is from right to left, then (kA _x ) means that the window W _x [kA _x , k+B _x ] is located in the data stream relative to the potential split point k The boundary in the direction opposite to the split point search (ie, the right boundary), (k+B _x ) means that the window W _x [kA _x , k+B _x ] is located at the boundary of the data flow split point search direction relative to the potential split point k (ie, the left boundary).

Those of ordinary skill in the art can realize that, in combination with the units and algorithm steps of the examples described in Figure 20 to Figure 33 of the embodiment of the present invention, the key features of the embodiment of the present invention can be combined with other technologies to present in a more complex form , but still contain the key features of the present invention. In a real environment, alternate split points may be used. For example, one embodiment is to determine 11 windows W _x [kA _x , _k +B _x ] and The predetermined condition C _x corresponding to the window W _x [kA _x , k+B _x ], x is a continuous natural number from 1 to 11, when at least part of each of the 11 windows W _x [kA _x , k+B _x ] The data all meet the predetermined condition C _x , then the potential split point _ki is the data flow split point. When the maximum data block is exceeded, the split point has not been found yet. At this time, the alternate preset rule may be used. The alternate preset The rules are similar to the preset rules on the deduplication server 103, and the spare preset rules are: for example, 10 windows W _x [kA _x , _k +B _x ] and windows W _x [kA _x ,k+B _x ] corresponding to the predetermined condition C _x , x is a continuous natural number from 1 to 10, and it is determined that at least part of the data in each of the 10 windows W _x [kA _x ,k+B _x ] satisfies the predetermined condition C _x , then the potential split point _ki is the data stream split point. When the data stream split point is not found when the set maximum data block is exceeded, the end position of the largest data block is used as the forced split point.

According to the embodiments of the present invention shown in Figures 20 to 33, there are preset rules on the deduplication server 103, in which M windows are determined for a potential segmentation point k, and a potential segmentation point k is not necessarily required first. The potential segmentation point k can be judged through the determined M windows.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present invention.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided, it should be understood that the disclosed systems and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile storage The medium includes several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned non-volatile storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), magnetic disk or optical disk, and various media capable of storing program codes.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (16)

1. a kind of for searching the server of data flow cut-point, which is characterized in that the server includes processor and interface, The processor and the interface communication, preset regular, the rule on the server are as follows: true for potential cut-point k Determine M point p_x, point p_xCorresponding window W_x[p_x-A_x, p_x+B_x] and window W_x[p_x-A_x, p_x+B_x] corresponding predetermined condition C_x, In, x arrives the continuous natural number of M, M >=2, A for 1_xAnd B_xFor integer；

The interface flows for receiving data, and the processor is for executing following steps:

It a) is current potential cut-point k according to the rule_iDetermine point p_izAnd the point p_izCorresponding window W_iz[p_iz-A_z, p_iz+ B_x], i and z are integer, and 1≤z≤M；

B) judge the window W_iz[p_iz-A_z, p_iz+B_z] at least partly data whether meet predetermined condition C_z；

As the window W_iz[p_iz-A_z, p_iz+B_z] at least partly data be unsatisfactory for the predetermined condition C_z, from the point p_izEdge The N number of data flow cut-point minimum of the data flow cut-point search direction jump searches unit U, and N*U is not more than | | B_z||+max_x (||A_x||+||(k_i-p_ix) | |), new potential cut-point is obtained, step a) is executed；

C) as the current potential cut-point k_iEach of M window window W_ix[p_ix-A_x, p_ix+B_x] at least partly Data meet predetermined condition C_x, then the current potential cut-point k_iFor data flow cut-point.

2. server according to claim 1, which is characterized in that the rule further include: at least two point p_eAnd p_f, meet Condition A_e=A_f, B_e=B_f, C_e=C_f。

3. server according to claim 2, which is characterized in that the rule further include: the 1 points p_eAnd p_f, Relative to the potential cut-point k, searched on opposite direction in the data flow cut-point.

4. server according to claim 2 or 3, which is characterized in that the rule further include: the 1 points p_e And p_fThe distance between be 1 U.

5. server according to any one of claims 1 to 3, which is characterized in that the processor is specifically used for using random Function judges the window W_iz[p_iz-A_z, p_iz+B_z] at least partly data whether meet the predetermined condition C_z。

6. server according to claim 5, which is characterized in that the processor is specifically used for judging using hash function The window W_iz[p_iz-A_z, p_iz+B_z] at least partly data whether meet the predetermined condition C_z。

7. server according to any one of claims 1 to 3, which is characterized in that as the window W_iz[p_iz-A_z, p_iz+B_z] in At least partly data are unsatisfactory for the predetermined condition C_z, from the point p_izIt jumps along the data flow cut-point search direction N number of Data flow cut-point minimum searches unit U, obtains the new potential cut-point, is described new potential according to the rule The point p that cut-point determines_icCorresponding window w_ic[p_ic-A_c, p_ic+B_c] left margin and the window W_iz[p_iz-A_z, p_iz+B_z] The point p that right margin is overlapped or determines for the new potential cut-point_icThe corresponding window w_ic[p_ic-A_c, p_ic+ B_c] left margin be located at the window W_iz[p_iz-A_z, p_iz+B_z] within the scope of；Wherein, it is determined for the new potential cut-point The point p_icIt is according to the rule, the M point determined for the new potential cut-point is obtained according to data flow search direction The point of sequence first in the sequence obtained.

8. server according to claim 5, which is characterized in that the processor judges the window using random function W_iz[p_iz-A_z, p_iz+B_z] at least partly data whether meet the predetermined condition C_z, it specifically includes:

In the window W_iz[p_iz-A_z, p_iz+B_z] F byte of middle selection, the F byte is recycled H times, obtains F*H altogether A byte is denoted as a wherein each byte is formed by 8_{M, 1}...a_{M, 8}, indicate the 1st of m-th of byte in the F*H byte To the 8th, the corresponding position of the F*H byte can be indicated are as follows:Work as a_{M, n}When=1, V_{Am, n}=1, work as a_{M, n}When=0, V_{Am, n}=-1, wherein a_{M, n}Indicate a_{M, 1}...a_{M, 8}Any of, the F*H byte is corresponding Position is according to a_{M, n}With V_{Am, n}Transformational relation obtain matrix V_a, the matrix V_aIt indicates are as follows:F*H*8 random number of selection forms matrix R from the random number of service normal distribution, The matrix R is indicated are as follows:By the matrix V_aM row and the matrix R m row Random number be multiplied, then summation obtain a value, be embodied as s_am=V_{Am, 1}*h_{M, 1}+V_{Am, 2}*_{M, 2}+...+V_{Am, 8}*h_{M, 8}, together Reason obtains s_a1、s_a2... arrive s_aF*H, count s_a1、s_a2... arrive s_aF*HIt is middle meet greater than 0 value number K, when K be even number, then The window W_iz[p_iz-A_z, p_iz+B_z] at least partly data meet the predetermined condition C_z。

9. a kind of for searching the server of data flow cut-point, which is characterized in that the server includes processor and interface, The processor and the interface communication, preset regular, the rule on the server are as follows: true for potential cut-point k Determine M window W_x[k-A_x, k+B_x] and window W_x[k-A_x, k+B_x] corresponding predetermined condition C_x, wherein x arrives M continuously certainly for 1 So number, M >=2, A_xAnd B_xFor integer；

The interface flows for receiving data, and the processor is for executing following steps:

It a) is current potential cut-point k according to the rule_iDetermine corresponding window W_iz[k_i-A_z, k_i+B_z], i and z are integer, and And 1≤z≤M；

B) judge the window W_iz[k_i-A_z, k_i+B_z] at least partly data whether meet predetermined condition C_z；

As the window W_iz[k_i-A_z, k_i+B_z] at least partly data be unsatisfactory for the predetermined condition C_z, from described current potential Cut-point k_iThe N number of lookup of data flow cut-point minimum unit U, N*U are jumped no more than ‖ along the data flow cut-point search direction B_z ‖+max_x(‖A_x‖), new potential cut-point is obtained, step a) is executed；

C) as the current potential cut-point k_iEach of M window window W_ix[k_i-A_x, k_i+B_x] at least partly count According to meeting predetermined condition C_x, then the current potential cut-point k_iFor data flow cut-point.

10. server according to claim 9, which is characterized in that the rule further include: at least two window W_ie[k_i- A_e, k_i+B_e] and W_if[k_i-A_f, k_i+B_f], meet condition: | A_e+B_e|=| A_f+B_f|, C_e=C_f。

11. server according to claim 10, which is characterized in that be the server preset rules, the rule is also It include: A_eAnd A_fFor positive integer.

12. server described in 0 or 11 according to claim 1, which is characterized in that the rule further include: A_e- 1=A_f, B_e+1 =B_f。

13. according to any server of claim 9 to 11, which is characterized in that the processor be specifically used for use with Machine function judges the window W_iz[k_i-A_z, k_i+B_z] at least partly data whether meet the predetermined condition C_z。

14. server according to claim 13, which is characterized in that the processor is specifically used for sentencing using hash function Break the window W_iz[k_i-A_z, k_i+B_z] at least partly data whether meet the predetermined condition C_z。

15. according to any server of claim 9 to 11, which is characterized in that as the window W_iz[k_i-A_z, k_i+B_z] In at least partly data be unsatisfactory for the predetermined condition C_z, from the current potential cut-point k_iIt is looked into along the data flow cut-point The direction N number of data flow cut-point minimum of jump is looked for search unit U, obtain the new potential cut-point is according to the rule The window w that the new potential cut-point determines_ic[k_i-A_c, k_i+B_c] left margin and the window W_iz[k_i-A_z, k_i+B_z] The window w that right margin is overlapped or determines for the new potential cut-point_ic[k_i-A_c, k_i+B_c] left margin be located at institute State window W_iz[k_i-A_z, k_i+B_z] within the scope of；Wherein, the window w determined for the new potential cut-point_ic[k_i-A_c, k_i+B_c] it is to be obtained for the M window that the new potential cut-point determines according to data flow search direction according to the rule The window of sequence first in sequence.

16. server according to claim 13, which is characterized in that the processor judges the window using random function Mouth W_iz[k_i-A_z, k_i+B_z] at least partly data whether meet the predetermined condition C_z, it specifically includes:

In the window W_iz[k_i-A_z, k_i+B_z] F byte of middle selection, the F byte is recycled H times, obtains F*H altogether A byte is denoted as a wherein each byte is formed by 8_{M, 1}...a_{M, 8}, indicate the 1st of m-th of byte in the F*H byte To the 8th, the corresponding position of the F*H byte can be indicated are as follows:Work as a_{M, n}When=1, V_{Am, n}=1, work as a_{M, n}When=0, V_{Am, n}=-1, wherein a_{M, n}Indicate a_{M, 1}…a_{M, 8}Any of, the F*H byte is corresponding Position is according to a_{M, n}With V_{Am, n}Transformational relation obtain matrix V_a, the matrix V_aIt indicates are as follows:F*H*8 random number of selection forms matrix R from the random number of service normal distribution, The matrix R is indicated are as follows:By the matrix V_aM row and the matrix R m row Random number be multiplied, then summation obtain a value, be embodied as s_am=V_{Am, 1}*h_{M, 1}+V_{Am, 2}*h_{M, 2}+...+V_{Am, 8}*h_{M, 8}, Similarly, s is obtained_a1、s_a2... arrive s_aF*H, count s_a1、s_a2... arrive s_aF*HIt is middle meet greater than 0 value number K, when K be even number, The then window W_iz[k_i-A_z, k_i+B_z] at least partly data meet the predetermined condition C_z。